As a bookend to "Optimization Algorithms", some literature

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

We find the following:

The Unreasonable Effectiveness of Deep Learning in Artificial Intelligence

Terrence J. Sejnowski

2020-02-12

https://www.pnas.org/doi/10.1073/pnas.1907373117

Deep learning networks have been trained to recognize speech, caption photographs and translate text between languages at high levels of performance. Although applications of deep learning networks to real world problems have become ubiquitous, our understanding of why they are so effective is lacking. These empirical results should not be possible according to sample complexity in statistics and non-convex optimization theory. However, paradoxes in the training and effectiveness of deep learning networks are being investigated and insights are being found in the geometry of high-dimensional spaces. …

The essay concludes:

In his essay on “The unreasonable effectiveness of mathematics in the natural sciences,” (PDF) Eugene Wigner marveled that the mathematical structure of a physical theory often reveals deep insights into that theory that lead to empirical predictions. Also remarkable is that there are so few parameters in the equations, called physical constants. The title of this article mirrors Wigner’s. But unlike the laws of physics, there is an abundance of parameters in deep learning networks and they are variable. We are just beginning to explore representation and optimization in very high-dimensional spaces. Perhaps someday an analysis of the structure of deep learning networks will lead to theoretical predictions and reveal deep insights into the nature of intelligence. We can benefit from the blessings of dimensionality.

Having found one class of functions to describe the complexity of signals in the world, perhaps there are others. Perhaps there is a universe of massively-parallel algorithms in high-dimensional spaces that we have not yet explored, which go beyond intuitions from the three-dimensional world we inhabit and the one-dimensional sequences of instructions in digital computers. Like the gentleman square in flatland and the explorer in the Flammarion engraving, we have glimpsed a new world stretching far beyond old horizons.

A followup by Leslie S. Smith is here:

1 Like
2

I have not yet read any of the papers linked here and have never studied non-convex optimization theory, but I don’t understand how Dr. Sejnowski squares this claim with the Universal Approximation Theorem. The UAT says all this absolutely is possible at least in theory. The only drawback of the UAT is that it gives you exactly zero help in actually finding such approximation functions. But then entering from stage left to take a bow, we have back propagation and conjugate gradient methods.

Thanks very much for deepening the discussions here!

1 Like