I asked ChatGPT about the look of the "weight landscape" and it gave good pointers

[dtonhofer]

See also this post by @paulinpaloalto

Deep Neural Networks Have Many Global Optima

[Note: this material was authored by mentor Gordon Robinson and is the contents of a thread he created on the Coursera Forums of the previous version of the course. I’m bringing it over to the new Discourse platform with Gordon’s permission. Thanks, @GordonRobinson!] A deep neural network has many, many global optima. This is true for any loss function: there is no convex loss function for a deep network. The reason is simple: the internal area of the network is so symmetric that we can change…

which references these papers:

Weight-space symmetry in deep networks gives rise to permutation saddles, connected by equal-loss valleys across the loss landscape

Weight-space symmetry in deep networks gives rise to permutation saddles,...

and

Visualizing the Loss Landscape of Neural Nets

Visualizing the Loss Landscape of Neural Nets