From these conversations I am confused again. I had an older post about scaling using mean and standard deviation across all features. Should not this reduce the difference between data points the most? Yet in my experiments done here doing this actually results in a much slower convergence…
My guess is that normalizing across all features probably mars the distinct contribution from each feature. This is very non-precise unfortunately.