Yes, the solution spaces here are incredibly complex with lots of non-linear coupling all over the place. Even before you start fiddling with hyperparameters 
…
Prof Ng does spend quite a bit of time here in Course 2 talking about how to approach tuning hyperparameters in a systematic way. But, as you say, there is still some “art” to it.
The phrase “solution space” always triggers memories of this thread, which is worth a look for the reference to the LeCun paper mentioned there.