Question 8 answer implies that since one needs an “exponentially larger” network when flattening a deep network, that “an exponentially smaller network” is required going in the opposite direction. Should’nt this be “Logarithmically smaller” ? ( Prof. Ng Also writes that the Layer count increase is O(logN) )
Hi @dave_merit , I am not a mathematician by profession but strictly speaking I think you are right. If SizeNetwork = a ^ LayerCount (so the network grows exponentially with the layer count) then LayerCount = Log a (SizeNetwork). So the network will reduce logarithmically if you reduce the number of layers. Probably the term logarithmically is not used as not to confuse the students.
1 Like
It depends on the deep network itself. The one Dr. Ng explained for Deep Neural Network has large number of input features (let’s say a 1000) then it’s followed by half of that nodes in the next layer(let’s say =500 ) and half of that in he next layer (250) and so on. in this case the order of this algorithm is log (n). So it is not always like that, this is just a special case and this is what the question is highlighting on as I think.