Sometimes still confusion of different multiplication types

Hi,

I identified two main problems (for me) in implementing the functions in NNs. One is using the right dimensions of the different data structures (like inputs, weights, activations asf). Andrew covers this quite detailed, and I guess you simply have to think about it every single time to get it right.

The other problem is the type of multiplication to use. So I repeatedly am in the situation when I know “I have to multiply this variable with this other variable”, but I am not clear about the type of multiplication to use.

  • Elementwise? Use .multiply() or the * operator?
  • dot product?
  • matrix multiplication?
  • with or without broadcasting?

So there are quite a few options for different multiplication types to choose, and I still manage to get it wrong sometimes and have to start understanding which of the many lines of code probably messed it up (because there are usally several multiplications in a function, and each is a candidate …).

Therefore, I wondered if there might be some easy “rules of thumb” that can be applied to this question. The kind of rule “Everytime you want to do x, you know it’s a dot product.”, where x is some typical operation (like applying the weights to the the inputs or so).

If you have any thoughts on this or know any help, please let me know.

Thanks and best regards
Matthias

Hi @Matthias_Kleine ,

Here is a thread written by one of our mentors, Paul M, may help to answer some of your questions.