A doubt on Week 3 Lecture

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
1

Dear Administrator,

Could you please guide me on this issue?

“Note that for any variable, foo and dfoo always have the same dimensions. That’s why, w and dw always have the same dimension. Similarly, for b and db, and z and dz, and so on.”

May i know the reason of this statement is valid?

image
image1386×660 139 KB

Thank you

2

In the assignment, you can confirm this statement by printing the shape of W and dW, b and db, Z and dZ, A and dA (of the same layer). You will see every pair has the same shape but the one contain values and another one it’s derivatives.

3

Dear Mr Saif,

image
image912×275 13 KB

According to the lecture,

  1. The dimension of Z[1] is (n[1], 1)
  2. There is no any explanation on why dZ[1] has a dimension (n[1],1) as same as Z[1], except just only this statement

“Note that for any variable, foo and dfoo always have the same dimensions. That’s why, w and dw always have the same dimension. Similarly, for b and db, and z and dz, and so on.”

  1. Based on the ambiguous assumption, dZ[1] is defined to have a dimension of (n[1],1). So we have to arrange a element-wise multiplication between (W[2].T) (dZ[2]) and g[1]'(Z[1]), just to fulfill the requirement of having (n[1],1) dimension.

  2. (W[2].T) (dZ[2]) is defined to calculate using dot product. In the same way, the element-wise multiplication between (W[2].T) (dZ[2]) and g[1]'(Z[1]) can be changed to using dot product as well.

May i have any idea to understand the abovementioned statement?

Thank you

4

Yes, i noticed that this is true when i tried it in the assignment. But i find that it is quite coincident so that i am looking for an idea of understanding the theory behind it.

5

The dimension of Z depends on the dimension of W and X, right? And the dimension of X depends on the number of features and number of examples. Moreover, the dimension of W depends on the number of neurons and number of features. Prof. Andrew explain the dimensions deeply in one of his video.

6

Yes, every term is clear except this dZ[1] = (W[2].T) (dZ[2]) * g[1]'(Z[1]).

According to the lecture, there is an assumption mentioning that dZ[1] has a same dimension as Z[1], So we have to arrange a element-wise multiplication (instead of dot product) between (W[2].T) (dZ[2]) and g[1]'(Z[1]), just to fulfill the requirement of having (n[1],1) dimension.

The dimension of dZ[1] depends on the dimension of W[2].T , dZ[2] ,and g[1]'(Z[1]).

Element-wise Multiplication:
dZ[1] = (W[2].T) (dZ[2]) * g[1]'(Z[1])

Dot Product
dZ[1] = (W[2].T) (dZ[2]) . g[1]'(Z[1])

Two different calculations produce different dimensions of result. How should we ensure that dimension of dZ[1] = dimension of Z[1]?

7

This is the derivative calculation. To understand this equation, you have to be familiar with Calculus. There are many posts related to this topic. You can read this and this and yet this one.