C4_W1_Ungraded_Lab_1: Dimension question

Course Q&A Natural Language Processing NLP with Attention Models
1

I am trying to makes sense of vector and matrix dimensions for ungraded Lab 1.

Under section 1: Calculating alignment scores

1st question:
“n is the hidden dimension size”. Does that imply vectors s and h are in R^n? If so, the matrix multiplication Ws and Uh are not defined, as W and H are nxm matrices. Perhaps s and h are in R^m so this matrix multiplication works?

2nd question:
Near the bottom of the diagram is listed “Dense(2m,n)”. Assuming s and h are in R^m, I believe this should be Dense(n,2m). The matrices W and H are stacked side-by-side: [W|H] to create a nx2m matrix. This is multiplied by the vector [s|h]^T in R^2m to produce [W|H][s|h]^T = Ws + Uh.

Please let me know if I am correct, or if I’m missing something.

Thank you.

2

I also found it challenging to make sense of the vector and matrix dimensions.

The lab says that n is the dimension of the hidden state. I take this to mean that n is the size of the vectors s and h, as you stated above.

If that is the case, it would make more sense if instead of W_a s_{i-1} we had s_{i-1}^T W_a, since W_a is n \times m. I think this latter formulation is also consistent with the solution to the exercise where we have np.matmul(inputs, layer_1).

Would appreciate confirmation of whether my understanding is correct.