# RNN Concepts too confusing **URL:** \ **Category:** NLP with Sequence Models\ **Tags:** week-module-2\ **Created:** [July 31, 2023, 9:30am UTC](https://community.deeplearning.ai/t/rnn-concepts-too-confusing/403097 "2023-07-31T09:30:35Z")\ **Posts on this page:** 1\ **Showing post:** 4
**Author:** ![arvyzukai](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/arvyzukai/32/165165_2.png) [@arvyzukai](https://community.deeplearning.ai/u/arvyzukai)\ **Post date:** [July 31, 2023, 1:06pm UTC](https://community.deeplearning.ai/t/rnn-concepts-too-confusing/403097/4 "2023-07-31T13:06:26Z")
Hi @Mayank11 Have you checked [this post](https://community.deeplearning.ai/t/creating-a-gru-model-using-trax/165167/4) because it addresses most of the questions? In this post you can see all the shapes (you can change batch dimension of 1 to any number without any problem). But to try to answer them again: > [@Mayank11](#): > > I am trying to point out that dimension of x is (batch\_size \* max\_len \* embed\_dim) while dimension of h is (embed\_dim). In forward\_GRU function, just before multiplying with Whh, lecture asked us to vertically concatenate x, h\_t. How can we vertically concatenate a (max\_len \* embed\_dim) dimension matrix with (embed\_dim \*1) dimension matrix ? If I understand you correctly, the lecture shows how you can take one input (x\_t) (one word, the embedding of that word, for exampe, tensor of shape (1, 512)), concatenate it with previous hidden\_state (h\_{t-1}) (tensor of shape (1, 512)) and do one matrix multiplication with W\_ht (which shape should be (1024, 512)). This way you do just one matrix multiplication which result is equivalent to having two (W\_hx and W\_hh) weight matrices and two inpus (x and h). Here the batch\_dimension is 1, but it could be any number. > [@Mayank11](#): > > My question here is - if input is embedding of one word at a time then why shift right? Wouldn’t this cause shift in the embedding dimension? eg if embedding for “a” is [[[54,23,35]]], it would become [[[0,54,23]]] which could be a different word altogether in that space. In other words, shift right makes your input to [0,54,23] and you have to predict target [54,23,35]. And of course, your sequence length (max\_len) is not 3 like in this example, but longer (like 64 in the assignment) , which usually does not cut off the last word, but some padding token. > [@Mayank11](#): > > The output of 1 time step of GRU is y(t) which becomes input to next layer of RNN. Similarly output of this layer of RNN is input to the third layer of RNN. By the time 3rd layer receives first input, 2 timesteps have passed. You misunderstand this concept. This is considered one time-step. First layer receives and input, produces the output which is the input for the layer “above it”, that layer receives this input which is the output for another layer that is “above it”. This is one time-step. When all the layers finished, the next token is the input and the whole thing repeats for every time-step. > [@Mayank11](#): > > This is interesting but I wonder what specific details / generalized features mean here - we got semantic dependencies from embedding, we have also learnt about algo dedicated to PoS tagging etc… What are these other generalized features? How deep we should go for use cases like machine translation or sentiment analysis? They are hard to interpret for humans. Some are obvious like punctuation or POS, some seam completely random. A [classic post about character level RNNs](http://karpathy.github.io/2015/05/21/rnn-effectiveness/) and the features they learn (in the middle of the article) Cheers --- _[View the full topic](https://community.deeplearning.ai/t/rnn-concepts-too-confusing/403097)._