RNN Concepts too confusing

[arvyzukai]

@Mayank11

a. Is it right to assume that in first time step embedding for “I” will be sent to the model, then in 2nd time step embedding for “am” will be sent and so on till 6th step when embedding for “learning” will be sent to model?

That is correct.

@Mayank11

b. Dimension of x should be (batch_size * max_len * embed_dim).

Yes, that would be the output of embedding layer and input to the RNN layer. (More on this)

@Mayank11

In forward function, for concatenation to work when one element of x is sent to the model, dimension of h_t should have been (max_len * embed_dim) But dimension of h is (embed_dim,1). How come?

I’m not sure I fully understand your question. What is the context of this? In other words what are referring here? To be short, for one output you need (embed_dim, 1), for more outputs you need (embed_dim, more_dim), but maybe I do not understand the context here.

@Mayank11

a. The ShiftRight layer shifts input to the right by n_positions which for n_positions = 1 will shift our input to “0”, “am”, “happy”, “I”, “am”, “learning”. Isn’t this causing the first word (“I” here) to be ignored altogether? What is the purpose of ShiftRight layer?

The purpose of the ShiftRight layer here is to get the targets, so [“I”, “am”, “happy”] would become [“<sos>”, “I”, “am”, “happy”]. Check this post for more details.

For custom GRU check this post.

@Mayank11

The lecture says that deep RNNs are just RNNs stacked on top of each other. The input to next layer RNN is the output / activation of previous layer. Since this happens after 1 time step, does this mean that for a network of k RNNs stacked on top of each other, the earliest we can get an output is after k+1 time step? If so, what use cases can this possibly be used ?

Stacking layers of Recurrent RNNs significantly improve the model’s ability to learn complex sequential patterns and dependencies in sequential data. Deep RNNs can improve generalization by learning a more compact and abstract representation of the data. The lower layers capture the specific details, while the higher layers learn more generalized features that are relevant to the task.
I’m not sure what you mean by “… the earliest we can get an output is after k+1 time step…”

Cheers

P.S. It is a better practice to ask one question a topic, this way others might find the same question/answers easier. And it also help to drive the conversation better (one answer might change the other question).