C5_W2_A2: LSTM output connections

Course Q&A Deep Learning Specialization Sequence Models
1

Greetings!

Can you help me to understand: how the highlighted LSTM connections are expressed in the code?

image
image1491Ă—871 77.3 KB

2

Hello @AKazak,

It’s a good question!

image
image876Ă—444 53.5 KB

The code snapshot shows four lines of the exercise 5’s function Emojify_V2 which implements the network. You should be able to easily locate them with the comments.

For the upward connection that you questioned, the first X of the lines is an output of sequence which encapsulates a^{[1]<t>} for all t, so when you use that X in the next line, you are building that connection. You build connection by using an output of a layer as an input to another layer.

For the rightward, since it is just how LSTM works, it is only natural that it is already implemented inside Tensorflow’s LSTM object, so we can’t see it in the lines of the exercise.

Cheers,
Raymond

3

Hello @AKazak,

A few added notes in the screenshot:

image
image1310Ă—679 143 KB

Even though we have many LSTM boxes, we need only two LSTM objects for code implementation. Likewise, for all the dropout boxes there, they are actually the same Dropout object.

Note that the graph is just expanding out all the timesteps, but one LSTM will handle all timesteps which is why LSTM is a kind of “Recurrent” neural network. :wink:

Cheers,
Raymond

4

Thank you for providing this very clear explanation.

A small follow-up question: what do you mean by a “time step” in the context of words (and their encodings)?

5

You are welcome, @AKazak. By a timestep, I mean a token. We have T_x boxes in the graph, because there are T_x tokens. Each box is a step.

6

Now it’s clear.

Thank you!

The time dimension has been just slightly discussed in lectures. I assume that it is called “time step” by historical reasons when they used recurrent network for time series prediction. Right?

7

At least that’s my reason. I am not the right person to talk about history because I think I have not done enough literature review.

I probably should be more careful next time when calling it “time step” because, after all, these networks are for sequential problem and time series is just one aspect of it.

Cheers,
Raymond