DLS C5W3 - "Attention" with LSTMs - Interesting discrepancy

Hi,
I browsed through the forums quickly to see whether it has already been mentioned before, but I didn’t find anything.

I noticed that the heat map offered by the text suggests that the input interval relevant for outputting the month digits includes only “May” and excludes “Saturday”:

However, running the 3.1 part of the exercise immediately below, gives a heat map output that includes the day “Tuesday”!

My intuition agrees with the latter, as months would often appear as full words both before and after the day in inputs, hence the model should have to examine both words to be able to decode the month. Am I missing something?

I think you did catch most of important points. A few things that I may want to clarify are;

Please see following two figures.

Both are to translate “Tuesday 09 October 1993” to the standard format. The difference is a threshold value to visualize data. In the 2nd figure, there is no significant “alpha” for “Tuesday” , just like you found. But, again, both are identical.

I suppose, the author thought that using the 2nd one is better to illustrate mappings between input and output.

My intuition agrees with the latter, as months would often appear as full words both before and after the day in inputs, hence the model should have to examine both words to be able to decode the month.

I suppose your interpretation is mostly right, but what you may want to be careful is,… This model is using “character” level transformation, not “word” level. And, the post_LSTM does not take the previous time step’s prediction y^{<t-1>} as input. There is no context like “before month/after month”. Probably, “blank” and character position contributed to the \alpha values.

Thanks, yeah I guess choosing an appropriately high cutoff, you can make one appear like the other.
Also, point well taken regarding the absence of any notion of words or ordering in this setup, I was speaking fast and loose - indeed I should have referred to the corresponding character locations.