Lstm_backward wrong output

Derek_Kim · November 22, 2021, 11:22pm

Hi,

my LSTM_backward keeps on outputting values removed from expected ones even though my LSTM_cell_backward seems to output the right ones.
I simply cannot understand what I may be doing wrong.

# UNGRADED FUNCTION: lstm_backward

def  lstm_backward(da, caches):
    
    """
    Implement the backward pass for the RNN with LSTM-cell (over a whole sequence).

    Arguments:
    da -- Gradients w.r.t the hidden states, numpy-array of shape (n_a, m, T_x)
    caches -- cache storing information from the forward pass (lstm_forward)

    Returns:
    gradients -- python dictionary containing:
                        dx -- Gradient of inputs, of shape (n_x, m, T_x)
                        da0 -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)
                        dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
                        dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
                        dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)
                        dWo -- Gradient w.r.t. the weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
                        dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)
                        dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)
                        dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)
                        dbo -- Gradient w.r.t. biases of the output gate, of shape (n_a, 1)
    """

    # Retrieve values from the first cache (t=1) of caches.
    (caches, x) = caches
    (a1, c1, a0, c0, f1, i1, cc1, o1, x1, parameters) = caches[0]
    
    ### START CODE HERE ###

   {*Moderator's Edit: Solution Code Removed*}
    
    ### END CODE HERE ###

    # Store the gradients in a python dictionary
    gradients = {"dx": dx, "da0": da0, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
                "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}
    
    return gradients

gradients["dx"][1][2] = [-0.00424213  0.28205375 -0.48292508 -0.43281115]
gradients["dx"].shape = (3, 10, 4)
gradients["da0"][2][3] = 0.36036164521813085
gradients["da0"].shape = (5, 10)
gradients["dWf"][3][1] = -0.185172422655972
gradients["dWf"].shape = (5, 8)
gradients["dWi"][1][2] = 0.4051243309298186
gradients["dWi"].shape = (5, 8)
gradients["dWc"][3][1] = -0.07937467355121491
gradients["dWc"].shape = (5, 8)
gradients["dWo"][1][2] = 0.03894877576298697
gradients["dWo"].shape = (5, 8)
gradients["dbf"][4] = [-0.17049796]
gradients["dbf"].shape = (5, 1)
gradients["dbi"][4] = [-0.50848333]
gradients["dbi"].shape = (5, 1)
gradients["dbc"][4] = [-0.42510818]
gradients["dbc"].shape = (5, 1)
gradients["dbo"][4] = [-0.17958196]
gradients["dbo"].shape = (5, 1)

Expected Output:

gradients["dx"][1][2] =	[0.00218254 0.28205375 -0.48292508 -0.43281115]
gradients["dx"].shape =	(3, 10, 4)
gradients["da0"][2][3] =	0.312770310257
gradients["da0"].shape =	(5, 10)
gradients["dWf"][3][1] =	-0.0809802310938
gradients["dWf"].shape =	(5, 8)
gradients["dWi"][1][2] =	0.40512433093
gradients["dWi"].shape =	(5, 8)
gradients["dWc"][3][1] =	-0.0793746735512
gradients["dWc"].shape =	(5, 8)
gradients["dWo"][1][2] =	0.038948775763
gradients["dWo"].shape =	(5, 8)
gradients["dbf"][4] =	[-0.15745657]
gradients["dbf"].shape =	(5, 1)
gradients["dbi"][4] =	[-0.50848333]
gradients["dbi"].shape =	(5, 1)
gradients["dbc"][4] =	[-0.42510818]
gradients["dbc"].shape =	(5, 1)
gradients["dbo"][4] =	[ -0.17958196]
gradients["dbo"].shape =	(5, 1)

# UNGRADED FUNCTION: lstm_cell_backward

def lstm_cell_backward(da_next, dc_next, cache):
    """
    Implement the backward pass for the LSTM-cell (single time-step).

    Arguments:
    da_next -- Gradients of next hidden state, of shape (n_a, m)
    dc_next -- Gradients of next cell state, of shape (n_a, m)
    cache -- cache storing information from the forward pass

    Returns:
    gradients -- python dictionary containing:
                        dxt -- Gradient of input data at time-step t, of shape (n_x, m)
                        da_prev -- Gradient w.r.t. the previous hidden state, numpy array of shape (n_a, m)
                        dc_prev -- Gradient w.r.t. the previous memory state, of shape (n_a, m, T_x)
                        dWf -- Gradient w.r.t. the weight matrix of the forget gate, numpy array of shape (n_a, n_a + n_x)
                        dWi -- Gradient w.r.t. the weight matrix of the update gate, numpy array of shape (n_a, n_a + n_x)
                        dWc -- Gradient w.r.t. the weight matrix of the memory gate, numpy array of shape (n_a, n_a + n_x)
                        dWo -- Gradient w.r.t. the weight matrix of the output gate, numpy array of shape (n_a, n_a + n_x)
                        dbf -- Gradient w.r.t. biases of the forget gate, of shape (n_a, 1)
                        dbi -- Gradient w.r.t. biases of the update gate, of shape (n_a, 1)
                        dbc -- Gradient w.r.t. biases of the memory gate, of shape (n_a, 1)
                        dbo -- Gradient w.r.t. biases of the output gate, of shape (n_a, 1)
    """

    # Retrieve information from "cache"
    (a_next, c_next, a_prev, c_prev, ft, it, cct, ot, xt, parameters) = cache
    
    ### START CODE HERE ###

    {*Moderator's Edit: Solution Code Removed*} 

    ### END CODE HERE ###
    
    # Save gradients in dictionary
    gradients = {"dxt": dxt, "da_prev": da_prev, "dc_prev": dc_prev, "dWf": dWf,"dbf": dbf, "dWi": dWi,"dbi": dbi,
                "dWc": dWc,"dbc": dbc, "dWo": dWo,"dbo": dbo}

    return gradients

gradients["dxt"][1][2] = 3.2305591151091875
gradients["dxt"].shape = (3, 10)
gradients["da_prev"][2][3] = -0.06396214197109236
gradients["da_prev"].shape = (5, 10)
gradients["dc_prev"][2][3] = 0.7975220387970015
gradients["dc_prev"].shape = (5, 10)
gradients["dWf"][3][1] = -0.1479548381644968
gradients["dWf"].shape = (5, 8)
gradients["dWi"][1][2] = 1.0574980552259903
gradients["dWi"].shape = (5, 8)
gradients["dWc"][3][1] = 2.3045621636876668
gradients["dWc"].shape = (5, 8)
gradients["dWo"][1][2] = 0.3313115952892109
gradients["dWo"].shape = (5, 8)
gradients["dbf"][4] = [0.18864637]
gradients["dbf"].shape = (5, 1)
gradients["dbi"][4] = [-0.40142491]
gradients["dbi"].shape = (5, 1)
gradients["dbc"][4] = [0.25587763]
gradients["dbc"].shape = (5, 1)
gradients["dbo"][4] = [0.13893342]
gradients["dbo"].shape = (5, 1)

Expected Output:

gradients["dxt"][1][2] =	3.23055911511
gradients["dxt"].shape =	(3, 10)
gradients["da_prev"][2][3] =	-0.0639621419711
gradients["da_prev"].shape =	(5, 10)
gradients["dc_prev"][2][3] =	0.797522038797
gradients["dc_prev"].shape =	(5, 10)
gradients["dWf"][3][1] =	-0.147954838164
gradients["dWf"].shape =	(5, 8)
gradients["dWi"][1][2] =	1.05749805523
gradients["dWi"].shape =	(5, 8)
gradients["dWc"][3][1] =	2.30456216369
gradients["dWc"].shape =	(5, 8)
gradients["dWo"][1][2] =	0.331311595289
gradients["dWo"].shape =	(5, 8)
gradients["dbf"][4] =	[ 0.18864637]
gradients["dbf"].shape =	(5, 1)
gradients["dbi"][4] =	[-0.40142491]
gradients["dbi"].shape =	(5, 1)
gradients["dbc"][4] =	[ 0.25587763]
gradients["dbc"].shape =	(5, 1)
gradients["dbo"][4] =	[ 0.13893342]
gradients["dbo"].shape =	(5, 1)

TMosh · November 25, 2021, 4:00am

Do you still need help with this question?

Derek_Kim · November 25, 2021, 4:41am

Hi, TMosh. Yes, I still haven’t figured it out.
Guidance would be much appreciated.

TMosh · November 25, 2021, 4:58am

I haven’t implemented this optional part of the assignment, so I’ll need to do some research before I can be very helpful.

Gabriel_Giurgica · November 30, 2021, 12:35pm

Hi Derek_Kim,

I have tried to reproduce your results on my computer, but I could not. Everything works fine. The implementation is good. I do not understand the cause of your problem.

Maybe you should try to restart the Jupyter kernel.

Derek_Kim · December 5, 2021, 2:21am

Gabriel,

Thank you for your input.
I’m not quite sure why, but it seems that everything from dx to dWf and dbf are wrong as per the output below, which is the result I get when I restart the kernel and run everything.
Unfortunately, I am just lost.
I am thinking my lstm_cell_backward is right, because the output is as expected after checking multiple times.
I am wondering if I am missing something at gradients = lstm_cell_backward(da[:,:,t] + da_prevt, dc_prevt, caches[t]) in lstm_backward.

gradients["dx"][1][2] = [-0.00424213  0.28205375 -0.48292508 -0.43281115]
gradients["dx"].shape = (3, 10, 4)
gradients["da0"][2][3] = 0.36036164521813085
gradients["da0"].shape = (5, 10)
gradients["dWf"][3][1] = -0.185172422655972
gradients["dWf"].shape = (5, 8)
gradients["dWi"][1][2] = 0.4051243309298186
gradients["dWi"].shape = (5, 8)
gradients["dWc"][3][1] = -0.07937467355121491
gradients["dWc"].shape = (5, 8)
gradients["dWo"][1][2] = 0.03894877576298697
gradients["dWo"].shape = (5, 8)
gradients["dbf"][4] = [-0.17049796]
gradients["dbf"].shape = (5, 1)
gradients["dbi"][4] = [-0.50848333]
gradients["dbi"].shape = (5, 1)
gradients["dbc"][4] = [-0.42510818]
gradients["dbc"].shape = (5, 1)
gradients["dbo"][4] = [-0.17958196]
gradients["dbo"].shape = (5, 1)

bensch98 · December 17, 2021, 8:08pm

Hi Derek_Kim,
I got exactly the same results as you in lstm_backward(). Have your found the error yet?
The only difference to your code is, that I used np.multiply in lstm_cell_backward() for element-wise multiplication instead of the * operator. But this is exactly the same and the outputs of lstm_cell_backward are also correct.

bensch98 · December 17, 2021, 8:27pm

Did anyone already do this exercise successfully and can confirm that the values of the Jupyter Notebook are indeed correct?
Help from a tutor would be much appreciated.

TMosh · December 21, 2021, 5:59am

I’m working through the backpropagation exercise, but it’s neither easy nor particularly fun.

weic · May 4, 2022, 7:43pm

I have the exact same problem. With the results for single cell working out and the overall lstm backprop results not matched. Since the logic is similar to rnn backprop code I’m starting to suspect maybe the notebook’s expected output is not correct.

TMosh · May 5, 2022, 3:53am

There haven’t been any other reports of possible incorrect notebook “expected output” values, so I suspect they are correct.

If you have further questions, please start a new thread so another mentor will pick up the conversation, since I’m going to be off-line for a few days.

Gabr · June 14, 2022, 8:26pm

Just to give you a heads-up, I had this exact problem and the exact same outputs as the OP. Turns out the problem was in the lstm_forward function. Unfortunately, I had ignored the following instruction given in the (Section 2.2):

“Note: create c_next as its own variable with its own location in memory. Do not initialize it as a slice of the 3D tensor 𝑐c. In other words, don’t do c_next = c[:,:,0].”

Once I fixed this using np.copy, everything worked as expected.

Tom_Pham · November 25, 2022, 11:17pm

I notice that in your lstm_cell_backward function, you retrieved da_next’s shape instead of a_next’s. However, I have no clue if it has anything to do with the problem you’re having.
Furthermore, in your lstm_backward function, under the instruction to Set the first activation’s gradient to the backpropagated gradient da_prev, I use da0 = gradients['da_prev'] instead.

ravibilla · February 29, 2024, 2:50am

Thank you @Gabr. I had the exact same problem and your suggestion fixed the issue.