Course_5_Encoder layer

Course Q&A Deep Learning Specialization Sequence Models
1

I tried this section so many times, The hints given for this section is not enough. Please support me to identify the error and finish this part.

START CODE HERE

    # calculate self-attention using mha(~1 line). Dropout will be applied during training
    attn_output = self.mha(x, x, x, mask)
    attn_output = self.dropout_ffn(attn_output, training=training) # Self attention (batch_size, input_seq_len, fully_connected_dim)
    
    # apply layer normalization on sum of the input and the attention output to get the  
    # output of the multi-head attention layer (~1 line)
    out1 = self.layernorm1(x + attn_output)   # (batch_size, input_seq_len, fully_connected_dim)

    # pass the output of the multi-head attention layer through a ffn (~1 line)
    ffn_output = self.ffn(out1)  # (batch_size, input_seq_len, fully_connected_dim)
    
    # apply dropout layer to ffn output during training (~1 line)
    ffn_output = self.dropout_ffn(ffn_output, training=training)
    
    # apply layer normalization on sum of the output from multi-head attention and ffn output to get the
    # output of the encoder layer (~1 line)
    encoder_layer_out = self.layernorm2(out1 + ffn_output)  # (batch_size, input_seq_len, fully_connected_dim)
    # END CODE HERE

AssertionError Traceback (most recent call last)
in
1 # UNIT TEST
----> 2 EncoderLayer_test(EncoderLayer)

~/work/W4A1/public_tests.py in EncoderLayer_test(target)
92 [[ 0.23017104, -0.98100424, -0.78707516, 1.5379084 ],
93 [-1.2280797 , 0.76477575, -0.7169283 , 1.1802323 ],
—> 94 [ 0.14880152, -0.48318022, -1.1908402 , 1.5252188 ]]), “Wrong values when training=True”
95
96 encoded = encoder_layer1(q, False, np.array([[1, 1, 0]]))

AssertionError: Wrong values when training=True

1 Like
2

There is no dropout layer after self.mha().

3

That’s the only problem I see in your code.