C5-Week 4 UNQ_C8 : Programming Assignment - Transformer - Assertion Error

Course Q&A Deep Learning Specialization Sequence Models
1

This the last step in the assignment, wherein all the functions created earlier are called. I received success for all the previous graded functions. However for the Transformer function in the last step I get assertion Error as follows:
AssertionError Traceback (most recent call last)
in
1 # UNIT TEST
----> 2 Transformer_test(Transformer, create_look_ahead_mask, create_padding_mask)

~/work/W4A1/public_tests.py in Transformer_test(target, create_look_ahead_mask, create_padding_mask)
286 assert np.allclose(translation[0, 0, 0:8],
287 [0.017416516, 0.030932948, 0.024302809, 0.01997807,
→ 288 0.014861834, 0.034384135, 0.054789476, 0.032087505]), “Wrong values in translation”
289
290 keys = list(weights.keys())

AssertionError: Wrong values in translation

Query : The last step mentions to pass decoder output through linear and softmax layer and mentions approx 2 lines of code. However the function defined is Dense layer with activation as softmax. Is there an error in the comment stating 2 lines instead of 1?

I am not able to figure out why I am receiving assertion error. Please assist.

2 Likes
2

You do not need two lines for the last step.

The final Transformer_test can catch errors in either the Encoder or Decoder which were not previously detected.

3

Which step in encoder or decoder could be an issue and how can I figure out where the error is (decoder or encoder). As if I change any input in either Encoder or Decoder it gives an error.

4

I did some further investigation and observe that in function scaled_dot_product_attention, attention weights instruction reads “softmax is normalized on the last axis (seq_len_k) so that the scores add up to 1.”. However when I output the attention weights and sum up the columns in the row, the sum is 1.00000006, but not exactly 1.
However the execution mentions all test passed.
Inputs to softmax is scaled_attention_logits and axis=-1.

5

Well, we are doing floating point here, so rounding errors are always a possibility. That would be a relatively high error with 64 bit floats, but it’s perfectly reasonable for 32 bit floats. You can take a look at the IEEE 754 standard to understand more details, but the resolution in the mantissa of a binary32 float is on the order of 10^{-7}. For binary64, it’s on the order of 10^{-15}.

Because of rounding behavior, they can’t use exact comparisons, but there are numpy primitives that are specifically designed to handle this situation. Have a look at some of the test cases in public_tests.py and you see that they typically use np.isclose or np.allclose to check your answers against the expected values.

1 Like
6

This ~2 lines made me crazy! I added a linear layer and nothing works. When I see this message and know we do not need 2 lines, I just deleted the linear layer and it worked.

I would suggest changing that ~2 lines to ~1 line. Thanks