# C4W2 Assignment Exercise 5 - next\_word **URL:** \ **Category:** NLP with Attention Models\ **Tags:** week-module-2\ **Created:** [April 9, 2024, 6:17pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313 "2024-04-09T18:17:52Z")\ **Posts on this page:** 16\ **Page:** 1
**Author:** ![Ardavan\_Sherafat](https://avatars.discourse-cdn.com/v4/letter/a/ecd19e/32.png) [@Ardavan\_Sherafat](https://community.deeplearning.ai/u/Ardavan_Sherafat)\ **Post date:** [April 9, 2024, 6:17pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/1 "2024-04-09T18:17:52Z")
I have completed the Exercise 5 - next\_word of the assignment and all the test cases pass, I also get the full grade for the assignment, but the model does not output anything. Not sure if I am doing something wrong that none of the test cases are capturing it or there is problem with the notebook: Predicted token: Predicted word: ##### **Expected Output** Predicted token: [[14859]] Predicted word: masses Training set example: [SOS] amanda: i baked cookies. do you want some? jerry: sure! amanda: i’ll bring you tomorrow 🙂 [EOS] Human written summary: [SOS] amanda baked cookies and will bring jerry some tomorrow. [EOS] Model written summary: [SOS] ---
**Author:** ![jyadav202](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jyadav202/32/162439_2.png) [@jyadav202](https://community.deeplearning.ai/u/jyadav202)\ **Post date:** [April 10, 2024, 2:34am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/2 "2024-04-10T02:34:44Z")
Strange that unit test did not catch the error. Please send me a copy of your code on my private chat. I give it a look. ---
**Author:** ![jyadav202](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jyadav202/32/162439_2.png) [@jyadav202](https://community.deeplearning.ai/u/jyadav202)\ **Post date:** [April 11, 2024, 4:06am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/3 "2024-04-11T04:06:16Z")
The issue was found in the `output` target variable manipulation. ---
**Author:** ![scheine](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@scheine](https://community.deeplearning.ai/u/scheine)\ **Post date:** [April 12, 2024, 8:03pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/4 "2024-04-12T20:03:46Z")
````auto ---> 67 mult_attn_out2, attn_weights_block2 = self.mha2(Q1, enc_output, enc_output, padding_mask, return_attention_scores=True) 69 # apply layer normalization (layernorm2) to the sum of the attention output and the output of the first block (~1 line) 70 mult_attn_out2 = self.layernorm2(mult_attn_out2 + Q1) InvalidArgumentError: Exception encountered when calling layer 'softmax_58' (type Softmax). {{function_node __wrapped__ AddV2_device_/job:localhost/replica:0/task:0/device:GPU:0}} required broadcastable shapes [Op:AddV2] name: Call arguments received by layer 'softmax_58' (type Softmax): • inputs=tf.Tensor(shape=(1, 2, 2, 150), dtype=float32) • mask=tf.Tensor(shape=(1, 1, 1, 2), dtype=float32)``` got an error at next_word, but passed all previous unit tests, not sure how to proceed. ```` ---
**Author:** ![jyadav202](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jyadav202/32/162439_2.png) [@jyadav202](https://community.deeplearning.ai/u/jyadav202)\ **Post date:** [April 13, 2024, 5:21am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/5 "2024-04-13T05:21:02Z")
@scheine I cant be very sure without looking at the code. But have a look at `look_ahead_mask` or `dec_padding_mask` in next\_word function. That might be a point of failure. Note that `create_look_ahead_mask` requires `sequence_length` as argument unlike `create_padding_mask`. If that does not solve it, send me your code. ---
**Author:** ![scheine](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@scheine](https://community.deeplearning.ai/u/scheine)\ **Post date:** [April 13, 2024, 11:35am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/6 "2024-04-13T11:35:31Z")
Thanks, I fixed the shape issue after updating code in create\_padding\_mask. ---
**Author:** ![jyadav202](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jyadav202/32/162439_2.png) [@jyadav202](https://community.deeplearning.ai/u/jyadav202)\ **Post date:** [April 14, 2024, 4:45am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/7 "2024-04-14T04:45:10Z")
@scheine I am glad. I would also, request you to remove the solution code from your reply as sharing solutions is against our community guidelines. Regards, ---
**Author:** ![scheine](https://avatars.discourse-cdn.com/v4/letter/s/50afbb/32.png) [@scheine](https://community.deeplearning.ai/u/scheine)\ **Post date:** [April 14, 2024, 11:07am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/8 "2024-04-14T11:07:20Z")
thanks, updated. ---
**Author:** ![Lu\_Huang](https://avatars.discourse-cdn.com/v4/letter/l/a4c791/32.png) [@Lu\_Huang](https://community.deeplearning.ai/u/Lu_Huang)\ **Post date:** [April 23, 2024, 9:34pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/9 "2024-04-23T21:34:24Z")
@jyadav202 I have the same error, all unit test passed. Can you give some hint on what is wrong? mult\_attn\_out2, attn\_weights\_block2 = self.mha2(Q1, enc\_output, enc\_output, attention\_mask = padding\_mask, return\_attention\_scores=True) 69 # apply layer normalization (layernorm2) to the sum of the attention output and the output of the first block (~1 line) 70 mult\_attn\_out2 = self.layernorm2(Q1 + mult\_attn\_out2) InvalidArgumentError: Exception encountered when calling layer ‘softmax\_92’ (type Softmax). {{function\_node \__wrapped\_\_AddV2\_device_/job:localhost/replica:0/task:0/device:GPU:0}} required broadcastable shapes [Op:AddV2] name: Call arguments received by layer ‘softmax\_92’ (type Softmax): • inputs=tf.Tensor(shape=(1, 2, 2, 150), dtype=float32) • mask=tf.Tensor(shape=(1, 1, 1, 2), dtype=float32) ---
**Author:** ![TMosh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/tmosh/32/9321_2.png) [@TMosh](https://community.deeplearning.ai/u/TMosh)\ **Post date:** [April 23, 2024, 9:47pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/10 "2024-04-23T21:47:53Z")
I’m not a mentor for this course, but the assignment seems similar to one from DLS Course 5 Week 4. I would guess the issue is that your Q1 is not the correct shape. ---
**Author:** ![jyadav202](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/jyadav202/32/162439_2.png) [@jyadav202](https://community.deeplearning.ai/u/jyadav202)\ **Post date:** [April 24, 2024, 3:59am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/11 "2024-04-24T03:59:39Z")
Hi! Yes, please recheck Q1. Also whether your output matched the “Expected Output” . If the error still persists, send me a copy of your assignment, I will take a look. ---
**Author:** ![John\_Murphy1](https://avatars.discourse-cdn.com/v4/letter/j/eb8c5e/32.png) [@John\_Murphy1](https://community.deeplearning.ai/u/John_Murphy1)\ **Post date:** [May 2, 2024, 6:21pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/12 "2024-05-02T18:21:25Z")
Hi, My problem is similar, except my error is “softmax\_404”. My grade function next\_word gives the expected output [[14859]] and masses and it passes the tests and as a part of this: w2\_unittest.test\_next\_word(next\_word, transformer, encoder\_input, output) " All tests passed!" But when I try to summarize a sentence, this line: summarize(transformer, document[training\_set\_example]) references this code from DecoderLayer.call: mult\_attn\_out2, attn\_weights\_block2 = self.mha2(… ) which throws the following error: InvalidArgumentError: Exception encountered when calling layer ‘softmax\_404’ (type Softmax). {{function\_node \__wrapped\_\_AddV2\_device_/job:localhost/replica:0/task:0/device:GPU:0}} required broadcastable shapes [Op:AddV2] name: Call arguments received by layer ‘softmax\_404’ (type Softmax): • inputs=tf.Tensor(shape=(1, 2, 2, 150), dtype=float32) • mask=tf.Tensor(shape=(1, 1, 1, 2), dtype=float32) I printed these from next\_word: enc\_padding\_mask: tf.Tensor( [[[1. 1. 1. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0. 0.]]], shape=(1, 1, 150), dtype=float32) look\_ahead\_mask: tf.Tensor([[[1.]]], shape=(1, 1, 1), dtype=float32) dec\_padding\_mask: tf.Tensor([[[1.]]], shape=(1, 1, 1), dtype=float32) output: tf.Tensor([[7]], shape=(1, 1), dtype=int32) Predicted token: [[14859]] Predicted word: masses Any suggestions would be greatly appreciated. Thanks. John ---
**Author:** ![bsatom](https://avatars.discourse-cdn.com/v4/letter/b/8c91f0/32.png) [@bsatom](https://community.deeplearning.ai/u/bsatom)\ **Post date:** [May 4, 2024, 8:20pm UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/13 "2024-05-04T20:20:28Z")
Using dec\_padding\_mask = create\_padding\_mask(encoder\_input) rather than dec\_padding\_mask = create\_padding\_mask(output) and it works. My understanding is that the `dec_padding_mask` is based on the `encoder_input`, as it’s intended to mask out padding tokens in the original input sequence which flows through the encoder and becomes the `enc_output`. The padding applied to the second MHA input, which receives the encoder’s output. However, this padding should be identical to the padding applied to the original encoder input Unfortunately, the term “output” is also used as the target name, specifically referring to the summarized sentences. This usage conflicts with the “output” of the encoder used for the second MHA input. The first MHA receives the target (variable called output) and therefore should use the causal, look ahead padding, applied on the target (output). Is that correct? ---
**Author:** ![mats](https://avatars.discourse-cdn.com/v4/letter/m/b4bc9f/32.png) [@mats](https://community.deeplearning.ai/u/mats)\ **Post date:** [August 21, 2024, 9:41am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/14 "2024-08-21T09:41:30Z")
Could you say a bit more? In what place is the bug in the code. I had the same issue, I corrected input to look\_ahead\_mask = create\_look\_ahead\_mask(…) and it seems to be working better. However, I’m not sure if that’s the right solution. I can share the code if you wanted. ---
**Author:** ![Deepti\_Prasad](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/deepti_prasad/32/165831_2.png) [@Deepti\_Prasad](https://community.deeplearning.ai/u/Deepti_Prasad)\ **Post date:** [August 30, 2024, 6:15am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/15 "2024-08-30T06:15:13Z")
@mats create a new thread whenever you any issue, your comments don’t get notified or get missed when you comment on old threads regards DP ---
**Author:** ![Rahul\_Ohlan](https://avatars.discourse-cdn.com/v4/letter/r/7ab992/32.png) [@Rahul\_Ohlan](https://community.deeplearning.ai/u/Rahul_Ohlan)\ **Post date:** [January 6, 2025, 4:12am UTC](https://community.deeplearning.ai/t/c4w2-assignment-exercise-5-next-word/607313/16 "2025-01-06T04:12:23Z")
this worked for me, thanks a lot! – yeah it makes sense, we gotta mask out the padding tokens from original input sequence