Hello,
I have some trouble with exercise 4, what should I do now, and how to fix this?
Also, in exercise 5, I have another trouble, how should I do to fix it
Many thanks
Dung
Please click my name and message your notebook as an attachment.
You’ve already applied dropout to self.mha (see EncoderLayer.__init__). So, there’s no need to apply dropout to the output of self attention.
Hint: Keep in mind that a layer is used exactly once. So, don’t use the dropout_fn twice (once after mha and the other place which is right place).
1 Like
Somehow this still doesn’t work for me. IIUC in self.mha I need to pass both the mask, and the training parameter - to get dropout, and then pass the training parameter in self.dropout_ffn layer to get dropout there as well. Also, the self.mha should get x as both Q, K and V, right? Other than that everything else seems straight-forward. But I still get the “Wrong values when training=True” message, so probably I’m missing something
You pass the dropout parameter to MultiHeadAttention.__init__ The output of the __call__ invocation is the attention with dropout applied. So, there’s no need to apply dropout to the output of this layer.
Apply dropout as indicated here in the starter code:
# apply dropout layer to ffn output during training (~1 line)
ffn_output = None
Thanks a lot for helping me. I have solved this problem and finished the course.
Thanks. This is exactly what I did - I just passed training=training in the call function of the self.mha so the dropout will be applied in training, and then added the dropout exactly where you indicated. Probably I’m missing something trivial…
Oops, found my issue - forgot to add the second skip connection… thanks!