# C5\_W4\_A1\_Transformer\_Subclass\_v1: Why is look\_ahead mask used before the padding\_mask **URL:** \ **Category:** Sequence Models\ **Tags:** coursera-platform\ **Created:** [October 1, 2023, 5:34am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468 "2023-10-01T05:34:09Z")\ **Posts on this page:** 7\ **Page:** 1
**Author:** ![Ayush\_Nigade](https://avatars.discourse-cdn.com/v4/letter/a/74df32/32.png) [@Ayush\_Nigade](https://community.deeplearning.ai/u/Ayush_Nigade)\ **Post date:** [October 1, 2023, 5:34am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/1 "2023-10-01T05:34:09Z")
Just a theory question. There are two multiheaded self-attention layers. Why is look\_ahead mask used before the padding\_mask? ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [October 1, 2023, 6:19am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/2 "2023-10-01T06:19:18Z")
Please explain why the markdown notes in sections 2.1 and 2.2 don’t answer your question. ---
**Author:** ![Ayush\_Nigade](https://avatars.discourse-cdn.com/v4/letter/a/74df32/32.png) [@Ayush\_Nigade](https://community.deeplearning.ai/u/Ayush_Nigade)\ **Post date:** [October 1, 2023, 7:10am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/3 "2023-10-01T07:10:23Z")
Ok this is my understanding from 2.1-2.2: This is why doing padding before look\_ahead is not useful according to my understanding: Using look\_ahead on a padded input isn’t useful because we would have lost some information if we apply look ahead on a padded input. This is because padding either cuts down sentences or adds 0’s so that all sentences are the same length. So in look ahead, we could be predicting a 0 (because it got padded with 0’s) or predicting a part of the sentence we wouldn’t have wanted to (because it got cut down). Let me know if I got something wrong. Thanks! ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [October 1, 2023, 7:47am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/4 "2023-10-01T07:47:32Z")
Look ahead is meant to treat tokens in the future as unknown when performing attention. Consider the following text translation task from english to french: - Input: Complete english text as input. - Output: French translation. Since we have access to the entire input sentence at the encoder, we can perform self attention across all text tokens. This is why we don’t have any look ahead mask at the encoder. Decoder on the other hand is supposed to generate the french text with only [START TOKEN] marker signifying the previous input. When training this model, we have pairs of English and French sentences as input / output pairs. Keep in mind that while we have the full output, at each step in the output, we can perform attention only across the tokens we’ve encountered / generated so far. As a result, we want to remove references to all tokens in future timesteps when we have data up to the current timestep. This is the purpose of the look ahead mask. Padding mask is to ensure that softmax probabilities for padded positions are very low. This ensures that we pick a valid token as the translation at each timestep. Please go through the markdown texts again with this information in mind. ---
**Author:** ![Ayush\_Nigade](https://avatars.discourse-cdn.com/v4/letter/a/74df32/32.png) [@Ayush\_Nigade](https://community.deeplearning.ai/u/Ayush_Nigade)\ **Post date:** [October 1, 2023, 11:34pm UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/5 "2023-10-01T23:34:51Z")
Thank you! ---
**Author:** ![Oleksandra\_Sopova](https://avatars.discourse-cdn.com/v4/letter/o/e99b99/32.png) [@Oleksandra\_Sopova](https://community.deeplearning.ai/u/Oleksandra_Sopova)\ **Post date:** [October 31, 2023, 11:53pm UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/6 "2023-10-31T23:53:02Z")
what markdown notes in sections 2.1 and 2.2 are you referring to? I am seeing only the pdfs ([DLS Course 5: Lecture Notes](https://community.deeplearning.ai/t/dls-course-5-lecture-notes/11884)). Could you please share the link to these markdown notes in sections 2.1 and 2.2? ---
**Author:** ![balaji.ambresh](https://sea1.discourse-cdn.com/dlai/user_avatar/community.deeplearning.ai/balaji.ambresh/32/128697_2.png) [@balaji.ambresh](https://community.deeplearning.ai/u/balaji.ambresh)\ **Post date:** [November 1, 2023, 4:30am UTC](https://community.deeplearning.ai/t/c5-w4-a1-transformer-subclass-v1-why-is-look-ahead-mask-used-before-the-padding-mask/459468/7 "2023-11-01T04:30:44Z")
I’m referring to the markdown text in the notebook (C5\_W4\_A1\_Transformer\_Subclass\_v1.ipynb). See this [link](https://www.coursera.org/learn/nlp-sequence-models/programming/roP5y/transformers-architecture-with-tensorflow).