C5 W4 A1: AssertionError: Wrong masked weights

[Aman]

Hi, I have tried all the solutions applied in the two threads link1 and link2 but none worked.
to be specific I did following:

1. applied -1e9 to the mask before adding up to scaled tensor.
2. used tf.keras.activations.softmax(scaled_attention_logits, axis=-1)
3. used tf.cast(tf.shape(k)[-1], tf.float32) to get dk
4. used tf.matmul(q, k, transpose_b=True)
   but I am still getting error: AssertionError: Wrong masked weights

I also refreshed my notebook but same issue happened again.
Since I can’t post my full code so I am posting parts only. ( I can send my code in message if you want to have a look at it)

[Ihme11]

Reminder : … . Multiply (1. - mask) by -1e9 before applying the softmax.
Adding the wrong quantity to the scaled_attention_logits might be the problem.

[Aman]

Thanks @Ihme11 ,
This was the problems.

[jlecornu]

@Ihme11 - Hi I have just hit this same issue.

I thought I had already applied this step in the below snippet:

# add the mask to the scaled tensor.
    if mask is not None:
        scaled_attention_logits += (1. - mask * -1.0e9)

Is this in the wrong place?

[ans]

operator precedence …

[MiNeves]

Ahaha lost about 1h30 because of overlooking operator precedence, thanks ans

[Mohamed_Nooh13]

i have same issues how to over come it?