[Week4] programming assignment, EncoderLayer duplicated self.layernorm and self.dropout?

I am new to tensorflow. the code gives two norm layers (with same settings) and two dropout layers ( with same settings). is there any reason to do this? Thanks!

    self.layernorm1 = LayerNormalization(epsilon=layernorm_eps)
    self.layernorm2 = LayerNormalization(epsilon=layernorm_eps)

    self.dropout1 = Dropout(dropout_rate)
    self.dropout2 = Dropout(dropout_rate)

I get the EncoderLayer exercise passed. I tried reuse layernorm1 and dropout1. it seems fine. so…maybe it is just for readability?

Hi Awesomele,

The two pairs of LayerNormalization and Dropout layers’ instances MUST be separated. The first pair are for multi-head attention layer, the 2nd pair are for feed-forward network. Both have their own trainable weights. In UNIT TEST cell, these trainable weights have the same default initial values and haven’t learned (updated) yet. That’s the reason why you can pass the exercise even using the same layers’ instances.

2 Likes