Dropout Layer order - impact on the result?

Course Q&A TensorFlow Developer Professional Natural Language Processing in TensorFlow
1

Hi,

I would like to know if the dropout layer can considerably impact the result depending if it’s at the beginning or at the end ?

For example:

    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=maxlen, weights=[embeddings_matrix], trainable=False), 
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Conv1D(64, 5, activation = 'relu'),
        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units=64)),
        tf.keras.layers.Dense(1, activation="sigmoid")

or

    tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=maxlen, weights=[embeddings_matrix], trainable=False), 
        tf.keras.layers.Conv1D(64, 5, activation = 'relu'),
        tf.keras.layers.Bidirectional(tf.keras.layers.LSTM(units=64)),
        tf.keras.layers.Dropout(0.2),
        tf.keras.layers.Dense(1, activation="sigmoid")

If yes, why ? What is the optimal way ? Thank you for your insights.

1 Like
2

Hello there,

You should not post code solutions in here, but I guess this is just mock.

Yes the dropout layer should be used when there are many neurons, so some of them are shut down and let the flow go through the rest of the neurons. Think of it like a stream of water going through many channels, if there are many pipes there want be much water in either of the channels (in our case the weights will become very small and have little effect to transforming the input to a desired output). So, a dropout in the beginning where there are just a few neurons doesn’t make sense.

1 Like
3

Yes it’s just a mock because it’s not working as expected.
Ok got it ! It makes sens. So better to put this layer at the end like the second exemple.

Thank you @gent.spah !

1 Like
4

Very good u added a code snippet here for educational purpose!

However, I noticed a difference here because there is a description in assignment:

  • The last two layers should be Dense layers.
    I was wondering if you made a mode with one Dense layer intentionally or something else?

Cheers,