Dropout Layer and some extra questions!

Course Q&A TensorFlow Developer Professional Natural Language Processing in TensorFlow
1

Hi,
I have couple of questions:

  1. I am using dropout layer in assignment W3. does it matter where in the model I add the dropout layer? does the dropout layer applies to the previous layer or on the all layers in the model ? if it applies only on the previous layer, should I apply multiple dropout layer after each layer to avoid overfitting ?
  2. I noticed that sometimes the accuracy of cross_val increases or stays constant but the loss values also increases ( I know it shows the overfitting). I know higher loss values shows higher error between prediction and actual value, then how can I have higher error and constant or better accuracy in model ? I am a bit confuse regarding this matter.
  3. In example notebooks we used different models like Conv, Lstm, … methods separately. can I combine the or they don’t work together ? like Can I have conv layer with lstm layer in the same model ?
  4. why the accuracy calculated with model.evaluate(x,y) gives difference accuracy compare to the accuracy we calculate when fitting the model. my assumption was the last epoch accuracy on cross_val should be the same as model.evaluate()
    Thanks,
2
  1. A dropout layer:
    a. Can be applied anywhere in the model. I’ve never applied dropout to the output layer.
    b. Zeros out inputs to the layer that follows it.
    c. Of count 1 is sufficient between layers.
  2. When using dropout, you have to train the model longer for layers to become robust to input from dropout layer. So, the loss curves will bounce around. If this is not your architecture, please explain the details of the setup.
  3. You can combine conv (eg. Conv1d) and lstm in the same model. Since a conv layer is responsible for generating smaller dimensions of the input i.e. fewer timesteps in the case of sequence problems, consider using conv before lstm.
  4. This will happen when the batch_size parameter to model.fit is different from model.evaluate. Is this your case?
3

So, if this is the code , then does the Dropout only applies to Conv1D? is this a correct place to apply ? I have a feeling that better place to apply is before Dense layers, am I right ? also If I had two dense layers and I want to apply dropout to each dense layer, should I write the dropout layers separately before each dense layer?

tf.keras.layers.Embedding(vocab_size+1, embedding_dim, input_length=maxlen, weights=[embeddings_matrix], trainable=False),
tf.keras.layers.Dropout(0.3),
tf.keras.layers.Conv1D(64,5,activation=‘relu’),
tf.keras.layers.GlobalMaxPooling1D(),
tf.keras.layers.Dense(24,activation=‘relu’),
tf.keras.layers.Dense(1,activation=‘sigmoid’)

4

In your code, dropout is applied to input of tf.keras.layers.Conv1D.
There can be multiple dropout layers in a NN. Have a dropout layer before the layer whose input you want influenced.
Where you place a dropout layer and the level of dropout are design choices you’ll have to make based on experimentation.

Note: If the code you’ve shared belongs to an assignment, remove it please.