Meaning of the Dense Layer

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

Hi. The implementation of the Dense Layer was introduced in C4W1 in the ConvNet Application. I am not too sure what the Dense Layer is and why is this used from the Keras documentation. Can someone provide an introduction to it.

2

Hi, @shamus!

As you just said, the keras documentation explains dense layers as:

output = activation(dot(input, kernel) + bias)

Let me break it down:

  • First, the input tensor, is multiplied by the kernel, which refers to the vector of neurons of the dense layer. Note that the dimensions of both tensors have to match in order to do the dot product
  • After that, the bias vector is added to the previous results. The values (weights) of the kernel and bias vectors are the numeric values that are modified in each optimizer step.
  • Finally, the activation corresponds to an element-wise function that maps each output value to the activation function (ReLU, SELU, softmax, etc). This last step is essential for the neural network to approximate non-linear functions. Otherwise, it could only learn linear transformations of the input data.
3

What is a Dense Layer in Neural Network?
The dense layer is a neural network layer that is connected deeply, which means each neuron in the dense layer receives input from all neurons of its previous layer. In the background, the dense layer performs a matrix-vector multiplication. The values used in the matrix are actually parameters that can be trained and updated with the help of backpropagation.

Dense Layer from Keras

Keras provide dense layers through the following syntax:

tf.keras.layers.Dense(
    units,
    activation=None,
    use_bias=True,
    kernel_initializer="glorot_uniform",
    bias_initializer="zeros",
    kernel_regularizer=None,
    bias_regularizer=None,
    activity_regularizer=None,
    kernel_constraint=None,
    bias_constraint=None,
    **kwargs
)

How to Implement the Dense Layer?
A sequential model with a single dense layer.

import tensorflow
model = tensorflow.keras.models.Sequential()
model.add(tensorflow.keras.Input(shape=(16,)))
model.add(tensorflow.keras.layers.Dense(32, activation='relu'))
print(model.output_shape)
print(model.compute_output_signature)

I hope I answer your question. If you need more guidance let me know.