C3_W1_UNQ_C4 Dense Layer

Hi everyone,

While implementing the Dense Layer class, I produced the expected weights and forward function output in the first test.

However, in my second test, it failed to return the forward output with the following error message:
TypeError: dot_general requires contracting dimensions to have the same shape, got [4] and [3].

In the unittest, I got the same error message.

I suspect it has something to do with dense or w, but cannot identify the solution. Help appreciated!

Hi @Artie_Lam

I’m guessing the most probable place for your mistake is weights initialization in init_weights_and_state function.

You have two lines of code in this function. The first one is pretty easy:

input_shape = None

Here you just have to assign the value of input_signature attribute .shape.

Second line is a bit trickier:

# Generate the weight matrix from a normal distribution,
# and standard deviation of 'stdev'
w = None * trax.fastmath.random.normal(key = None, shape = (None, None))

The first None is the value of standard deviation for initializing weights that has default value of 0.1 and is assigned when initializing this class in self._init_stdev.

The second None is random key that fastmath.random.normal function requires. Naturally this would be the argument random_key.

The third and fourth Nones is where you probably made a mistake. The shape of the weight matrix for Dense layer should be (“the last dimension of inputs” x “the last dimension of outputs”). For example, if your X (inputs) for Dense layer is of shape (4, 5) and you want your output to be the shape of (4, 10), then the w should have a shape of (5, 10), because: (4, 5) dot (5, 10)(4, 10).
So this is all to explain that the shape= should be (“the last dimension of input_shape x “n_units of the output dimension”). You can get the last dimension with [-1] and n_units is saved in the class attribute self._n_units.

Cheers

2 Likes