[W4] Course 5, Exercise 5 - Encoder

Passed Exercise 1 to 4, but saw below error.
My implementation is nearlly the same as https://www.tensorflow.org/tutorials/text/transformer

(Solution code removed by staff)

AssertionError                            Traceback (most recent call last)
<ipython-input-147-2f05cf553abf> in <module>
     28     print("\033[92mAll tests passed")
---> 30 Encoder_test(Encoder)

<ipython-input-147-2f05cf553abf> in Encoder_test(target)
     24                         [[-0.3489219,   0.31335592, -1.3568854,   1.3924513 ],
     25                          [-0.08761203, -0.1680029,  -1.2742313,   1.5298463 ],
---> 26                          [ 0.2627198,  -1.6140151,   0.2212624 ,  1.130033  ]]]), "Wrong values"
     28     print("\033[92mAll tests passed")

AssertionError: Wrong values

Mmm it’s no easy task to understand what’s wrong in the call method of the Encoder just like this… The error only tells that the numerical output is different with respect to the expected one. Can you be more specific about how you implemented it? Can you (even partially) post it?

Here is my implementation

(Solution code removed by staff)

I figured it out.
the x value should be re-assigned each time.

Glad to hear you solved it, but yes, you should reassign the value of x every time, otherwise the output of an encoder layer is not used in the following one, but rather all the layers are working on the input tensor x… it should be something like

x = self.enc_layers[i](x, training, mask)

Following the TF guide on Transformers, I implemented the exact same code as @Kamel (except with reassigning x in the for loop), as you pointed out @edoale. However, I get an an exception thrown (excerpted):

—> 61 x += self.pos_encoding[:, :seq_len, :]
InvalidArgumentError: Incompatible shapes: [2,3,4] vs. [1,3,16] [Op:AddV2]

Indeed the shape of self.pos_encoding[:, :seq_len, :] is (1, 3, 16). The shape of x after passing it through the embedding layer and multiplying the result by the sqrt of the embedding dimension is (2, 3, 4). The dimensions are, indeed, not compatible, so I am doing something wrong.

Can anyone point me in the right direction?

hi @marcus-waldman , I just checked and the correct shapes should be (2, 3, 4) for x and (1, 3, 4) for self.pos_encoding[:, :seq_len, :] (which will be eventually be broadcasted into (2, 3, 4) automatically).
Therefore, if your shape for it is (1, 3, 16), I would say that the issue arises when calling the function positional_encoding() in Exercise 2, something is wrong in the last dimension of its output, try to check it out :slight_smile:

1 Like

Yep, @edoale! I hard coded the dimension for some reason earlier in the assignment, rather than passing it on as an argument! Good catch and thanks so much!

Hi, I am unable to figure out what is wrong with my code for encoder class.
Here is my code for encoder class. Unit test says “wrong values.”

(Solution code removed by staff)

Issue is resolved by using + instead of sum () in encoderlayer

1 Like

same issue solved by this method, thank you!

Submitting the assignment the grader complained: Can’t compile the student’s code. Error: InvalidArgumentError(). The cell included the “unit test” call to Encoder_test(Encoder).

All the tests, including this one, displayed “All tests passed”.

========== UPDATE ===========
I resubmitted and the grader “passed” the assignment.
Moving on to the optional notebooks.