Transformer NB get_angles equations broken?

Course Q&A Deep Learning Specialization Sequence Models
1

Hi I don’t understand how on earth we are supposed to program the get_angles() function in the Transformer Notebook because it literally says:
"Hint […] If k = [0, 1, 2, 3, 4, 5] , then, i must be i = [0, 0, 1, 1, 2, 2]", which means that ‘i’ must be 0 twice. Then plug that into the equation “theta(pos, i, d) = pos/(10000*(2i/d))” (also quoted verbatim from instructions). Then you get infinities (or NaNs) at the 0s which causes it to fail the asserts in the unit tests.

Am I missing something here?

2

The instructions are very confusing.

The “hint” is intended to tell you that the ‘i’ values come from (edited) floor division of the k values.

So that converts in code into “i = k // 2”.

Then to compute “angles”, you use pos, np.power(), a constant of 10000, 2*i, and np.float32(d).

You’re only implementing this equation:
image
… where theta is the “angles” variable.

1 Like
3

Actually your implementation is wrong, (2i/d) is an exponent of 10000, not a product.

2 Likes
4

Oh I see that appears to be the root of the problem, I can tell now the (2i/d) is ever so slightly elevated but not obviously enough for me to realize it was a power.

image

P.S. ‘//’ is just truncated division not modulo (modulo is ‘%’)

Anyhow thanks for your answer!! No idea how long that would’ve taken me to figure out lol

5

Thanks for pointing out my incorrect term, “modulo” was the first word that popped into my head. I’ll update my reply.

1 Like