Ungraded Lab 1: Doubts in our_hash_vectors

Course Q&A Natural Language Processing NLP with Attention Models
1

Hey Guys,
I have a couple of doubts in the markdown/implementation of our_hash_vectors.

Step 1 (Markdown)

Each vector will be hashed into a hash table and into rot_size//2 buckets. We use rot_size//2 to reduce computation.

  • I am a little confused about the highlighted markdown, since in step 3, it is mentioned that n_buckets = rotsize. Moreover, in the function’s implementation too, we have used rot_size = n_buckets. Now, the below statement is also mentioned in step 1.
  • So, is the above markdown incorrect, or like, is it only temporarily true, till Step 3 to be more precise.

Later in the routine we will form the negative rotations with a simple negation and concatenate to get a full rot_size number of rotations.

Step 2 (Code)

rotated_vecs = np.einsum(“tf,fhb->htb”, vecs, random_rotations)

Last, I am unable to find the meaning of the subscripts used in the above statement. Can someone provide the link which describes these subscripts, or elaborate these subscripts? I was unable to find the link, that’s why the doubt.

Cheers,
Elemento

2

Hi @Elemento

or like, is it only temporarily true, till Step 3 to be more precise.

Yes, this is the case.

For example, (in case of n_hashes=3, n_buckets = 4, seq_len = 16) at the end of Step 2, rotated_vecs could be rot_size//2 for each hash, like :
image

and after Step 3 would concatenate each:

image
image471×1060 25.9 KB

If I remember correctly, jax uses the same subscripts as numpy (numpy.einsum — NumPy v1.26 Manual). It is just matrix multiplication with different shapes 2d and 3d.

Cheers

3

Hey @arvyzukai,
Thanks a lot for your quick response. I am still a little confused in my second query, since even in numpy, I am unable to find an exhaustive list of subscripts, which can tell me, the meaning of each character. Can you please help me with finding that list?

Cheers,
Elemento

4

No problem @Elemento

I cannot explain all the variants the np.einsum could be used (there are a lot of examples in the documentation I posted previously), but I can try to explain the one used in the Lab.

rotated_vecs = np.einsum(“tf,fhb->htb”, vecs, random_rotations)

The best way for me is to explain with an example:

For example, if the vecs array is of shape (16 x 5, here t=16, f=5) and random_rotations is of shape (5 x 3 x 2, here f=5, h=3, b=2), then the result is matrix multiplication (or dot product) of shape (3 x 16 x 2, or htb). The matrix multiplication is done on the f (5) dimension. So the rotated_vecs has the shape of (3 x 16 x 2).

I hope that makes it somewhat clearer? If not, feel free to ask. :slight_smile:

Cheers

5

Hey @arvyzukai,
Thanks a lot for the explanation. Just one last small query. Do these tf and fhb subscripts are specific to denote the dimensions, or can we replace them with any alphabet. For instance, can we use ab, bcd -> cad, and will it produce the same results?

Cheers

6

Hey @Elemento

Yes, that would be equivalent to tf,fhb->htb

Cheers