Dinosaurus_Island_Character_level_language_model - Exercise 4 Model

trying to debug exercise 4.

I suspect there is something wrong with theY.

I have set Y to equal to elements fromX starting from the first element, so this should take care of the link between y and x highlighted in yellow. but the very last element (green) is not added to Y.

I thought at first that it would be resolved by appending Y with last_dino_name as a list addition, but this does not work either. Not sure what last_dino_name does in the code

Do I need to forward propagate the last element of x to get the y and add it to the current Y value? Or is it the character to represent end of the word?

The code expects a 1 line solution, so not sure if this is what is expected in this exercise.

image

j 0 range range(0, 22001)
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-102-725c093d6b91> in <module>
----> 1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
      3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

<ipython-input-101-10a8c686571a> in model(data_x, ix_to_char, char_to_ix, num_iterations, n_a, dino_names, vocab_size, verbose)
     64         # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters
     65         # Choose a learning rate of 0.01
---> 66         curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)
     67 
     68         ### END CODE HERE ###

<ipython-input-93-fca0aba78113> in optimize(X, Y, a_prev, parameters, learning_rate)
     32 
     33     # Forward propagate through time (≈1 line)
---> 34     loss, cache = rnn_forward(X, Y, a_prev, parameters)
     35 
     36     # Backpropagate through time (≈1 line)

~/work/W1A2/utils.py in rnn_forward(X, Y, a0, parameters, vocab_size)
    100 
    101         # Update the loss by substracting the cross-entropy term of this time-step from it.
--> 102         loss -= np.log(y_hat[t][Y[t],0])
    103 
    104     cache = (y_hat, a, x)

IndexError: list index out of range
  1. Loop change

I have adjusted the optimisation loop to divide number of iterations by the length of examples using module operator.
Not clear why the hint suggest adding 1 in the numerator.
here, we have 1536 examples and 22001 iterations. Simple modulor operator allows to loop through all examples.

  1. Y
    set Y to equal now elements of X starting from 1st element onwards and added a newline character through list addition signalling end of the word.

but the code still does not work.

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

AssertionError: Wrong expected output
j 0 range range(0, 22001)
examples 1536
num_iterations 22001
idx 0
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 0]
j =  0 idx =  0
single_example = aachenosaurus
single_example_chars ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
 X =  [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21] 
 Y =        [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 0] 

Iteration: 0, Loss: 23.090636

Nkzxwtdmeqoeyhsqwasjjjvu
Kneb
Kzxwtdmeqoeyhsqwasjjjvu
Neb
Zxwtdmeqoeyhsqwasjjjvu
Eb
Xwtdmeqoeyhsqwasjjjvu


j 1 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1
single_example: 	 aardonyx
single_example_chars: 	 ['a', 'a', 'r', 'd', 'o', 'n', 'y', 'x']
single_example_ix: 	 [1, 1, 18, 4, 15, 14, 25, 24]
X: 	 [None, 1, 1, 18, 4, 15, 14, 25]
Y: 	 [1, 1, 18, 4, 15, 14, 25, 0]
j 2 range range(0, 22001)
examples 1536
num_iterations 22001
idx 2
single_example: 	 abdallahsaurus
single_example_chars: 	 ['a', 'b', 'd', 'a', 'l', 'l', 'a', 'h', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21]
Y: 	 [1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21, 0]
examples 1536
num_iterations 22001
idx 495
single_example: 	 eustreptospondylus
single_example_chars: 	 ['e', 'u', 's', 't', 'r', 'e', 'p', 't', 'o', 's', 'p', 'o', 'n', 'd', 'y', 'l', 'u', 's']
single_example_ix: 	 [5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21, 19]
X: 	 [None, 5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21]
Y: 	 [5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21, 0]
j 22000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 496
single_example: 	 fabrosaurus
single_example_chars: 	 ['f', 'a', 'b', 'r', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 6, 1, 2, 18, 15, 19, 1, 21, 18, 21]
Y: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 0]
Iteration: 22000, Loss: 20.333626

Loustreprosauru
Eule
Eurupelepodu
Localtregaru
Yurpengosauru
Cabdrope
Tromelosauru


---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"```

Here’s my output from running the training. Notice that you seem to be truncating X, when you convert single_example_ix to X. Compare to what I show below:

j =  0 idx =  0
single_example = turiasaurus
single_example_chars ['t', 'u', 'r', 'i', 'a', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19]
 X =  [None, 20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19] 
 Y =        [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19, 0] 

Iteration: 0, Loss: 23.087336

Nkzxwtdmfqoeyhsqwasjkjvu
Kneb
Kzxwtdmfqoeyhsqwasjkjvu
Neb
Zxwtdmfqoeyhsqwasjkjvu
Eb
Xwtdmfqoeyhsqwasjkjvu


j =  1535 idx =  1535
j =  1536 idx =  0
Iteration: 2000, Loss: 27.884160

Liusskeomnolxeros
Hmdaairus
Hytroligoraurus
Lecalosapaus
Xusicikoraurus
Abalpsamantisaurus
Tpraneronxeros


Iteration: 4000, Loss: 25.901815

Mivrosaurus
Inee
Ivtroplisaurus
Mbaaisaurus
Wusichisaurus
Cabaselachus
Toraperlethosdarenitochusthiamamumamaon


Iteration: 6000, Loss: 24.608779

Onwusceomosaurus
Lieeaerosaurus
Lxussaurus
Oma
Xusteonosaurus
Eeahosaurus
Toreonosaurus

@paulinpaloalto

Yes, i see, when selecting the corresponding value for the single_example_ix, i did not need to select all apart for the last element as the output was a list and not dictionary key followed by the value.

restarted kernel, reran the cell, still the result is different and is not accepted by the Autograder.

j 0 range range(0, 22001)
examples 1536
num_iterations 22001
idx 0
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
[1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19, 0]
j =  0 idx =  0
single_example = aachenosaurus
single_example_chars ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
 X =  [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19] 
 Y =        [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19, 0] 

Iteration: 0, Loss: 23.093932

Nkzxwtdmeqoeyhsqwasjjjvu
Kneb
Kzxwtdmeqoeyhsqwasjjjvu
Neb
Zxwtdmeqoeyhsqwasjjjvu
Eb
Xwtdmeqoeyhsqwasjjjvu


j 1 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1
single_example: 	 aardonyx
single_example_chars: 	 ['a', 'a', 'r', 'd', 'o', 'n', 'y', 'x']
single_example_ix: 	 [1, 1, 18, 4, 15, 14, 25, 24]
[1, 1, 18, 4, 15, 14, 25] single_example_ix[:-1]
X: 	 [None, 1, 1, 18, 4, 15, 14, 25, 24]
Y: 	 [1, 1, 18, 4, 15, 14, 25, 24, 0]
j 2 range range(0, 22001)
examples 1536
num_iterations 22001
j 2000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 464
single_example: 	 eousdryosaurus
single_example_chars: 	 ['e', 'o', 'u', 's', 'd', 'r', 'y', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19]
[5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19, 0]
Iteration: 2000, Loss: 26.681733

Iiytronheoravhoss
Eola
Eytosaurus
Idaahosalaus
Xuskeonoraveros
Cdalosan
Tosaurus
j 6000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1392
single_example: 	 traukutitan
single_example_chars: 	 ['t', 'r', 'a', 'u', 'k', 'u', 't', 'i', 't', 'a', 'n']
single_example_ix: 	 [20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14]
[20, 18, 1, 21, 11, 21, 20, 9, 20, 1] single_example_ix[:-1]
X: 	 [None, 20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14]
Y: 	 [20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14, 0]
Iteration: 6000, Loss: 23.075217

Titotomanosaurus
Sidachssalchunthenus
Sutotcmenonus
Tha
Utoterchomunmtocongitochusus
Laaiton
Totaresaurus
j 22000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 496
single_example: 	 fabrosaurus
single_example_chars: 	 ['f', 'a', 'b', 'r', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
[6, 1, 2, 18, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19, 0]
Iteration: 22000, Loss: 20.578871

Hutusaurus
Euca
Eustrioppn
Hocamptopanceus
Xuspeodon
Elacropechus
Uspeodon


---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

AssertionError: Wrong expected output

Ok, it looks like you’ve got the generation of X and Y fixed now. Next I think you should investigate why your single example values are different than mine. Did you mess with this code that was given to you as part of the template for model:

    # Shuffle list of all dinosaur names
    np.random.seed(0)
    np.random.shuffle(examples)

Notice that you are starting with aachenosaurus which is literally the first name on the original list. You’re supposed to shuffle the list first.

1 Like

! the issue was that instead of applying the index to the shuffled examples, in my code the index was applied to text corpus passed as one of the function arguments :slight_smile:

that’s why the result was different

Thank you so much!

It’s great that you found the solution! Thanks for confirming.

Really stuck at the same problem. Not getting how to take shuffled list. Please help! Here are the lines:

{mentor edit: code removed - not allowed by the Code of Conduct}

I have made no changes in default code written.
Output getting :-

Solved it myself !
The error was in initialising the idx .
Now it is {mentor: code removed}

1 Like

Hi @Khushboo_chauddhary

That is great you found the error.

1 Like

Hi Paulinpaloalto, I’m facing a problem here too:
My first print (iteration: 0) is identical to yours, but than it starts to differ.
I realize that you have values:

j =  1535 idx =  1535
j =  1536 idx =  0

My results are:
j = 1535 idx = 2
j = 1536 idx = 3
After that my loss drops sharply:
Iteration: 2000, Loss: 9.894264

grafik

I think I got the modulo right by using “j % dino_names”.
I didn’t change anything outside the Start/End your code -section.
Do you have any idea?
Thanks in advanced!

Sorry, that is wrong, which is exactly what causes this problem. That value has nothing to do with the training: watch how it is used. It only controls how many names get printed every 1000 iterations, right? The point is that you are indexing into the list of actual examples, so it’s the length of that list that you need to use as the divisor on the modulo there.

Hi,
now it works.
Thanks for the fast response!
Maybe I was having a bad day, but the Dinosaur Assignment (Exercise 2 and 4) was by far the hardest one of the entire specialization for me to understand. I’m a little worried about what’s to come in the next three weeks…

This course is the most challenging of DLS for sure. I hate to be the bearer of bad news, but there is a fair amount of complexity still ahead here. :scream_cat: But the culmination in Week 4 is really the core technology of everything that’s been happening in the last few years with LLMs, so it’s a “must see”. Onward! :nerd_face:

The disappointing thing about the Week 4 assignment is that you create a Transformer model, but you don’t get the payoff of seeing it do anything useful.

There are some optional labs after the assignment that show what a Transformer can do.

1 Like