Dinosaurus_Island_Character_level_language_model - Exercise 4 Model

Kaiju · January 29, 2022, 4:07pm

trying to debug exercise 4.

I suspect there is something wrong with theY.

I have set Y to equal to elements fromX starting from the first element, so this should take care of the link between y and x highlighted in yellow. but the very last element (green) is not added to Y.

I thought at first that it would be resolved by appending Y with last_dino_name as a list addition, but this does not work either. Not sure what last_dino_name does in the code

Do I need to forward propagate the last element of x to get the y and add it to the current Y value? Or is it the character to represent end of the word?

The code expects a 1 line solution, so not sure if this is what is expected in this exercise.

j 0 range range(0, 22001)
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
---------------------------------------------------------------------------
IndexError                                Traceback (most recent call last)
<ipython-input-102-725c093d6b91> in <module>
----> 1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
      3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

<ipython-input-101-10a8c686571a> in model(data_x, ix_to_char, char_to_ix, num_iterations, n_a, dino_names, vocab_size, verbose)
     64         # Perform one optimization step: Forward-prop -> Backward-prop -> Clip -> Update parameters
     65         # Choose a learning rate of 0.01
---> 66         curr_loss, gradients, a_prev = optimize(X, Y, a_prev, parameters, learning_rate = 0.01)
     67 
     68         ### END CODE HERE ###

<ipython-input-93-fca0aba78113> in optimize(X, Y, a_prev, parameters, learning_rate)
     32 
     33     # Forward propagate through time (≈1 line)
---> 34     loss, cache = rnn_forward(X, Y, a_prev, parameters)
     35 
     36     # Backpropagate through time (≈1 line)

~/work/W1A2/utils.py in rnn_forward(X, Y, a0, parameters, vocab_size)
    100 
    101         # Update the loss by substracting the cross-entropy term of this time-step from it.
--> 102         loss -= np.log(y_hat[t][Y[t],0])
    103 
    104     cache = (y_hat, a, x)

IndexError: list index out of range

Kaiju · January 29, 2022, 6:21pm

Loop change

I have adjusted the optimisation loop to divide number of iterations by the length of examples using module operator.
Not clear why the hint suggest adding 1 in the numerator.
here, we have 1536 examples and 22001 iterations. Simple modulor operator allows to loop through all examples.

Y
set Y to equal now elements of X starting from 1st element onwards and added a newline character through list addition signalling end of the word.

but the code still does not work.

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

AssertionError: Wrong expected output

Kaiju · January 29, 2022, 6:22pm

j 0 range range(0, 22001)
examples 1536
num_iterations 22001
idx 0
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 0]
j =  0 idx =  0
single_example = aachenosaurus
single_example_chars ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
 X =  [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21] 
 Y =        [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 0] 

Iteration: 0, Loss: 23.090636

Nkzxwtdmeqoeyhsqwasjjjvu
Kneb
Kzxwtdmeqoeyhsqwasjjjvu
Neb
Zxwtdmeqoeyhsqwasjjjvu
Eb
Xwtdmeqoeyhsqwasjjjvu


j 1 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1
single_example: 	 aardonyx
single_example_chars: 	 ['a', 'a', 'r', 'd', 'o', 'n', 'y', 'x']
single_example_ix: 	 [1, 1, 18, 4, 15, 14, 25, 24]
X: 	 [None, 1, 1, 18, 4, 15, 14, 25]
Y: 	 [1, 1, 18, 4, 15, 14, 25, 0]
j 2 range range(0, 22001)
examples 1536
num_iterations 22001
idx 2
single_example: 	 abdallahsaurus
single_example_chars: 	 ['a', 'b', 'd', 'a', 'l', 'l', 'a', 'h', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21, 19]
X: 	 [None, 1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21]
Y: 	 [1, 2, 4, 1, 12, 12, 1, 8, 19, 1, 21, 18, 21, 0]

examples 1536
num_iterations 22001
idx 495
single_example: 	 eustreptospondylus
single_example_chars: 	 ['e', 'u', 's', 't', 'r', 'e', 'p', 't', 'o', 's', 'p', 'o', 'n', 'd', 'y', 'l', 'u', 's']
single_example_ix: 	 [5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21, 19]
X: 	 [None, 5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21]
Y: 	 [5, 21, 19, 20, 18, 5, 16, 20, 15, 19, 16, 15, 14, 4, 25, 12, 21, 0]
j 22000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 496
single_example: 	 fabrosaurus
single_example_chars: 	 ['f', 'a', 'b', 'r', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
X: 	 [None, 6, 1, 2, 18, 15, 19, 1, 21, 18, 21]
Y: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 0]
Iteration: 22000, Loss: 20.333626

Loustreprosauru
Eule
Eurupelepodu
Localtregaru
Yurpengosauru
Cabdrope
Tromelosauru


---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"```

paulinpaloalto · January 29, 2022, 6:43pm

Here’s my output from running the training. Notice that you seem to be truncating X, when you convert single_example_ix to X. Compare to what I show below:

j =  0 idx =  0
single_example = turiasaurus
single_example_chars ['t', 'u', 'r', 'i', 'a', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19]
 X =  [None, 20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19] 
 Y =        [20, 21, 18, 9, 1, 19, 1, 21, 18, 21, 19, 0] 

Iteration: 0, Loss: 23.087336

Nkzxwtdmfqoeyhsqwasjkjvu
Kneb
Kzxwtdmfqoeyhsqwasjkjvu
Neb
Zxwtdmfqoeyhsqwasjkjvu
Eb
Xwtdmfqoeyhsqwasjkjvu


j =  1535 idx =  1535
j =  1536 idx =  0
Iteration: 2000, Loss: 27.884160

Liusskeomnolxeros
Hmdaairus
Hytroligoraurus
Lecalosapaus
Xusicikoraurus
Abalpsamantisaurus
Tpraneronxeros


Iteration: 4000, Loss: 25.901815

Mivrosaurus
Inee
Ivtroplisaurus
Mbaaisaurus
Wusichisaurus
Cabaselachus
Toraperlethosdarenitochusthiamamumamaon


Iteration: 6000, Loss: 24.608779

Onwusceomosaurus
Lieeaerosaurus
Lxussaurus
Oma
Xusteonosaurus
Eeahosaurus
Toreonosaurus

Kaiju · January 29, 2022, 6:50pm

@paulinpaloalto

Yes, i see, when selecting the corresponding value for the single_example_ix, i did not need to select all apart for the last element as the output was a list and not dictionary key followed by the value.

restarted kernel, reran the cell, still the result is different and is not accepted by the Autograder.

j 0 range range(0, 22001)
examples 1536
num_iterations 22001
idx 0
single_example: 	 aachenosaurus
single_example_chars: 	 ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
[1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19, 0]
j =  0 idx =  0
single_example = aachenosaurus
single_example_chars ['a', 'a', 'c', 'h', 'e', 'n', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19]
 X =  [None, 1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19] 
 Y =        [1, 1, 3, 8, 5, 14, 15, 19, 1, 21, 18, 21, 19, 0] 

Iteration: 0, Loss: 23.093932

Nkzxwtdmeqoeyhsqwasjjjvu
Kneb
Kzxwtdmeqoeyhsqwasjjjvu
Neb
Zxwtdmeqoeyhsqwasjjjvu
Eb
Xwtdmeqoeyhsqwasjjjvu


j 1 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1
single_example: 	 aardonyx
single_example_chars: 	 ['a', 'a', 'r', 'd', 'o', 'n', 'y', 'x']
single_example_ix: 	 [1, 1, 18, 4, 15, 14, 25, 24]
[1, 1, 18, 4, 15, 14, 25] single_example_ix[:-1]
X: 	 [None, 1, 1, 18, 4, 15, 14, 25, 24]
Y: 	 [1, 1, 18, 4, 15, 14, 25, 24, 0]
j 2 range range(0, 22001)
examples 1536
num_iterations 22001

j 2000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 464
single_example: 	 eousdryosaurus
single_example_chars: 	 ['e', 'o', 'u', 's', 'd', 'r', 'y', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19]
[5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [5, 15, 21, 19, 4, 18, 25, 15, 19, 1, 21, 18, 21, 19, 0]
Iteration: 2000, Loss: 26.681733

Iiytronheoravhoss
Eola
Eytosaurus
Idaahosalaus
Xuskeonoraveros
Cdalosan
Tosaurus

j 6000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 1392
single_example: 	 traukutitan
single_example_chars: 	 ['t', 'r', 'a', 'u', 'k', 'u', 't', 'i', 't', 'a', 'n']
single_example_ix: 	 [20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14]
[20, 18, 1, 21, 11, 21, 20, 9, 20, 1] single_example_ix[:-1]
X: 	 [None, 20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14]
Y: 	 [20, 18, 1, 21, 11, 21, 20, 9, 20, 1, 14, 0]
Iteration: 6000, Loss: 23.075217

Titotomanosaurus
Sidachssalchunthenus
Sutotcmenonus
Tha
Utoterchomunmtocongitochusus
Laaiton
Totaresaurus

j 22000 range range(0, 22001)
examples 1536
num_iterations 22001
idx 496
single_example: 	 fabrosaurus
single_example_chars: 	 ['f', 'a', 'b', 'r', 'o', 's', 'a', 'u', 'r', 'u', 's']
single_example_ix: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
[6, 1, 2, 18, 15, 19, 1, 21, 18, 21] single_example_ix[:-1]
X: 	 [None, 6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19]
Y: 	 [6, 1, 2, 18, 15, 19, 1, 21, 18, 21, 19, 0]
Iteration: 22000, Loss: 20.578871

Hutusaurus
Euca
Eustrioppn
Hocamptopanceus
Xuspeodon
Elacropechus
Uspeodon


---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
<ipython-input-13-725c093d6b91> in <module>
      1 parameters, last_name = model(data.split("\n"), ix_to_char, char_to_ix, 22001, verbose = True)
      2 
----> 3 assert last_name == 'Trodonosaurus\n', "Wrong expected output"
      4 print("\033[92mAll tests passed!")

AssertionError: Wrong expected output

paulinpaloalto · January 29, 2022, 8:45pm

Ok, it looks like you’ve got the generation of X and Y fixed now. Next I think you should investigate why your single example values are different than mine. Did you mess with this code that was given to you as part of the template for model:

    # Shuffle list of all dinosaur names
    np.random.seed(0)
    np.random.shuffle(examples)

Notice that you are starting with aachenosaurus which is literally the first name on the original list. You’re supposed to shuffle the list first.

Kaiju · January 29, 2022, 9:13pm

! the issue was that instead of applying the index to the shuffled examples, in my code the index was applied to text corpus passed as one of the function arguments

that’s why the result was different

Thank you so much!

paulinpaloalto · January 29, 2022, 9:18pm

It’s great that you found the solution! Thanks for confirming.

Khushboo_chauddhary · October 20, 2022, 7:29am

Really stuck at the same problem. Not getting how to take shuffled list. Please help! Here are the lines:

{mentor edit: code removed - not allowed by the Code of Conduct}

I have made no changes in default code written.
Output getting :-

Khushboo_chauddhary · October 20, 2022, 8:51am

Solved it myself !
The error was in initialising the idx .
Now it is {mentor: code removed}

Kic · October 20, 2022, 11:02am

Hi @Khushboo_chauddhary

That is great you found the error.

Bernhard_Wieczorek · February 21, 2024, 4:23pm

Hi Paulinpaloalto, I’m facing a problem here too:
My first print (iteration: 0) is identical to yours, but than it starts to differ.
I realize that you have values:

j =  1535 idx =  1535
j =  1536 idx =  0

My results are:
j = 1535 idx = 2
j = 1536 idx = 3
After that my loss drops sharply:
Iteration: 2000, Loss: 9.894264

grafik

I think I got the modulo right by using “j % dino_names”.
I didn’t change anything outside the Start/End your code -section.
Do you have any idea?
Thanks in advanced!

paulinpaloalto · February 21, 2024, 5:00pm

Sorry, that is wrong, which is exactly what causes this problem. That value has nothing to do with the training: watch how it is used. It only controls how many names get printed every 1000 iterations, right? The point is that you are indexing into the list of actual examples, so it’s the length of that list that you need to use as the divisor on the modulo there.

Bernhard_Wieczorek · February 21, 2024, 5:24pm

Hi,
now it works.
Thanks for the fast response!
Maybe I was having a bad day, but the Dinosaur Assignment (Exercise 2 and 4) was by far the hardest one of the entire specialization for me to understand. I’m a little worried about what’s to come in the next three weeks…

paulinpaloalto · February 21, 2024, 6:05pm

This course is the most challenging of DLS for sure. I hate to be the bearer of bad news, but there is a fair amount of complexity still ahead here. But the culmination in Week 4 is really the core technology of everything that’s been happening in the last few years with LLMs, so it’s a “must see”. Onward!

TMosh · February 21, 2024, 6:48pm

The disappointing thing about the Week 4 assignment is that you create a Transformer model, but you don’t get the payoff of seeing it do anything useful.

There are some optional labs after the assignment that show what a Transformer can do.

Topic		Replies	Views
Dinosaurus_Island_Character_level_language_model Sequence Models	6	1097	July 26, 2023
Dino character level language model - indexing examples Sequence Models	5	791	July 4, 2021
Dinosaurus_Island_Character_level_language_model - Sample Function not Working Sequence Models	12	631	January 7, 2023
Dinosaur Island-Character-Level Language Modeling- error in final exercise Sequence Models	3	1005	June 16, 2021
Issue with Dinosaur Island ex 4. Not learning properly Sequence Models	5	524	January 16, 2023

Dinosaurus_Island_Character_level_language_model - Exercise 4 Model

Related topics