Course 3 Week 1 Assignment 1 Training the Model

Course Q&A Natural Language Processing NLP with Sequence Models
1

I get the following error while running train_model()

Step 1: Total number of trainable weights: 2327042
Step 1: Ran 1 train steps in 1.52 secs
Step 1: train CrossEntropyLoss | 0.70368373
Step 1: eval CrossEntropyLoss | 0.69825441
Step 1: eval Accuracy | 0.56250000

Step 10: Ran 9 train steps in 6.08 secs
Step 10: train CrossEntropyLoss | 0.65071505
Step 10: eval CrossEntropyLoss | 0.68158495
Step 10: eval Accuracy | 0.50000000
Wrong loss function. WeightedCategoryCrossEntropy_in3 was expected. Got CrossEntropyLoss_in3.
Wrong metrics in evaluations task. Expected [‘WeightedCategoryCrossEntropy’, ‘WeightedCategoryAccuracy’]. Got [‘CrossEntropyLoss’, ‘Accuracy’].
4 Tests passed
2 Tests failed

All previous tests pass. What should I be looking at?

Thanks,
Drew

2

Seems like the loss and accuracy are incorrect.

  1. Use tl.WeightedCategoryCrossEntropy instead of tl.CrossEntropyLoss
  2. Use tl.WeightedCategoryAccuracy instead of tl.Accuracy
3

Hi @balaji.ambresh :

Seems like the loss and accuracy are incorrect.

(I am assuming this also makes my exercise 8 test fails)

Thanks for responding. I am trying to figure out where I’m making a mistake. Perhaps one culprit is how I calculate example_weights in exercise 2. My lab id is tqxhipjp

Thanks,
Drew

4

Only course staff and site admins can access your lab by id.

Can you check the place where you create training.TrainTask and training.EvalTask?

5

Hi @balaji.ambresh:

I don’t touch the code that defines training.TrainTask and EvalTask.

I was told my example_weights was wrong so, after re-reading, I assigned it to
example_weights = np.ones_like(targets). If this is wrong, what should I be referring to?

Thanks,
Drew

6

Sorry for the confusion. What I meant was, check the place where you create instances of training.TrainTask and training.EvalTask.

7

Hello

I get the same error, but the TrainTask and EvalTask are created in a cell I’m not supposed to modify. (see below)
does that mean I should just ignore the failed test ?

PLEASE, DO NOT MODIFY OR DELETE THIS CELL

from trax.supervised import training

def get_train_eval_tasks(train_pos, train_neg, val_pos, val_neg, vocab_dict, loop, batch_size = 16):
[…]
metrics=[tl.CrossEntropyLoss(), tl.Accuracy()]

8

@Florent_Tho
Please click my name and message your notebook as an attachment.

9

my bad, I was using np.ones to generate the weights resulting in float.
once corrected to integer (as the data_generator test suggested), the error disappears.

I thus guess that the tl.CrossEntropyLoss auto-upgrades to WeightedCrossEntropyLoss when weights are available

I’m surprised, in my practice (ex. optimizing MonteCarlo simulations), weights were floats to get fine grained control.

sorry for the trouble!

10

Hello @balaji.ambresh

I am getting the similar error.

Wrong loss function. WeightedCategoryCrossEntropy_in3 was expected. Got CrossEntropyLoss_in3.
Wrong metrics in evaluations task. Expected [‘WeightedCategoryCrossEntropy’, ‘WeightedCategoryAccuracy’]. Got [‘CrossEntropyLoss’, ‘Accuracy’].

but the TrainTask and EvalTask have been pre-defined with ‘CrossEntropyLoss’ and, ‘Accuracy’

11

@shanup
Please fix the task definition inside get_train_eval_tasks based on this feedback:

  1. 1st line is the expectation for training task. The test says that you should use WeightedCategoryCrossEntropy instead of CrossEntropyLoss
  2. 2nd line is the expectation for evaluation task. Use the expected to replace the ones you have.
Wrong loss function. WeightedCategoryCrossEntropy_in3 was expected. Got CrossEntropyLoss_in3.
Wrong metrics in evaluations task. Expected ['WeightedCategoryCrossEntropy', 'WeightedCategoryAccuracy']. Got ['CrossEntropyLoss', 'Accuracy'].
12

@balaji.ambresh

I already tried that. But then we have a mismatch with the expected values printed in the Jupyter notebook for the output of train_model (but this does not throw an error, though)

Step      1: Total number of trainable weights: 2327042
Step      1: Ran 1 train steps in 1.97 secs
Step      1: train WeightedCategoryCrossEntropy |  0.69622028
Step      1: eval  WeightedCategoryCrossEntropy |  0.69192076
Step      1: eval      WeightedCategoryAccuracy |  0.50000000
Expected output (Approximately)
Step      1: Total number of trainable weights: 2327042
Step      1: Ran 1 train steps in 1.54 secs
Step      1: train CrossEntropyLoss |  0.70209986
Step      1: eval  CrossEntropyLoss |  0.69093817
Step      1: eval          Accuracy |  0.50000000

I also checked the unit testcases and found this in the exception block

try:
        assert test_func(output_loop) == [
            "WeightedCategoryCrossEntropy",
            "WeightedCategoryAccuracy",
        ]  # ["CrossEntropyLoss", "Accuracy"]
        successful_cases += 1

It looks like the code in the unit testcase has been commented out for CrossEntropyLoss", "Accuracy. So we can pass the unit testcases by modifying the code in train_task and eval_task to the Expected values of [‘WeightedCategoryCrossEntropy’, ‘WeightedCategoryAccuracy’]. However, the expected values after running train_model are still looking for [‘CrossEntropyLoss’, ‘Accuracy’]

Seems like a mismatch all around between the expected values in the unit testcases and the expected values for the train_model output printed in the Jupyter notebook.

13

You should talk to an NLP mentor. I don’t have access to the current code. Adding
@arvyzukai

14

Hello @arvyzukai

Could you please take a look at this issue. Looks like a typo that needs to be fixed in the assignment notebook.

15

Hi, @shanup

Thank you for reporting the issue. There were a lot of code changes in August and some bugs came out as a result. The staff will try to fix it as soon as possible. :+1:

Please send me your Jupyter Notebook Assignment code so I could check how you got the mismatched output.

16

@ arvyzuka
I have the same issue.

17

Hello!

I am facing with the same error.
My ID lab is dfoqyqjk

Thanks in advance for your help.

18

Hello, same here, id skljxmaf .

More: if this is wrong, how can we trust the rest of the notebook?

19

Hi,

I wanted to report that I got a notification saying that the notebook was changed and after using the new version and double-checking all the implementation code, I’m seeing some suspicious dimension mismatch. This is the error I’m getting when trying to train the model:

TypeError: mul got incompatible shapes for broadcasting: (32, 2), (16, 2).

There are a bunch of cells that we are not supposed to touch and after double-checking all possible values I can’t seem to really find the issue. Can that be an issue with this latest notebook update?

20

Hi to all, I confirm that now everything works good. Thank you!