C4 W4 A2 train_step 28 March 2022 update note

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

Hi guys

There is something not quite right with the notebook since today’s update, which coincided with me reaching the train_step exercise.

I noticed that the new notebook did not have my earlier code, so I rewrote each step, checking the results and all steps were passed.

I completed train_step and all steps passed, so I saved and checkpointed.
I re-highlighted the # UNQ_C4 cell by mistake, so I reran and then reran the next cell.

This time I got: "AssertionError: Unexpected cost for epoch 0: " and I checked several more times: each time the unexpected cost was different. Finally I reran all the cells above # UNQ_C4, even though code for the entire notebook had been written in one go, and it passed.

Maybe it needs an instruction to run ‘all cells above’ before running the train_step code?

2

I’m thinking this is universally true since training depends on model state. If you execute training a second (or more) time without reinitializing the parameters, you will get different results because you are starting with a baseline of trained parameters instead of the initial- often random - values.

3

Thank you ai_curious.

I haven’t come across this before while working on Jupyter notebooks.

Good to know.

4

In this Specialization, the first debugging step is always “restart the kernel and run all the cells again”.

This is because many of the notebooks have internal states, so the sequence of running the cells can be critical.

5

Thank you TMosh

onward(s)

6

Not to put too fine a point on it, but what you’re telling us is that you simply have not been paying attention. All the notebooks have worked this way from the “get go”. They are stateful. But you lose the state anytime you close and reopen them or do a “kernel restart”. This was explained in Course 1 Week 2 when we first started on all this.

Of course statefulness also implies the point that Tom made: the order in which you run the cells or rerun them also matters in the general case.

Also note that there is this comment in the cell right after the one that defines train_step:

# Show the generated image at some epochs
# Uncoment to reset the style transfer process. You will need to compile the *train_step* function again

That is new. In the previous version, it simply said this:

# You always must run the last cell before this one. You will get an error if not.

Sigh. I will file a bug about the fact that they apparently can’t spell “comment”.

7

A TensorFlow Model has an attribute that is a collection of Layers. Each Layer has a collection of weights, which are initialized when the Layer is instantiated. Unless they are designated as non_trainable_weights they are updated from the initial values during backprop. Those values persist in the Layer, and thus in the Model, after training completes and will be there until the Model is garbage collected or otherwise explicitly reinitialized. Especially since some of the initializers depend on random numbers, it is a good idea to restart and rerun the entire notebook in order to obtain repeatable output from training.

Useful references:

See for example the attributes list and the get_weights() method, which you could use to examine weights of one layer after successive train steps.

Cheers