Course: Improving Deep Neural Networks: Hyperparameter Tuning, Regularization and Optimization
Week 3. Assignment: /notebooks/release/W3A1/Tensorflow_introduction.ipynb
Jupyter notebook crashes each time.
There are several possibilities when you get that type of error:
- Have you added any print statements in your code that get executed in the training loop? Anything that causes lots of additional output can give you failures like this.
- It could just be “bad weather” in the cloud, meaning that the servers are overloaded or the particular AWS VM that you are connected to is overloaded.
If you’ve checked 1) and don’t see any excess output, then for 2) you can just wait half an hour and try again. Or restart the notebook: save the contents and then reopen it from the “Work in Lab” link. Or if that doesn’t help, then do the old “log out and log in again” trick and hope you get connected to a VM that’s not as heavily loaded.
Hi again,
Thank you very much for your answer.
I didn’t add any print statements to the code.
And after 24 hours break, I got the same result - the Jupyter notebook fell with the same message and sign “kernel dead”.
So, I can’t move forward with the assignment.
What would you advice to do?
Thank you in advance,
Dmitry
I assume that you have not had this problem with any of the notebooks up to this point in DLS Course 1 and Course 2. If that’s true, then maybe there is something damaged about your notebook. I can try to take a look at it. We can’t do that on a public thread, but please check your DMs for a message from me about how to proceed.
A kernel dies or tries to reconnect or restart, when the codes in a given cell was able to run down completely, this could be network issue, or if repeated occuring then one needs to have a look at their codes as it is not able to rundown completely. one way to resolve is as mentioned by paul, to clear out kernel output and re-run the cell from beginning till you encounter the issue, otherwise check your code again if it requires any changes.
If you don’t have any additional print statements, then the other thing that can cause this behavior is an infinite loop in your code. It’s not clear how you could get that result with the logic here, but it’s worth taking a look at your code again with that idea in mind. The only place with a loop is the model function that does the training. They gave you the structure of the loop in the template, so please check that you didn’t modify the limit on the loop or anything else in that vein.
I’ve sent a DM requesting the notebook and will let you know if I can find anything …
Please check your DMs: you can recognize a DM in your “feed” by the little envelope icon.
I modify nothing exept the code I was asked to add.
Where can I see DM? In my email?
The two main causes of that message:
- Your notebook is too big (i.e. takes too long to load). The cause is often printing too much data to the console while debugging your code.
- Your code causing an infinite loop, and the kernel times-out.
To close the loop on the public thread, the problem is that the logic in forward_propagation was written using numpy functions, instead of TF functions. The test cell for that function uses gradient tape, so including numpy functions in the compute graph breaks that. Although I can’t really explain why it does the “spinning wheel of death” in that case, instead of throwing an error about “can’t compute gradients”.