Week 3 Programming assignment section 7

Course Q&A Deep Learning Specialization Neural Networks and Deep Learning
,
1

When creating a post, please add:

I submitted the programming assignment for week 3 and the autograder gave me 100%, so I decided to go back and play with the NN on other datasets, in section 7.

I noticed that my outputs are non-deterministic. For example, when I set n_h = 5, and then train a model on gaussian_quantiles data set, the first execution of this block gives me the following costs:

Cost after iteration 0: 0.693124
Cost after iteration 1000: 0.068776
Cost after iteration 2000: 0.038353
Cost after iteration 3000: 0.031598
Cost after iteration 4000: 0.027778
Cost after iteration 5000: 0.025106
Cost after iteration 6000: 0.023074
Cost after iteration 7000: 0.021457
Cost after iteration 8000: 0.020128
Cost after iteration 9000: 0.019009

and the 2nd execution (with no changes to the code) outputs different costs

Cost after iteration 0: 0.693122
Cost after iteration 1000: 0.080631
Cost after iteration 2000: 0.058538
Cost after iteration 3000: 0.039226
Cost after iteration 4000: 0.031690
Cost after iteration 5000: 0.028178
Cost after iteration 6000: 0.026181
Cost after iteration 7000: 0.024730
Cost after iteration 8000: nan
Cost after iteration 9000: nan

Any idea what might be the issue?

Here is my code:

# Datasets
noisy_circles, noisy_moons, blobs, gaussian_quantiles, no_structure = load_extra_datasets()

datasets = {"noisy_circles": noisy_circles,
            "noisy_moons": noisy_moons,
            "blobs": blobs,
            "gaussian_quantiles": gaussian_quantiles}

### START CODE HERE ### (choose your dataset)
dataset = "gaussian_quantiles"
visualize_orig = 0
### END CODE HERE ###

X, Y = datasets[dataset]
X, Y = X.T, Y.reshape(1, Y.shape[0])

# make blobs binary
if dataset == "blobs":
    Y = Y%2

if visualize_orig:
    # Visualize the data
    plt.scatter(X[0, :], X[1, :], c=Y, s=40, cmap=plt.cm.Spectral);
else:  
    # set hidden layers
    hidden_nodes = 5
    
    # Build a model with a n_h-dimensional hidden layer
    parameters = nn_model(X, Y, hidden_nodes, num_iterations = 10000, print_cost=True)

    # Plot the decision boundary
    plot_decision_boundary(lambda x: predict(parameters, x.T), X, Y)
    plt.title("Decision Boundary for hidden layer size " + str(hidden_nodes))
2

Hi @ldz ,

Add a print statement after print statement for cost in the nn_model() to find out the values of A2 might give some insights.

3

When you ran the notebook the second time, did you restart the kernel first?

If you just run the training portion of the notebook a second time, the initial weights of the NN layers may not be re-initialized. In that case, you’d be just be fine-tuning the solution you got the first time, rather than starting over.

Also:
Welcome to the wonderful world of cost functions that have local minima.

The NN cost function isn’t convex, so you could get different results each time you train the model.

2 Likes
4

Very clear, thank you! I did not restart the kernel so when I did that, things are much more deterministic, thanks!

1 Like