Hi, I am having a problem with the update function for adam algorithm. Anyone care to explain?

Hi @flaves, and welcome to Discourse. Are you referring to the update in the ADAM algorithm to model parameters? Can you be more specific as to which part of the update is less understandable?

yes I am referring to it

def update_parameters_with_adam(parameters, grads, v, s, t, learning_rate = 0.01,

beta1 = 0.9, beta2 = 0.999, epsilon = 1e-8):

This function. My codes keep giving error. Here is the error message

KeyError Traceback (most recent call last)

in

7 epsilon = 1e-2

8

----> 9 parameters, v, s, vc, sc = update_parameters_with_adam(parametersi, grads, vi, si, t, learning_rate, beta1, beta2, epsilon)

10 print(f"W1 = \n{parameters[‘W1’]}")

11 print(f"W2 = \n{parameters[‘W2’]}")

in update_parameters_with_adam(parameters, grads, v, s, t, learning_rate, beta1, beta2, epsilon)

34 # Moving average of the gradients. Inputs: “v, grads, beta1”. Output: “v”.

35 ### START CODE HERE ### (approx. 2 lines)

—> 36 v[“dW” + str(l)] = beta1 * v[“dW” + str(l)] + (1 - beta1) * grads[‘dW’ + str(l)]

37 v[“db” + str(l)] = beta1 * v[“db” + str(l)] + (1 - beta1) * grads[‘db’ + str(l)]

38 ### END CODE HERE ###

KeyError: ‘dW0’

If I’m not wrong the code in line 36 is within a for loop scope. That loop runs the index between 0 and L - 1. So, str(l) is 0 for the first iteration. Are you sure the dictionary was initialized for ‘l’ and not ‘l+1’?

## I have changed it to strl(l+1) but then a new error has occured

NameError Traceback (most recent call last)

in

----> 1 parametersi, grads, vi, si = update_parameters_with_adam_test_case()

2

3 t = 2

4 learning_rate = 0.02

5 beta1 = 0.8

NameError: name ‘update_parameters_with_adam_test_case’ is not defined

Hi, @yanivh.

In my notebook weights and biases (and their derivatives) are 1-indexed, so the loop runs from 1 to *L*. Maybe there was a bug in a previous version of the notebook.

@flaves, did you make any changes to the *for* statement?

Yes I did I made changes to the for statement after taking asking for help offline…

Even after loading the initial there was still error

What could be the problem?

Is there a way I an set this lab back to default. So i code from the scearch.

Restarting the kernel doesnt do that it only goes back to the previous check point

Hi @flaves, you can do it by clicking on Help | Get latest version. I would recommend you to download your current Jupyter notebook in the case you want to reuse part of the code you’ve done already.

Kind regards

The error you are having: `KeyError: dW0`

suggests that there is something wrong with the `for`

loop, it should go in the `range(1, L+1)`

… therefore it should not check `dW0`

. You should also check the previous function `initialize_adam`

as there is where the dictionaries are initialized.

reloaded it. Still the same error.

My for loop was iterated from 0 to l-1 (in range(l)

then my codes the string was in range of (l+1) to be able to get the key.

This is the error

ValueError Traceback (most recent call last)

in

7 epsilon = 1e-2

8

----> 9 parameters, v, s, vc, sc = update_parameters_with_adam(parametersi, grads, vi, si, t, learning_rate, beta1, beta2, epsilon)

10 print(f"W1 = \n{parameters[‘W1’]}")

11 print(f"W2 = \n{parameters[‘W2’]}")

ValueError: not enough values to unpack (expected 5, got 3)

adjusted the initalise function here is the error

UnboundLocalError Traceback (most recent call last)

in

1 parameters = initialize_adam_test_case()

2

----> 3 v, s = initialize_adam(parameters)

4 print(“v[“dW1”] = \n” + str(v[“dW1”]))

5 print(“v[“db1”] = \n” + str(v[“db1”]))

in initialize_adam(parameters)

27

28 # Initialize v, s. Input: “parameters”. Outputs: “v, s”.

—> 29 for l in range(1,l+1):

30 ### START CODE HERE ### (approx. 4 lines)

31 v[“dW” + str(l)] = np.zeros_like(parameters[“W” + str(l)])

UnboundLocalError: local variable ‘l’ referenced before assignment

NameError Traceback (most recent call last)

in

1 parameters = initialize_adam_test_case()

2

----> 3 v, s = initialize_adam(parameters)

4 print(“v[“dW1”] = \n” + str(v[“dW1”]))

5 print(“v[“db1”] = \n” + str(v[“db1”]))

in initialize_adam(parameters)

27

28 # Initialize v, s. Input: “parameters”. Outputs: “v, s”.

—> 29 for L in range(1,l+1):

30 ### START CODE HERE ### (approx. 4 lines)

31 v[“dW” + str(l)] = np.zeros_like(parameters[“W” + str(l)])

NameError: name ‘l’ is not defined

The variable `L`

holds the number of layers in the neural work, in our case its 2. Thus for each layer, you want to initialize the gradients/derivatives. You can refer the first code block in Python Looping Through a Range to solve it.

I have solved the range problem. There seems to be another error in the update adam function. This error

ValueError Traceback (most recent call last)

in

7 epsilon = 1e-2

8

----> 9 parameters, v, s, vc, sc = update_parameters_with_adam(parametersi, grads, vi, si, t, learning_rate, beta1, beta2, epsilon)

10 print(f"W1 = \n{parameters[‘W1’]}")

11 print(f"W2 = \n{parameters[‘W2’]}")

ValueError: not enough values to unpack (expected 5, got 3)

new error

ValueError Traceback (most recent call last)

in

7 epsilon = 1e-2

8

----> 9 parameters, v, s, vc, sc = update_parameters_with_adam(parametersi, grads, vi, si, t, learning_rate, beta1, beta2, epsilon)

10 print(f"W1 = \n{parameters[‘W1’]}")

11 print(f"W2 = \n{parameters[‘W2’]}")

in update_parameters_with_adam(parameters, grads, v, s, t, learning_rate, beta1, beta2, epsilon)

59 ### START CODE HERE ### (approx. 2 lines)

60 parameters[“W” + str(l)] = parameters[“W” + str(l)] - learning_rate * v_corrected[“dW” + str(l)] / np.sqrt(s_corrected[“dW” + str(l)] + epsilon)

—> 61 parameters[“b” + str(l)] = parameters[“b” + str(l)] - learning_rate * v_corrected[“db” + str(l)] / np.sqrt(s_corrected[“db” + str(l)] + epsilon)

62 ### END CODE HERE ###

63

ValueError: operands could not be broadcast together with shapes (2,1) (3,1)

I am really sorry to bother

Hi, I guess you already solved this one, just in case, pay attention to the way the loop was defined `for l in range(1, L + 1)`

, your change “broke” the loop.

In fact, you were not expected to change this line of code in the function you are iterating over the number of layers in the neural network (variable `L`

defined at the beginning of the function).

Hi,

To debug this error I would suggest to check the content of `parameters`

, `v_corrected`

and `s_corrected`

before performing the assignment i.e. print the content of those variables to pinpoint any potential error in the way you are performing the operation with them.

SOLVED. Thanks everyone