TensorFlow function train_step()

Course Q&A Deep Learning Specialization Improving Deep Neural Networks: Hyperparameter tun
1

Hi,

I’m following the video on tensorflow and trying to implement the steps of the function train_step() on my own but my gradients remain at zero and don’t change. What am I missing here ? Thanks.

import numpy as np

import tensorflow as tf


w = tf.Variable(0,dtype=tf.float32)

optimizer = tf.keras.optimizers.Adam(0.1)

def train_step():

    with tf.GradientTape() as tape:

        cost = w ** 2 - 10 + 25

    trainable_variables = [w]

    grads = tape.gradient(cost,trainable_variables)

    optimizer.apply_gradients(zip(grads,trainable_variables))


for i in range(1000):

    train_step()

print(w)

The result is:

<tf.Variable 'Variable:0' shape=() dtype=float32, numpy=0.0>

2

Just took a quick glance, but are you confident in that indentation? Seems like only the cost = line is inside the with tf.GradientTape() as tape: loop

It also looks like the value for w is always 0^2 + 15, which means cost Is also a constant with that value.

Haven’t looked back at that notebook for a while but that makes my spidey sense tingle

3

Good point about the indentation. But once you fix that bug, notice that the minimum value of that cost function occurs at w = 0 and that is the initial value you specified for w. So by all rights the gradients should all be zero, right? Perhaps you intended the formula to be:

J = w ^2 - 10w + 25

4

oh yes right, I forgot about w on 10w. that is the bug ! thanks