I do not get the advantages of TensorFlow from this video. Espectially the “tape.gradient()”. I feel like it helps only just a little

In the first course , when we had to implement the compute_gradient code.

Which it is quite simple.

From the video, when use TensorFlow, it looks like this:

I do agree that the “record the steps” can be useful , but the "“tape.gradient()” seems like it facilitate just a little, instead of fully implement "((w*x[i] + b) - y[i])*x[i]" , it helps to do like tape.gradient((w*x-y)**2,[w])

Do I miss something?

The teacher said " So this procedure allows you to implement gradient descent without ever having to figure out yourself how to compute this derivative term. This is a very powerful feature of TensorFlow called Auto Diff. "

But “having to figure out yourself” , this part, is that complex? from our lab assignment in the Course 1, it seems not that complex.

Hello @Surasin_Tancharoen,

Like you said, I think `tf.GradientTape`

saves our time in deriving the derivative formula and writing codes about it. Isn’t this something a software framework is powerful at? That we write code that requires human analysis and can’t be automated, then the software takes care of what must happen according to established rules. In this way we increase productivity, and be less prone to bugs.

Our course 1 assignment is simple, and I think I cannot give you any example that tensorflow’s auto differentiation can do but human can’t. In principle I think we can, however, if we speak about complexity, such as a Neural network with 1 million parameters, then I believe we should thank auto differentiation for taking care of all the labour work of keeping track of how each parameter contributes to the cost. The framework makes our modeling work scalable.

Cheers,

Raymond

1 Like

This paper looks into auto differentiation. For example, it compares 4 ways we can do the job:

C. C. Margossian. 2019. A Review of Automatic Differentiation

and its Efficient Implementation.

2 Likes