What is Eager Tensor / Eager Execution

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

Hi,

I do not understand the concept of eager tensor and eager execution. ChatGPT gave me this explanation but, still, I am not clear:

In the traditional graph-based execution model, TensorFlow operations are first defined within a graph, and then the graph is executed within a session. The graph represents the flow of data and operations, and computations are performed by executing the graph in a session. This model provides benefits such as optimization and the ability to distribute computations across multiple devices or machines. However, it can be less intuitive and interactive for certain tasks, such as debugging or working with small-scale models.

With eager execution, TensorFlow operations are executed immediately and the results are returned directly. This allows for a more intuitive and interactive development experience, similar to how you would write regular Python code. It enables you to use Python control flow statements (e.g., loops and conditionals) to dynamically control the execution of operations, and it simplifies the process of inspecting intermediate results and debugging TensorFlow code.

And example is:

import tensorflow as tf

# Graph-based execution
tf.compat.v1.disable_eager_execution()
a = tf.constant(2)
b = tf.constant(3)
c = tf.add(a, b)
print(c)

Output: Tensor("Add:0", shape=(), dtype=int32)

# Eager execution
tf.compat.v1.enable_eager_execution()
d = tf.constant(2)
e = tf.constant(3)
f = tf.add(d, e)
print(f)

Output: tf.Tensor(5, shape=(), dtype=int32)

Best,
Saif.

PS: This query does not belong to any of the course assignments but a few months ago, I saw some error of eager tensor in DLS 4.

2
3

Hello Saif,

Can you tell the difference between the two outputs in that example?

Raymond

4

Here’s a case where eager mode helps with debugging:
image

without eager mode:

image
image752Ă—416 43.6 KB

5

Thank you, Tom, for the link. So, in Graph Execution, we first use tf.function decorator to convert the Python function to a graph function and then run it. However, in Eager Execution, we do not need that extra tf.funcntion. “Eager execution allows you to work with TensorFlow operations and tensors directly as you would with regular Python code, without the need to explicitly construct a computational graph.”

Thank you, Balaji, for showing me the example of debugging. So, in summary, Eager mode can be used to execute a natural Python code, without the need to explicitly construct a computational graph.

Hello Raymond! The enable_eager_execution() executes the code, adds the 2 & 3 and print the result as 5 (with shape and type). Conversely, disable_eager_execution() did not execute the code yet, but I don’t know what “Add:0” means. What is it?

Below is the code from the above article:

def eager_function(x):
  result = x ** 2
  print(result)
  return result

x = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0])
eager_function(x)

Output: tf.Tensor([ 1.  4.  9. 16. 25.], shape=(5,), dtype=float32)

def eager_function(x):
  result = x ** 2
  print(result)
  return result

x = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0])
graph_function = tf.function(eager_function)
graph_function(x)

Output: Tensor("pow:0", shape=(5,), dtype=float32)

In the above code, I was expecting the same output as tf.function is used. But what is that “pow:0”?

6

Good.

Still referring to the example in your first post, would you please change the second last line to the following by adding the name as shown

f = tf.add(d, e, name='I_am_the_add_operation')

Then run the example again and check the outputs.

Note that you can give every tensorflow operation a name.

In a tensorflow graph, we have operation and tensors. The general idea is, an operation operates on one or more tensors to produce one or more tensors.

As the tf.Graph  |  TensorFlow v2.14.0 said

Each graph contains a set of tf.Operation objects, which represent units of computation; and tf.Tensor objects, which represent the units of data that flow between operations.

Here is a discussion about the naming convention.

Let me know your latest thought about it. Perhaps you might have connected more dots up

7

The output of both lines is the same (in case of Eager Execution):

f = tf.add(d, e, name='I_am_the_add_operation')
f2 = tf.add(d, e)
print(f)
print(f2)
Output:
tf.Tensor(5, shape=(), dtype=int32)
tf.Tensor(5, shape=(), dtype=int32)

But in case of Graph Execution, the output is:

c = tf.add(a, b, name='I_am_the_add_operation')
c2 = tf.add(a, b)
print(c)
print(c2)
Output:
Tensor("I_am_the_add_operation:0", shape=(), dtype=int32)
Tensor("Add:0", shape=(), dtype=int32)

I got you, Raymond. “Add:0” is the name. And thanks for the link which discuss the same.

According to ChatGPT:
The output "pow:0" is a symbolic representation of the tensor, indicating the name and properties of the tensor within the TensorFlow graph. It does not directly show the concrete values of the tensor.

If you want to obtain the actual values of the tensor, you can evaluate it within a TensorFlow session or using the .numpy() method:

x = tf.constant([1.0, 2.0, 3.0, 4.0, 5.0])
graph_function = tf.function(eager_function)
graph_function(x)
Output: Tensor("pow:0", shape=(5,), dtype=float32)

result = graph_function(x)
print(result.numpy())
Output:
Tensor("pow:0", shape=(5,), dtype=float32)
[ 1.  4.  9. 16. 25.]

To conclude, I can say that Graph Execution needs more lines of code to execute and show the output. It also doesn’t execute the Python code directly, which is why we need a decorator before execution. But Eager Execution makes life easy. Right?

8

Eager execution lets you see the effect immediately.

In TF2, eager mode is by default enabled, and that’s why you need tf.compat.v1.disable_eager_execution to disable it.

Referring to the examples in the first post, a is a tensor created and called constant1:0 by the operation constant. It is similar for b. You can imagine two tensors are placed on the graph but not connected. They are then connected through an operator called add that produces a tensor called add:0 and remembered by the variable c.

When eager mode is disabled, the graph is built, and if you print c, you get the name of the tensor returned by the add operation of the graph remembered by the variable c. For now, it actually doesn’t quite care what the value of a (which was 2) and b (which was 3) are, because it does not really execute the graph.

When eager mode is enabled, if you print c, you get the value evaluated as graphed.

You need to know the benefit of building a graph to see its beauty.

Raymond

1 Like
9

For example,

a = tf.keras.layers.Input(...)
b = tf.math.add(a, 3)
c = tf.math.multiply(a, 2)

model = tf.keras.Model(inputs=a, outputs=c)

If you run this model, because tensorflow will first build a graph that connects a, b, and c up, and because it knows that a is the input and c is the output, it knows that b is not necessary because it does not contribute to the outputs. Therefore, it can skip b at execution. For example, you won’t see the multiply operation in model.summary(). This saved our time from running the part of graph that contributes nothing to the outputs.

Cheers,
Raymond

10

There is a blog post too.

Btw, there should be no graph built in eager mode.

11

I got it, Raymond. The article shared by Tom also discusses that Graph Execution is fast and for experts while Eager is for beginner-average and slow. I will explore it slowly slowly…

12

I agree with that. We can keep surprising ourselves by how we misunderstood someone else’s work :laughing:. Tensorflow is someone else’s work.

Cheers,
Raymond