C4_W4_Lab_3_CelebA_GAN_Experiments tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence

Course Q&A TensorFlow: Advanced Techniques Specialization Generative Deep Learning with TensorFlow
1

Hello,

When I do Run all cells (without any modification) in notebook C4_W4_Lab_3_CelebA_GAN_Experiments : week 4 lab 3 " Ungraded Lab: CelebA GAN Experiments" of " Generative Deep Learning with TensorFlow" course,
when I execute the cell to train the GAN and generating images :

# generate a batch of noisy input
test_z = tf.random.normal(shape=(64, 1, 1, z_dim))

# start loop
for epoch in range(30):
    with tqdm(dataset) as pbar:
        pbar.set_description(f"[Epoch {epoch}]")
        for step, (X1, X2) in enumerate(pbar):
            # train on the current batch
            d_loss, g_loss, fake = train_on_batch(X1, X2)

            # display the losses
            pbar.set_postfix({"g_loss": g_loss.numpy(), "d_loss": d_loss.numpy()})

        # generate fake images
        fake_img = model_G(test_z)

    # save output
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)
    file_path = out_dir+f"/epoch_{epoch:04}.png"
    save_img(fake_img.numpy()[:64], file_path, 8)

    # display gallery of fake faces
    if epoch % 1 == 0:
        with Image.open(file_path) as img:
            plt.imshow(np.asarray(img))
            plt.show()

At each epoch, at step 1580, I get the error :

Epoch 0]: : 1580it [01:09, 30.46it/s, g_loss=1e+4, d_loss=33.3]Exception ignored in: <function Executor.__del__ at 0x7e9554558c10>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__
    self.wait()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[Epoch 0]: : 1582it [01:09, 22.84it/s, g_loss=1e+4, d_loss=33.3]

=> and the generated images does not improve and stay all yellow

Help appreciated
Best regards
Thierry

2

Hello,

I ran the notebook right now and yes I got an exception which was ignored and it seems the images are improving. Close the page and rerun the whole thing if the images dont improve at all with time.

3

Hello,
Indeed, you are right, re-running the notebook, I still have the error but the generated images are improving
Thank you
Do you know how to fix the error ? Do you know the root cause ?
Best regards
Thierry

4

No I cannot take a a guess right now as to what the problem might be!