C4_W4_Lab_3_CelebA_GAN_Experiments tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence

Hello,

When I do Run all cells (without any modification) in notebook C4_W4_Lab_3_CelebA_GAN_Experiments : week 4 lab 3 " Ungraded Lab: CelebA GAN Experiments" of " Generative Deep Learning with TensorFlow" course,
when I execute the cell to train the GAN and generating images :

# generate a batch of noisy input
test_z = tf.random.normal(shape=(64, 1, 1, z_dim))

# start loop
for epoch in range(30):
    with tqdm(dataset) as pbar:
        pbar.set_description(f"[Epoch {epoch}]")
        for step, (X1, X2) in enumerate(pbar):
            # train on the current batch
            d_loss, g_loss, fake = train_on_batch(X1, X2)

            # display the losses
            pbar.set_postfix({"g_loss": g_loss.numpy(), "d_loss": d_loss.numpy()})

        # generate fake images
        fake_img = model_G(test_z)

    # save output
    if not os.path.exists(out_dir):
        os.makedirs(out_dir)
    file_path = out_dir+f"/epoch_{epoch:04}.png"
    save_img(fake_img.numpy()[:64], file_path, 8)

    # display gallery of fake faces
    if epoch % 1 == 0:
        with Image.open(file_path) as img:
            plt.imshow(np.asarray(img))
            plt.show()

At each epoch, at step 1580, I get the error :

Epoch 0]: : 1580it [01:09, 30.46it/s, g_loss=1e+4, d_loss=33.3]Exception ignored in: <function Executor.__del__ at 0x7e9554558c10>
Traceback (most recent call last):
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/executor.py", line 46, in __del__
    self.wait()
  File "/usr/local/lib/python3.10/dist-packages/tensorflow/python/eager/executor.py", line 65, in wait
    pywrap_tfe.TFE_ExecutorWaitForAllPendingNodes(self._handle)
tensorflow.python.framework.errors_impl.OutOfRangeError: End of sequence
[Epoch 0]: : 1582it [01:09, 22.84it/s, g_loss=1e+4, d_loss=33.3]

=> and the generated images does not improve and stay all yellow

Help appreciated
Best regards
Thierry

Hello,

I ran the notebook right now and yes I got an exception which was ignored and it seems the images are improving. Close the page and rerun the whole thing if the images dont improve at all with time.

Hello,
Indeed, you are right, re-running the notebook, I still have the error but the generated images are improving
Thank you
Do you know how to fix the error ? Do you know the root cause ?
Best regards
Thierry

No I cannot take a a guess right now as to what the problem might be!