Loss value changes from one run to the other

AI Discussions
1

Hello,

I made two models for the multi-class classification of images. The first model consists of 3( Conv2D → Maxpooling2d) to a flattened layer, followed by two dense layers.

the second model - I use inceptiveV3 as my base model then I pass the output of the ‘mixed7’ layer into a flattened layer and then into two dense layers.

base_model = InceptionV3(input_shape = ( 150 , 150 , 3), include_top = False)
base_model.trainable = False

there are about 14k image shapes (150, 150, 3) images in the training set and about 3k images in the validation set.

I have two big issues with these models:

1: they both take so long to run about 2-4 min per epoch. I run for 20- 25 epochs

  1. for the second model both the training and validation loss values change significantly from one run to another like 0.7 points or more. so every time I train the model I get different loss values.

any idea how I can shorten the training time and stabilize the loss values for the second model?

1 Like
2

Please move your topic to the correct subcategory. Since this topic is unrelated to the course notebooks, General Discussions is the right place for this.
Here’s the community user guide to get started.

  1. Explain your setup by providing the following details:
    a. GPU used
    b. Are you using ImageDataGenerator (see this as well)?
    c. 14k images is is still a lot of images. Are you using them as (150, 150, 3) or can you afford to make them smaller when reading? Details about the preprocessing pipeline is important here.
    d. Batch size
    e. CPU RAM usage.
    f. GPU RAM used.
    g. CPU usage
  2. See enable_op_determinism regarding reproducible results.
1 Like