Optimizing Model - Imbalance?

AI Discussions
,
1

enter preformatted text hereI am creating a model which does multi-class classification, and I tried to optimize it by using weighted loss.

My dataset contains 7 classes. Here is the number of data points I have for each:
MEL: 1113
NV: 6705
BCC: 514
AKIEC: 327
BKL: 1099
DF: 115
VASC: 142

Could the model be “exploiting” this weighted loss by only returning “AKIEC” every time?
In other words, I by making certain classes “more wrong”, is it making those classes “more right” and other ones “less wrong”?
(not sure why it isn’t returning VASC every time, but AKIEC is what I observed)

Here is the output my model gave on a “MEL” image:
[1.9501211e-02, 8.6555272e-02, 7.0136093e-02, 5.3331017e-02, 7.6896048e-01,
1.2218184e-03, 2.9415233e-04]

And here is the weighted loss weights my model used:
{0: 0.888866699950075, 1: 0.3305042436345482, 2: 0.9486769845232151, 3: 0.9673489765351972, 4: 0.8902646030953569, 5: 0.9885172241637543, 6: 0.9858212680978532}

This resulted in a categorical cross-entropy loss of ~0.33 and a binary accuracy of ~80% in validation. However, separate testing on individual images suggested otherwise.

And in case if it’s necessary, here is my model code:

def classi(input_shape):
    inputs = layers.Input(shape=input_shape)
    vgg19 = k.applications.VGG19(include_top=False, weights="imagenet", input_tensor=inputs)
    x = vgg19(inputs, training=False)
    
    x = layers.Conv2D(64, 3, padding="same")(x)
    x = layers.Activation("relu")(x)
    x = layers.BatchNormalization()(x)
    #classi layers
    for filters in [96, 128, 256]:#, 320]:#, 512]:#, 1024, 2048]:
        x = layers.Conv2D(filters, 3, padding="same")(x)
        x = layers.Activation("relu")(x)
        x = layers.BatchNormalization()(x)

        x = layers.Conv2D(filters, 3, padding="same")(x)
        x = layers.Activation("relu")(x)
        x = layers.BatchNormalization()(x)

        x = layers.MaxPool2D(3, strides=2, padding="same")(x)

    #output
    x = layers.Dropout(rate=0.3)(x)
    x = layers.Flatten()(x)
    x = layers.Dense(128, activation="sigmoid")(x)

    output = layers.Dense(7, activation="softmax")(x)

    model = k.Model(inputs=inputs, outputs=output, name="classification")
    return model

My model was trained with:

batch_size=8
steps_per_epoch = 10015//batch_size
epochs = 60

Is the problem here because of my weighted loss, or is my model just not “good” enough? (not trained enough, too few layers…) Or does it seem like a bug/logic error in my code?

2

** Move to AI Questions Category by Moderator **

3

Have you converted your seven classes into a one-hot representation?
What exactly do you mean by “binary accuracy”?

4

I think so, I return my classes in the form of a array of binary labels.

Binary accuracy was a metric automatically returned when I did .fit. I think it represents the fraction of labels the model predicted correctly.

5

Your dataset has a lot of skew - it is going to have a difficult time leaning to identify the 6th and 7th labels in your list.

What is your code for compile and fit?

6

compile & fit code

optimizer = tf.keras.optimizers.SGD(learning_rate=0.2)
classification.compile(optimizer=optimizer, loss="categorical_crossentropy", metrics=["binary_accuracy", f_score, precision_score, 'AUC']) 

callback_list = [tf.keras.callbacks.EarlyStopping(patience=1.2)] #can adjust to improve accuracy

seed = 123

cls_val = r'validation/ISIC2018_Task3_Validation_Input/'
cls_val_gt = "validation_ground_truth/ISIC2018_Task3_Validation_GroundTruth/ISIC2018_Task3_Validation_GroundTruth.csv"

cls_train = r'train/ISIC2018_Task3_Training_Input/'#r"classi/ISIC2018_Task3_Training_Input/ISIC2018_Task3_Training_Input/" 
cls_train_gt = 'train_ground_truth/ISIC2018_Task3_Training_GroundTruth/ISIC2018_Task3_Training_GroundTruth.csv'#("classi/ISIC2018_Task3_Training_GroundTruth/ISIC2018_Task3_Training_GroundTruth/ISIC2018_Task3_Training_GroundTruth.csv")

epochs=50
batch_size = 4
spe=4

print(batch_size*spe*epochs)
train_ds, train_gt = load_images_and_labels(cls_train, cls_train_gt, batch_size*spe*epochs, (256,256), True)
val_ds, val_gt = load_images_and_labels(cls_val, cls_val_gt, batch_size*spe*epochs, (256,256), True)
print(f"train_ds len: {len(train_ds)}, train labels len: {len(train_gt)}")
cls_train_gen = datagen.flow(x=train_ds, y=train_gt, seed=seed, batch_size=batch_size*spe, shuffle=True)
val_train_gen = datagen.flow(x=val_ds, y=val_gt, seed=seed, batch_size=batch_size*spe, shuffle=True)

history = classification.fit(cls_train_gen, steps_per_epoch=spe, epochs=epochs,
                                class_weight=get_weights(pd.read_csv(cls_train_gt)), 
                                batch_size=batch_size, callbacks=callback_list, verbose=1,
                                validation_data=(val_train_gen.x, val_train_gen.y), validation_steps=spe)

My image loading function is here:

def load_images_and_labels(images_path, labels_path, batch_size, image_shape, verbose=False):
    ds_images = []
    ds_labels = []
    data_indexes = []
    labels = pd.read_csv(labels_path)
    images = os.listdir(images_path)
    if verbose:
        print(f"loading images from {images_path} and labels from {labels_path}")
    for i in range(batch_size):
        random_index = np.random.randint(0, len(images)-2)
        if random_index >= len(images):
            random_index -=1
        img = cv2.imread(os.path.join(images_path, images[random_index]))
        row = labels.iloc[random_index, 1:]

        if img is not None and row is not None:
            if random_index not in data_indexes:
                data_indexes.append(random_index)
                ds_images.append(np.array(cv2.resize(img, dsize=image_shape)))
                ds_labels.append(row.values)
    return np.array(ds_images).astype(np.int16), np.array(ds_labels).astype(np.int16)
7

I don’t have any other suggestions on this topic for now.

Maybe someone from the community will have some thoughts.