(C4W3A2) What is the meaning of Accuracy in the Segmentation assignment?

Course Q&A Deep Learning Specialization Convolutional Neural Networks
1

Unlike previous assignments and examples, the Semantic Segmentation assignment does not explicitly separate training and validation subsets, doing it instead via a Val_subsplit parameter that is not explained. In turn, the “accuracy” measure is a single one, which leads to my question; well, actually, four related questions:
Is this accuracy applicable to the validation subset?
Or, how can a validation accuracy be extracted?
Does the shuffling in the end blurs the distinction between train and validation?
Shouldn’t a true evaluation be done on a separate “test” subset of images?

Thanks for the education

2

Hello @EduardoChicago,

Let’s be clear about some facts first. Below are based on the latest version of notebook:

  1. There is a variable called VAL_SUBSPLITS but it is never used anywhere.

  2. The training uses only the training set, and no validation set.

Agree?

Raymond

1 Like
3

Dear RMW. My mistake; I thought that Val_subsplits WAS used. How then do we get a true measure of accuracy?
"The training uses only the training set, and no validation set. "
Again, I didn’t know that. How do we get an objective measure of performance/merit of the segmenter?
Could you briefly delineate the changes in the code to split the dataset into Tr, Val, and perhaps Test?
Many thanks for your help
…
(And why is Val_subsplits there if never used?).

4

Hey @EduardoChicago,

It’s alright. It takes both of us to build a good foundation for discussion.

From your questions, I assume you already know how to measure the accuracy - give it a validation set and perhaps a test set.

As for how to split the training set, I suggest you to follow through this discussion from top and bottom including the comments, because there are many interesting points, experience, and codes.

For how to give a validation set to the training process of a model, see this and you will find parameters with eye-catching names that include the word “validation”.

For how to use the test set, well, you just make prediction on it and pass the true labels and the predictions to a metric function!

Good luck!
Raymond

5

Thanks RMW!

“For how to use the test set…make a prediction on it and pass the true and predicted to a metric.…”
I tried, using Keras binary crosscorrelation matrix, and found that it failed. Couldn’t build the confusion matrix because the shapes of the “true masks” and “predicted masks” (built using the Coursera code) were different. I looked into it and found that “true” was (M, height, width, 1) and “predicted” was (M,height, width, 2). I guess this is a Keras bug, because the Coursera code just used the functions model.fit and model.predict.
in any case, what is the meaning of arrays [:, :, :, 0] and [:, :, :, 1] in predicted mask? If I knew that, I would know which one to use and solve the shape mismatch in the confusion matrix.

6

The basic factor to distinguish supervised from unsupervised is the labels. There are labels in that assignment which are the masks. Is there any reason for you to consider that it may be unsupervised?

7

I see my error. I thought that the validation stage was what made the process supervised.
But of course, it’s the labels.
It is amazing, though, the level of accuracy/merit that the trained unet model is capable of, trained as it was, without resort to a Validation set. I say this from my own initial prediction runs, on a small “test” dataset just built. My trained model was getting accuracies of 91% on the training set, and the predictions look almost equally accurate, on a tough set of electron microscope images.
And given how the Coursera Semantic Segmentation model is applied, there is no real difference in this case between a Validation and a Test set. In reality, there are only two sets: used in training and not used/unseen by the model.

8

Wow! Almost equally accurate and 91%. That’s a good news!

9

Of course, the accuracy on the test set is just by eyeball, as I still haven’t figured a good (dataset-oriented) way to calculate a metric, be it xentropy or IoU.

10

I see!

11

Again, and since you are so kind, what is the meaning of arrays [:, :, :, 0] and [:, :, :, 1] in predicted mask? If I knew that, I would know which one to use and solve the shape mismatch in the confusion matrix that is used to calculate binary crossentropy.

12

I have many sides :smiling_imp:, and you will see after reading the following:

  1. For [:,:,:,q], I suggest you to do the following investigation steps:

    1. print the shape for a prediction mask and the shape of the corresponding input image
    2. go to the model summary (that you have printed in one of the previous steps), and look at the output shape of the output layer of the model
    3. you will see “23”
    4. use ctrl+F to find out all appearance of “23”, and get the meaning of this number
    5. with your understanding on the meaning of 23, go back to the training set, and brainstorm a way to figure when a mask corresponds will correspond to 0, and when 1.

  2. I am actually going to move on to my own stuff, so a new thread will be more quickly responded to, as it will catch other mentors’ attention too.

Cheers :wink:

Raymond

1 Like
13

Sure Raymond. Your help went way beyond what I expected or deserved. Thanks a lot.

1 Like