I have a question regarding the answer option as follows:
“So long as the synthesized fog looks realistic to the human eye, you can be confident that the synthesized data is accurately capturing the distribution of real foggy images(or a subset of it), since human vision is very accurate for the problem you’re solving.”
I thought the 1,000 pictures of fog(seeming realistic to human-eye) might be the small portion of the all fog pictures. (as was mentioned in the lecture). So I thought this could not be an answer. Isn’t this a small portion of fog pics?
Great observation. It could be the case that the 1000 images of fog are only a subset of all possible fog. We also see that the answer includes a reservation about overfitting:
(or a subset of it)
If you look at the other two answers we have to choose from, they are more wrong than the answer you asked about, so that is why that answer is the correct choice.
I was similarly puzzled by the answer choices. This is really a poorly worded question & answer set.
Stating that one can be “confident” that simulated fog is representative of the distribution of real fog is over reaching in the absence of any measurements of the underlying distribution. Particularly considering the nuance of other questions in the same quiz lean heavily on.
This quiz deserves a thorough review & revision.
Another question:
You decide to focus on the dev set and check by hand what the errors are due to. Here is a table summarizing your discoveries:
| Overall dev set error |
15.3% |
| Errors due to incorrectly labeled data |
4.1% |
| Errors due to foggy pictures |
3.0% |
| Errors due to partially occluded elements. |
7.2% |
| Errors due to other causes |
1.0% |
In this table, 4.1%, 7.2%, etc. are a fraction of the total dev set (not just examples of your algorithm mislabeled). For example, about 7.2/15.3 = 47% of your errors are due to partially occluded elements in the image.
From this table, we can conclude that if we fix the incorrectly labeled data we will reduce the overall dev set error to 11.2%. True/False?
“True” is graded as incorrect, because
“The 4.1 only gives you an estimate of the ceiling of how much the error can be improved by fixing the labels.”.
While that rational is true in general cases —where your mitigation strategy may be only partially effective— if one were to truly correct all of the incorrectly labeled images, one should reasonably expect to recover all of the error due to mislabeled data.
Just as cost should be considered when selecting error mitigation strategies, so should the expected efficacy of the proposed mediation tactic.
I found the case study very frustrating and I strongly agree that it deserves a revision. The difficulty is not to recall the concepts from the course, but trying to interpret the often vague, debatable questions / answers. Especially the fog question / answer in this is really badly worded in my opinion. But yeah, I’m really confident that I’ll answer the questions correctly! (or a subset of them). 
