Hi Team,
I noticed five issues on the quiz which should be fixed to allow other learners to make progress quickly and with less confusion. To facilitate, I’ve proposed fixes below. Since there are five issues here and only fifteen questions, this means a “perfect” learner might score only 66.7% when the passing threshold is 80%. Therefore, I’d also propose that passing this quiz be temporarily removed as a requirement for earning the certificate, or that the passing threshold be lowered to 53% on this quiz, since 80% of 66.7% is about 53%.
Thx!
Issue 1:
Q: Which of the following best answers why it is important to identify optimizing and satisficing metrics?
Answer marked as correct that is actually wrong: “Identifying the metric types sets the thresholds for satisficing metrics. This provides explicit evaluation criteria.”
Edit which would make this answer be correct: “Identifying the metric types allows us to know which metrics require that we set thresholds. This provides explicit evaluation criteria.”
Explanation: The “correct” answer is wrong and reads like a “trick” answer. It is false that “identifying the metrics types sets the thresholds.” We, the researchers, in consultation with the city counsel, have to set the thresholds. Identifying the metrics types does not do so. On this quiz, this minor error might not be a problem, except that a “correct but less strong” answer is also included, which apt learners will “incorrectly” choose.
Issue 2:
Q: One member of the City Council knows a little about machine learning, and thinks you should add the 1,000,000 citizens’ data images to the test set. You object because:
Correct option marked as “incorrect”: “the 1M citizens’ data images do not have a consistent x-> y mapping as the rest of the data.”
To prevent this problem, please reword the quiz question to: “One member of the City Council knows a little about machine learning, and thinks you should add the 1,000,000 citizens’ data images to the test set. The images are labeled using procedures and personnel identical to that for the security camera images. You object because:”
Explanation: With the question as-is, learners will have to guess at the labeling method as well as the image type, which is of course itself problematic as reflected in other correct answer options. Even so, the learner must check all correct options, and there isn’t enough info in the question as originally worded.
Issue 3:
Q: “The City Council thinks that having more cats in the city would help scare off birds. They are so happy with your work on the Bird detector that they also hire you to build a Cat detector. You have a huge dataset of 100,000,000 cat images. Training on this data takes about two weeks. Which of the statements do you agree with? (Check all that agree.)”
Please modify the question to add the following, or similar: “Image quality and Bayes error is similar to that for the bird project” in order for the question to be answerable.
Issue 4:
Q: “You train a system, and the train/dev set errors are 3.5% and 4.0% respectively. You decide to try regularization to close the train/dev accuracy gap. Do you agree?”
Proposed rewording: “You train a system, and the train/dev set errors are 3.5% and 4.0% respectively. A member of your team proposes to try regularization to close the train/dev accuracy gap. Do you agree?”
Explanation: the non-standard use of the word “you” in the question is confusing.
Issue 5:
The city revises its criteria to:
“We need an algorithm that can let us know a bird is flying over Peacetopia as accurately as possible.”
"We want the trained model to take no more than 10 sec to classify a new image.”
“We want the model to fit in 10MB of memory.”
Given models with different accuracies, runtimes, and memory sizes, how would you choose one?
For the answer marked “correct” to actually be correct, please replace instances of the word “want” with “need” in the question prompt.
Explanation: In standard language, the word “want” reflects a soft rather than hard requirement. Such requirements are addressed by the slides under “When to change dev/est sets and metrics” not under “satisficing and optimizing metrics.” Learners with industry experience will likely interpret the word “want” and “need” as reflecting a “should” and “shall” respectively in a PRD or standards doc. To reflect industry norms, please reword the question as noted.