Understanding Human Level Performance Lecture V

Hi Sir/Mam,

@paulinpaloalto @lucaswener @Vasanth @carloshvp @ashimono @Sabs @manifest @fginac @Jendoubi

We had doubts in the lecture video Understanding Human Level Performance , can you please help to clarify the below doubts, we are struck into this video ?

Statement 1 : If you use 0.7 as your proxy for Bayes error, you would have estimated avoidable bias as pretty much 0%, and you might have missed that. You actually should try to do better on your training set.

Doubt in statement 1: what does it meaning "you might have missed that " and also "You actually should try to do better on your training set. I dont know why should we do better on training set since the avoidable bias 0%

Statement 2: How machine learning problem becomes harder when training error approaches bayes error ? How proffesor saying that ? If the training error equal to bayes error means 0% avoidable bias and so we directly reduce variance problem right. I dont know why they are saying harder to use tools reduce bias / variance.

Statement 3: In fact, if all you knew was that a single typical doctor achieves 1% error, and it might be very difficult to know if you should be trying to fit your training set even better. Can u please explain this statement ?

@paulinpaloalto @lucaswener @Vasanth @carloshvp @ashimono @Sabs @manifest @fginac @Jendoubi

Dear Mentors,

Can you please help to clarify the doubts ?

Hello @Anbu ,
the mentors are volunteers and every time anyone mentions them, they are all getting an email with a notification. This is in my experience counter-productive. Please avoid @ mentioning.

1 Like

Hey @Anbu,

As Carlos pointed out it’s not necessary to tag all mentors to get an answer. It’s quite a bad tone. In fact, you’re lowering your chances to get an answer.

As for your doubts, let’s clear them out :slight_smile:

Statement 1 : If you use 0.7 as your proxy for Bayes error, you would have estimated avoidable bias as pretty much 0%, and you might have missed that. You actually should try to do better on your training set.

Doubt in statement 1: what does it meaning "you might have missed that " and also "You actually should try to do better on your training set.

In the beginning of the video, Andrew said that we usually choose the human-level performance as a proxy for Bayes error, meaning the lowest error proved by experiments (in the video, that is an error for a team of experienced doctors). He also mentioned that we free to choose any value for the human-level performance if that makes sense for a particular research we conduct.

To estimate avoidable bias, we subtract a training set error from the human-level performance. If we got 0.5% as the training set error, it would make a little or no sense to chose 0.7% as the human-level performance because it would mean 0% of avoidable bias (0.7% training set error - 0.7% the human-level performance = 0%).

It wouldn’t give us any intuition about which problem to address first: high bias vs high variance. In the example from the video, it makes sense to work on high bias first, because we would have a chance to improve our learning algorithm up to 0.2% (0.7% training set error - 0.5% the human-level performance). When working on high variance gives us only up to 0.1% (0.8% development set error - 0.7% training set error).

If we were choosing the human-level performance of 0.7% or 1%, we would miss that (0.7% - 0.7% = 0% of avoidable bias comparing to 0.8% - 0.7% = 0.1% suggests working on the high variance problem when we really have problem with high bias).

The rule of thumb – choose the human-level performance as a proxy for Bayes error, meaning the lowest error proved by experiments.

Statement 2: How machine learning problem becomes harder when training error approaches bayes error ? How proffesor saying that ? If the training error equal to bayes error means 0% avoidable bias and so we directly reduce variance problem right. I dont know why they are saying harder to use tools reduce bias / variance.

It makes the problem harder in terms of determining the problem we have: high bias vs high variance. These problems require different actions from us. Check out this post on actions that we might perform to address these problems.

Statement 3: In fact, if all you knew was that a single typical doctor achieves 1% error, and it might be very difficult to know if you should be trying to fit your training set even better. Can u please explain this statement ?

Trying to fit your training set even better means to work on the high bias problem. It’s just wording.

2 Likes