Categories from error analysis Vs categories from baseline performance

Course Q&A Machine Learning in Production
,
1

Hi, first time posting a question here, please bear with me.

I’m watching “Machine Learning in Production” course, now in Week 2: Modeling Challenges and Strategies. The doubt is specifically in “Prioritizing what to work on”. The very first slide shows the table with different dataset categories and their performance by the system, the baseline (H.L.P.), the gap, etc.

The voice explanation starts with:

In the last video, you learned about brainstorming and tagging your data with different attributes. Let’s see how you can use this to prioritize where to focus your attention.

Now, the last video is “Error analysis example”, and my understanding is that through error analysis you can discover categories within your dataset. You can categorize error examples, and by extending this effort you can then categorize all examples, even the correct ones. The categories from error analysis are some set that one discovers, e.g.: A, B, C, D.

The categories shown in baseline performance comparison should not necessarily be the same as those from error analysis. It is not yet 100% clear to me where do they come from. My first guess is that they come from the baseline. The whole baseline dataset, or their “published” performance are already segmented by those categories, e.g. W, X, Y, Z.

So I take my own dataset, if it’s not the one from the baseline, and I segment my samples by the same categories, so that I can compare my system performance and baseline performance on those same subsets.

Now, back on work prioritization. To me, what that initial voice explanation implies is that now I have categories A, B, C, D discovered through error analysis, and to proceed further I need to figure out how does the baseline performs on those categories, not the W, X, Y, Z originally used to segment performance. So, it implies that I should take the baseline dataset, tag it according to my error analysis categories, and split performance on my error analysis categories.

This part seems strange to me. So I guess I misunderstood something along the way.

A different explanation that I gave myself is:

the initial voice explanation is just a “verbal hook” to connect previous and current video contents. In reality, when prioritizing what to work on, one starts by looking at performance comparison w.r.t. the baseline, and error analysis categories are not involved at all. In this case, it might be worth to better clarify the starting point of such prioritization activity.

So a couple of questions that should help me understand are:

  • A) when showing performance comparison w.r.t. the baseline, where do those categories come from?
  • B) in order to prioritize what to work on, should I use categories from error analysis? Or from the baseline (if they are not the same thing)?

Feel free to explain/clarify better, or to ask for more details.

Thanks for your collaboration.

2

I exposed the same question to an AI assistant, and bounced back and forth a little to reason on this. Here is a summary of the conclusions I (…) reached:

\start

The categories used in the baseline comparison are the same as those resulting from error analysis: there are no two separate sets.

A complete per-category comparison is only possible when you have control over the dataset and the baseline, e.g., with HLP measured specifically on the same, already tagged dataset. Here, “complete per-category comparison” means that you have your own categories, discovered by error analysis, and over those you also have the baseline performance, gap, etc.

When the baseline is external (literature, public benchmarks, third-party ML system), a per-category comparison can only be performed if the baseline published categories match your own, or if you have access to the baseline predictions or model on the same dataset. Otherwise, the comparison remains aggregated, and prioritization is based on the aggregate gap plus the internal category analysis, weighted by product impact and ease of intervention.

\end

If anyone has time, I’d be interested in knowing if this sounds right. Thanks

3

Welcome to the forum! Thanks for posting such a detailed, thoughtful first question and your final summary is essentially correct. Let me confirm the two points.

A) Where do the categories come from? From your own error analysis. There’s no second category set. The tags you brainstormed and applied in the previous video (clean speech, car noise, etc.) are exactly the categories in the prioritization table — Andrew’s opening line is literal, not just a verbal hook.

So where does the baseline column come from? This is the key: the baseline is HLP, measured on your own tagged dev set. Humans perform the task on the same slices your model was evaluated on, giving you a per-category comparison on the same data. You never re-tag someone else’s dataset.

B) Which categories drive prioritization? The error analysis ones — they’re the only set in play. Workflow: tag dev set → measure your system per tag → measure HLP (or another system you can run) on the same slices → compute gaps → weight by % of data, business impact, and ease of improvement.

Your AI-assisted summary is right: per-category comparison requires a baseline you can evaluate on your data (trivially true for HLP). With an external published baseline, you’re stuck with aggregate comparison unless their slices match yours or you can get their predictions — in which case you prioritize on the aggregate gap plus your internal category analysis, as you wrote.

Treat “baseline” here as a measurement you perform, not an artifact you import. Then the table is just your dev set sliced one way, with two measurements per slice.

4

@sanjaypsachdev thanks for taking the time to reply and explain. Now I think I grasped this topic. Regards