I definitely agree with your point that it’s difficult to guarantee that the estimated error is realistic or statistically reasonable. However, let’s say you are a ML engineer and you are building a fraud detection system for a company. Let’s say that the company has a non-AI based fraud detection system that gives them an error of 10%, and they want you to build an AI system that will give an error of 5% at max.
Now, from the company’s perspective, I won’t spend lots of lots of dollars, just to make a system that only improves the error by 1-2%, and hence, it’s my job as a ML engineer, to make a model having an error rate of 5%, otherwise, the company will go elsewhere.
However, from your perspective and experience, an error rate of 5% won’t be realistic, so, you both agree with an error rate of 6.5%. Now, whether this is achievable or not, that you can only find out by making a system.
Note that in this case, the company won’t care about whether it has structured or unstructured data. If AI can’t prove to be better than it’s current system, it won’t simply give your company the contract to build one.
This is one of the key differences in traditional software and AI-based software. In the case of a traditional software, one can almost always deliver upto the expectations, but in the case of a AI-based software, the expectations are hard to define, and even if they are defined, there is always a great deal of uncertainty whether they will be met or not.
To further consolidate my point, I would like to cite something from the latest edition of the Batch:
Batch, May 18, 2022
Compared to traditional software that begins with a specification and ends with a deliverable to match, machine learning systems present a variety of unique challenges. These challenges can affect the budget, schedule, and capabilities of a product in unexpected ways.
How can you avoid surprising customers? Here’s a non-exhaustive checklist of ways that a machine learning system might surprise customers who are more familiar with traditional software:
- We don’t know how accurate the system will be in advance.
- We might need a costly initial data collection phase.
- After getting the initial dataset, we might come back and ask for more data or better data.
- Moreover, we might ask for this over and over.
- After we’ve built a prototype that runs accurately in the lab, it might not run as well in production because of data drift or concept drift.
- Even after we’ve built an accurate production system, its performance might get worse over time for no obvious reason. We might need help monitoring the system and, if its performance degrades over time, invest further to fix it.
- A system might exhibit biases that are hard to detect.
- It might be hard to figure out why the system gave a particular output. We didn’t explicitly program it to do that!
- Despite the customer’s generous budget, we probably won’t achieve AGI.
That’s a lot of potential surprises! It’s best to set expectations with customers clearly before starting a project and keep reminding them throughout the process.
I hope this helps.