Relevance: Probability output for business decisions?

Hi
In the Week1 of Course 2 in ML specialization (screenshot attached)., in the demand prediction slide, it is said that prediction is given as “probability in % that a tshirt shall be topseller.”

Is there a way to convert these probabilities to robust numbers for Deep learning specialists to advise businesses to help in their decision making?

for eg., a probability percentage output of 55 is too close for Yes/No scenario for a concrete decision.

@tennis_geek, edge cases being shown as edge cases is a good thing, because it indicates that your model + data(features) isn’t good enough to completely distinguish from being topseller and not being topseller. In other words, to reduce edge cases, you need to improve your model and data. There is no other way.

Raymond

1 Like

Thanks @rmwkwok!
After looking at your edge cases reply, wondering if anyone has built ML models (may not have the same model architectures as previous iterations) specifically for these edge cases to further enhance decision making, as function of probability prediciton output of course

If your idea is to train the first model to identify the subset of edge cases from your full training set, and then fit the second model to the subset of edge case, it sounds to be interesting but I have never tried it and haven’t thought it through. You may try it.

1 Like

However @tennis_geek , I have to warn you that please do evaluate both approaches to see whether having a second model does make a good difference. Simply “rewriting” the probability numbers can fool ourselves. I want to iterate again the following:

edge cases being shown as edge cases is a good thing, because it indicates that your model + data(features) isn’t good enough to completely distinguish from being topseller and not being topseller.

Just one last point to add, knowing some cases to be edge cases mean that you can make different business decisions for them.

1 Like

Couple of thoughts here. First, even a highly confident prediction by an ML model should be subjected to sanity check before committing business resources. I would always assume the model is wrong, and try to find out through other analyses what in the data supports its outcome. Second, an inconclusive outcome would make me look back at the customer data to see if there are clusters. Are there subpopulations that yield different predictions? What about price sensitivity? Maybe a limited A/B tracking clickthroughs to a page where the product concept is depicted (but not necessarily orderable) would more offer insight into consumer proclivities. I guess ‘doubt but verify’ was my motto.

[quote=“ai_curious, post:7, topic:224543”] @rmwkwok
Second, an inconclusive outcome would make me look back at the customer data to see if there are clusters. Are there subpopulations that yield different predictions? What about price sensitivity? Maybe a limited A/B tracking clickthroughs to a page where the product concept is depicted (but not necessarily orderable) would more offer insight into consumer proclivities. I guess ‘doubt but verify’ was my motto.
[/quote]

identify, incorporate & test: Unique features which would more aptly apply for specific Edge cases which ‘may not’ be required for so-called easier data points i.e, high conclusive ones.

That is what a (prediction) simulation is for with trained params to validate with a known test data for some trials?

Also, any quantitative model with good Information Criterion (IC) scores (Akaike, Bayesian) are best model candidates given static sized data and variable features? This also would help in suggesting the right candidate models for business?

@tennis_geek, it’s time for you to give your idea a try and convince yourself with solid results whether your idea is significant or not. My suggestion is to plan ahead how you would fairly compare your 2-model approach with the 1-model approach.

Indeed @ai_curious. Unlike language/image which are relatively stable over short to mid range of time, business decision related environment are always subject to change so modeling that environment requires a more agile approach.

on it!
Thanks for renforcing the thought.

Good luck @tennis_geek On your route to building the 2nd model, it’s natural for anyone to one-click apply existing ML model for setting up a baseline, but if it doesn’t give you anything interesting, maybe analyzing edge cases can give you insights on how you would want to improve your dataset. Remember it is a product of both the model assumption and the data, so both of them can be improved, including the data especially if you are working in a business. Good thing about working in a business is that you can explore more, and it’s something we can’t have playing with a fixed online-shared dataset.