I’m have done some research and found that there is an sub-field of AI called “Explainable AI”.
I would like to here other’s opinion about this field and it would be great if you can give me your view about the potential of XAI in the future, whether it’s worth pursuing or not
Greetings geekology,
This is a topic I spent quite a bit of time researching.
Explainable AI or xAI refers to a set of tools used to interpret machine learning models for non technical users.
One of the most popular tools is called SHAP. It’s a Python library whose full name is SHapley Addative exPlanations. It was named in honor of Lloyd Shapley, who introduced cooperative game theory in 1951 and won the Nobel Memorial Prize in Economic Sciences for it in 2012
SHAP treats each feature in your model as a player on a team. It calculates and then fairly distributes the marginal contribution of each feature to the model output and assigns a SHAP Value to show feature importance. You can interpret the model locally or globally using various shap plots. I have a good bit of experience with it, so if you have any questions, please let me know.
Here are some other tools:
LIME - Local interpretable model-agnostic explanations (LIME).
PDP and ICE plots (partial dependence plots and individual conditional expectation plots)
Partial dependence plots (PDP) show the dependence between the target response and a set of input features of interest, marginalizing over the values of all other input features (the ‘complement’ features).
Similar to a PDP, an individual conditional expectation (ICE) plot shows the dependence between the target function and an input feature of interest. However, unlike a PDP, which shows the average effect of the input feature, an ICE plot visualizes the dependence of the prediction on a feature for each sample separately with one line per sample.
One author who covers this topic very throughly is Chrisoph Molnar. He’s very active on linked and also wrote a book on SHAP and interpretable machine learning. I highly recommend the SHAP book if your curious.
Hope this helps.
Cheers
2 Likes
I am not familiar with the tools and methods that Dan lists there, but I would interpret that as being part of the current state of the art in Explainable AI. It is still a general research topic and I don’t think the experts in that field believe that it is “solved” yet. The first context in which I heard it discussed was in the field of AI as applied to Medicine. Our own Prof Ng and Prof Fei-Fei Li of Stanford have had a discussion of AI applications in Medicine that at least mentions the topic. Consider cases like using an AI system to replace or augment the work of human doctors by reading radiology images or making diagnoses. You’re effectively asking people to trust that the AI system is behaving correctly. So if you can’t explain how the AI system arrives at its conclusions, then that is a cause for concern. If you’re trusting an AI to make decisions in cases in which the consequences involve life and death, then what is the process for proving that such a system is reliable?
There are also a couple of TED talks about the area that might be a good way to get an introduction to it.
2 Likes
I’ve been to a seminar in my Uni about this. Did not remember (or understand clearly haha) how it works underneath. But instead just giving you the result from a “black box”, it focuses more on explaining the Why (Why produces these kinds of outputs)
For example:
- Black-Box AI: input an image → output: “Dog”
- XAI (Explainable AI): input an image → output: “Dog”, Because: “this area of the images could be the tail, this area of the images could be the nose, or this area of the images could be the tooth, …”
This kind of AI, for me, would be helpful in some cases like health diagnosis, and risk analysis, … As it would be able to give more information besides just the output
Really fascinating if you think about it
2 Likes
Just hearing for the first time
I have also never herded this terminology before. Can you brief about your what you have learned from your research?
@ThongLai Personally I have not had a chance to try/experiment with this yet, but a thought I have had (in other contexts) is what if you added a ‘tag’ to each of the points in your dataset ? Obviously this would have to be something contextually meaningful to a human and would serve no role in any of your calculations-- but it would mean you could effectively follow that point through your network and get a better sense of what data points (ultimately discovered features) are providing more/less contribution to the result.
Otherwise, unfortunately, when it is just formulae it is a ‘black box’, but something like this might allow us to peek into it a little bit.
Note: The end result might not be a ‘statement’ of sorts, but maybe you could produce a heat map or similar-- At least something comprehensible.
I think this field is quite exciting, at least in the finacial perspective and your sharing helps me a lot.
Do you have any information about the problems this fields is facing (paper/blog… that mention) and how can it be upgraded in the future
Here is the paper from Lundberg for SHAP published at Neurips in 2017.
SHAP is already used by corporates who have a mature data strategy. I originally became familiar with SHAP through my mentor who uses it at Microsoft to present predictions for customer churn for one of their B2C brands.
As you see more adoption of AI for analyzing data, you’ll see more demand for explainability.
You will need a data scientist to be at the controls. The tools work really well, from my personal experience.
1 Like