Detecting data issues lecture

In the detecting data issues lecture, there is a slide ‘detecting distribution skew’. I didn’t find the explanation very intuitive, but from looking at the formulas on the slide, I think the idea is simply that - Dataset Shift (change in joint probability of x,y) is either caused by Covariate shift (aka Data drift) or Concept drift, as illustrated through formulas.

Is that a fair takeaway from the slide?

Dear Uditgt,

Thank you very much for your question. Yes you got that right. You can think of it this way. Imagine you have a mechanical system which is performing a specific task. While reading data from this mechanical system you will realise that, as an example, the applied force in the joints or motors increases because the friction in this part of the system has increased, which is due to overheating. If you train a model for anomaly detection (better said predictive maintenance) on the read applied force from the state where the motors and joints are cold, then you would have so many false positives that the system is having a problem as the system operates, which is due to the changing condition of your system (impact of the friction in the system). In this example you have a gradual distribution shift in the read applied force to your system.
I hope that helps!

All the best,
Kiavash :slight_smile: