What detects the anomalies for testing the anomaly detector?

In the last few videos on anomaly detection it’s mentioned that tuning & testing an anomaly detector requires the test set or cross-validation set to have some labeled anomalies in it. If anomalies are so rare then how are they found so that they can be labeled? Obviously the anomaly detector itself can’t be used to find them; that would be cheating. The only way I can think of is to use anomalies that made themselves known by their real-world consequences, like the last transactions on a credit card before it got reported stolen or the manufacturer’s old testing records on a jet engine that caught fire. But is that really how it’s done? That way of collecting data seems like it would introduce bias - not to mention that it’s less than ideal to leave the anomaly detector untested until something goes horribly wrong.

1 Like

One could use a detailed and expensive expert investigation of a small population to discover the anomalies.

Evidence from past performance or failures could also be used.

Yes, that’s one method.