Unsupervised Learning on Meta data in Camera Surviellence

AI Discussions
1

I’m working on a project to detect anomalies in camera functionality. The goal is to alert if the camera is not working properly based on logs generated by the camera. Among the log data columns, the main relevant one is quality_of_img (which measures things like blurriness and darkness), along with a few other numeric columns. I do not have labeled examples of anomalies in advance.

Given this situation, what is the best approach to detect anomalies in an unsupervised way? One approach I am considering is to first create profiles of normal behavior (for example, the typical number of images captured per day and the usual distribution of quality_of_img values). Then, I would apply unsupervised anomaly detection methods such as Isolation Forest, clustering algorithms, or autoencoders to identify observations that significantly deviate from these learned normal patterns.

Does this approach sound appropriate for this scenario, or are there additional methods or considerations you would recommend?

2

Anomaly detection sounds good to me if anomalies happen rarely.

Possibly, you can use unsupervised clustering to group the images based on their quality and then some supervised method (maybe neural nets) to detect the occurrence of each abnormality if you have many occurrences for each of the clusters.

3

Thank you for the suggestion but if you could please let me confirm that I understand the pipeline correctly

first is unsupervised clustering: to group images by quality or other similarity (e.g., into clusters of normal and potentially abnormal images).
normal profiling or manually labelling ?: selecting the cluster(s) representing normal images and learning their characteristics
training a model: (e.g., One-Class SVM, autoencoder, or other novelty detection method) exclusively on these normal examples to capture the distribution of normal data
comparison and anomaly detection: at inference, comparing new images to this learned profile, where significant deviation (e.g., high reconstruction error or distance outside the normal boundary) would trigger an anomaly alert.

Is this understanding correct?

Also, to confirm only if anomalies are relatively rare and normal data is consistent, then training only on normal profiles and comparing new data is an effective correct?

Thanks again for clarifying.

4

No need for labelling in Clustering.

Probably neural nets (CNNs or suitable for images) if balanced groups or close to balanced groups are created from clustering.

Check anomaly detection algorithm on Machine Learning Specialization, others are there as well! It’s not effective as you describe.

5

The problem is I’m not dealing with images I’m dealing with meta data logs

6

No problem, a classification model either neural network, or perhaps better a tree based model (random forest)!