Hello @ai_curious,
You have many neighbours! I think these are the “interesting” photos you want to check on yourself.
So the problem is too many false positives?
Usually I prefer to make specific suggestion after looking at the data, because then the chance that it makes sense will be higher. However, I think I can rely on you to filter out the senseless part and perhaps turn them into feasible ideas 
I assume you have still cameras, and in the view of each camera, the overall background is more or less the same at the same hour of day, except may be due to some weather condition. So I was wondering, if you can get 100 “uninteresting photos” during the day, and another set of 100 during the night. Then you take an average on the day-set and another average on the night-set. Then when preparing your dataset for training or inference, you subtract any day-image with the day-average, and subtract any night-image with the night-average. In this way, you can get rid of a lot of useless background information at training/inference.
I talked about only a day-set and a night-set, but we can have 4 sets per day or 3 sets per day. We may need to redefine the hours for each set depending on the seasons.
What do you think?
Raymond