[Paper Review] CNNs for Surveillance Footage Scene Classification

Utkarsh Contractor, Chinmayi Dixit+1 more|arXiv (Cornell University)|Sep 8, 2018

Anomaly Detection Techniques and Applications16 references4 citations

TL;DR

This paper proposes a CNN-based approach for classifying surveillance video frames to detect abandoned luggage, leveraging transfer learning on CAVIAR and i-LIDS datasets. It achieves 98% accuracy by fine-tuning architectures like ResNet and DenseNet, with saliency maps revealing that models rely more on scene context than object-specific features, limiting generalization but enabling high performance in specific environments.

ABSTRACT

In this project, we adapt high-performing CNN architectures to differentiate between scenes with and without abandoned luggage. Using frames from two video datasets, we compare the results of training different architectures on each dataset as well as on combining the datasets. We additionally use network visualization techniques to gain insight into what the neural network sees, and the basis of the classification decision. We intend that our results benefit further work in applying CNNs in surveillance and security-related tasks.

Motivation & Objective

• Address the challenge of automating surveillance video analysis to reduce human workload and error in detecting anomalous events like abandoned luggage.
• Overcome limitations of existing methods that rely on background subtraction or motion tracking, which are sensitive to environmental variations and lack generalization.
• Develop a deep learning-based solution using CNNs that can be fine-tuned for specific surveillance environments with limited data.
• Investigate whether CNNs can learn to detect abandoned luggage effectively despite variations in camera angle, resolution, and background complexity.
• Understand the decision-making process of the model through visualization techniques to assess whether it learns object-specific features or scene-level patterns.

Proposed method

• Fine-tuned pre-trained CNN architectures (ResNet, DenseNet, VGG) on two surveillance datasets: CAVIAR and i-LIDS, focusing on frames with and without abandoned luggage.
• Trained models on individual datasets and on the combined dataset to evaluate performance and generalization across environments.
• Applied data augmentation techniques (e.g., random cropping, flipping) to increase training data diversity and improve model robustness.
• Generated saliency maps using gradient-based visualization to interpret model attention and determine which image regions influenced classification decisions.
• Used class activation maps and network visualization to analyze whether the model learned to detect the object or was influenced by background anomalies.
• Evaluated model performance using accuracy, false positive, and false negative rates to assess reliability in real-world deployment scenarios.

Experimental results

Research questions

• RQ1 Can pre-trained CNNs be effectively fine-tuned for abandoned luggage detection in surveillance footage with limited labeled data?
• RQ2 How does model performance vary when trained on a single dataset versus a combination of CAVIAR and i-LIDS datasets?
• RQ3 To what extent do saliency maps reveal whether the model detects the luggage object or is influenced by background scene features?
• RQ4 What causes false positives and false negatives in the model’s predictions, and how do these relate to visual context or image anomalies?
• RQ5 Can a model trained on one environment generalize to another, or is it inherently dependent on scene-specific patterns?

Key findings

• The model achieved a high accuracy of 98% on the combined dataset, demonstrating that fine-tuning on limited surveillance data can yield strong performance.
• False negatives occurred when luggage was near other static objects or partially obscured, indicating limitations in depth perception and spatial reasoning.
• False positives were often triggered by background anomalies such as camera flashes or lighting changes, suggesting the model learned spurious scene-level patterns.
• Saliency maps revealed that the model activated across most of the image, especially in dynamic scenes, indicating it relied on overall scene context rather than localized object features.
• Class visualizations showed that the model learned scene-specific patterns rather than generalizable object features, explaining its poor generalization across different environments.
• Despite limited data, data augmentation significantly improved model robustness and helped create a more balanced input distribution, enhancing training stability and performance.