[Paper Review] Active Mining of Parallel Video Streams

Samaneh Khoshrou, Jaime S. Cardoso+1 more|arXiv (Cornell University)|May 14, 2014

Data Stream Mining Techniques32 references3 citations

TL;DR

This paper proposes NEVIL, a semi-supervised active learning framework for incremental classification of video objects across parallel, evolving video streams in multi-camera surveillance. By adaptively selecting the most informative samples for human annotation and combining multiple classifiers using dynamic fusion rules, NEVIL achieves over 90% accuracy with less than 15% annotation effort on synthetic data and over 80% accuracy with under 30% labeling on real CAVIAR data, outperforming passive and baseline active learning methods.

ABSTRACT

The practicality of a video surveillance system is adversely limited by the amount of queries that can be placed on human resources and their vigilance in response. To transcend this limitation, a major effort under way is to include software that (fully or at least semi) automatically mines video footage, reducing the burden imposed to the system. Herein, we propose a semi-supervised incremental learning framework for evolving visual streams in order to develop a robust and flexible track classification system. Our proposed method learns from consecutive batches by updating an ensemble in each time. It tries to strike a balance between performance of the system and amount of data which needs to be labelled. As no restriction is considered, the system can address many practical problems in an evolving multi-camera scenario, such as concept drift, class evolution and various length of video streams which have not been addressed before. Experiments were performed on synthetic as well as real-world visual data in non-stationary environments, showing high accuracy with fairly little human collaboration.

Motivation & Objective

• To address the challenge of high annotation cost in multi-camera video surveillance where labeled data is scarce and expensive to obtain.
• To develop a robust, incremental learning framework that adapts to concept drift and class evolution in non-stationary visual environments.
• To minimize human labeling effort while maintaining high classification accuracy in parallel video streams with varying lengths and overlapping fields of view.
• To enable continuous learning without assumptions about camera overlap, supporting both overlapping and non-overlapping camera configurations.
• To evaluate the effectiveness of different classifier combinations, selection strategies, and fusion rules in real and synthetic surveillance data.

Proposed method

• The framework, named NEVIL, uses a semi-supervised incremental learning approach that processes video streams in batches, updating an ensemble of classifiers over time.
• It employs active learning strategies to select the most informative instances for human annotation, reducing labeling effort while maintaining performance.
• Multiple informativeness measures—such as modified margin and modified most confident—are used to guide oracle queries and prioritize uncertain or diverse samples.
• Classifier outputs are combined using fusion rules including sum rule, product rule, and geometric mean, with dynamic selection based on confidence and diversity.
• The system maintains a global object identity across cameras by tracking object appearances across multiple streams and aligning them temporally.
• The framework is evaluated using both synthetic data with controlled concept drift and real CAVIAR sequences with complex, high-dimensional visual features.

Experimental results

Research questions

• RQ1 How can active learning be effectively applied to evolving, multi-camera video streams with concept drift and class evolution?
• RQ2 Which combination of base classifiers, fusion rules, and informativeness measures yields the highest accuracy with minimal human annotation?
• RQ3 How does the performance of generative versus discriminative models vary across low-dimensional clean data versus high-dimensional noisy real-world video streams?
• RQ4 Can a unified framework handle both overlapping and non-overlapping camera views without prior assumptions about spatial configuration?
• RQ5 What is the optimal trade-off between labeling cost and classification accuracy in long-term, continuous video surveillance?

Key findings

• NEVIL achieves over 90% accuracy on synthetic datasets with less than 15% annotation effort, significantly outperforming passive learning and baseline active learning methods.
• On real CAVIAR sequences, NEVIL reaches over 80% accuracy with less than 30% labeling, demonstrating strong robustness to noise and complexity in real-world data.
• For low-dimensional, clean synthetic data, generative models (e.g., naive Bayes) outperform discriminative models, highlighting their flexibility in low-noise settings.
• In high-dimensional, noisy real-world data, discriminative models like SVM and logistic regression outperform generative models, likely due to better handling of local maxima in likelihood estimation.
• The combination of arithmetic mean (SUM) fusion and modified margin (MM) selection criteria yields the best performance on real data, while geometric mean and modified most confident perform best on synthetic data.
• The framework successfully handles concept drift and class evolution without prior assumptions about camera overlap, making it applicable to diverse surveillance configurations.