[Paper Review] Deep Multi-camera People Detection

Tatjana Chavdarova, François Fleuret|arXiv (Cornell University)|Feb 15, 2017

Video Surveillance and Tracking Methods20 references5 citations

TL;DR

This paper proposes the first end-to-end deep learning framework for multi-camera people detection that jointly fuses appearance features across multiple synchronized, overlapping camera views. By fine-tuning a monocular pedestrian detector on a large-scale single-view dataset and then jointly training a multi-view architecture on a smaller multi-camera dataset, the method achieves state-of-the-art performance on the PETS 2009 benchmark, outperforming existing methods by large margins while avoiding background subtraction and handling occlusions more robustly.

ABSTRACT

This paper addresses the problem of multi-view people occupancy map estimation. Existing solutions for this problem either operate per-view, or rely on a background subtraction pre-processing. Both approaches lessen the detection performance as scenes become more crowded. The former does not exploit joint information, whereas the latter deals with ambiguous input due to the foreground blobs becoming more and more interconnected as the number of targets increases. Although deep learning algorithms have proven to excel on remarkably numerous computer vision tasks, such a method has not been applied yet to this problem. In large part this is due to the lack of large-scale multi-camera data-set. The core of our method is an architecture which makes use of monocular pedestrian data-set, available at larger scale then the multi-view ones, applies parallel processing to the multiple video streams, and jointly utilises it. Our end-to-end deep learning method outperforms existing methods by large margins on the commonly used PETS 2009 data-set. Furthermore, we make publicly available a new three-camera HD data-set. Our source code and trained models will be made available under an open-source license.

Motivation & Objective

• To address the limitations of existing multi-camera people detection methods that rely on background subtraction, which degrades in crowded scenes due to interconnected foreground blobs.
• To leverage deep learning for multi-view people detection, overcoming the lack of large-scale multi-camera datasets by transferring knowledge from large monocular datasets.
• To develop a joint multi-view detection architecture that processes multiple camera streams in parallel and learns cross-view feature correspondences automatically.
• To release a new, high-quality three-camera HD dataset with improved calibration for future research.

Proposed method

• Fine-tune a state-of-the-art monocular pedestrian detection network (e.g., R-CNN or Faster R-CNN) on a large-scale single-view pedestrian dataset to create a robust base model.
• Construct a multi-view deep neural network by sharing early convolutional layers across views and connecting them with shared, trainable fusion layers to learn cross-view feature mappings.
• Train the joint multi-view network on a smaller, multi-camera dataset, initializing the early layers with the fine-tuned monocular model weights to improve convergence and performance.
• Use an end-to-end training strategy that jointly optimizes for detection confidence and localization accuracy across all views without relying on hand-crafted features or background subtraction.
• Apply contextual padding and data normalization to improve robustness to calibration inconsistencies across views, especially in scenes with slopes.
• Utilize a multi-fold cross-validation strategy on the PETS 2009 dataset to evaluate generalization, with separate training and testing splits to assess performance stability.

Experimental results

Research questions

• RQ1 Can an end-to-end deep learning approach outperform existing multi-camera people detection methods that rely on background subtraction and hand-crafted features?
• RQ2 Does joint processing of multiple camera views using deep features lead to improved detection accuracy and robustness in crowded scenes?
• RQ3 Can a model pre-trained on a large monocular dataset generalize effectively when fine-tuned on a smaller multi-camera dataset for joint detection?
• RQ4 To what extent do calibration inconsistencies across multiple views affect performance, and can the model learn to mitigate these effects?
• RQ5 Does using all seven views of the PETS 2009 dataset, including those with known calibration issues, improve detection performance compared to using only a subset?

Key findings

• The proposed method outperforms existing state-of-the-art multi-view methods on the PETS 2009 dataset, achieving a mean average precision (MODP) of 0.68 when using all seven views, compared to 0.66 for the best prior method.
• The method achieves a mean average detection accuracy (MODA) of 0.94 when fine-tuned on a subset of the PETS 2009 data, significantly outperforming prior methods that report MODA values around 0.75–0.79.
• Even when using all seven views—including those with known calibration inconsistencies—the method shows marginal performance improvement rather than degradation, suggesting robustness to calibration errors.
• The model generalizes well without requiring dataset-specific normalization or fine-tuning, achieving a MODP of 0.75 on the full test sequence without prior fine-tuning.
• The use of appearance-based features across views allows the model to filter out non-human foreground objects that commonly trigger false positives in background subtraction-based methods.
• The method demonstrates strong performance stability across multiple random train-test splits, with low standard deviation in performance metrics (e.g., 0.02 for MODP and 0.01 for precision in the fine-tuned setting).