[Paper Review] Geometry-Aware Video Object Detection for Static Cameras

Dan Xu, Weidi Xie+1 more|arXiv (Cornell University)|Sep 6, 2019

Advanced Neural Network Applications36 references4 citations

TL;DR

This paper proposes GAST-Net, a geometry-aware spatio-temporal video object detection framework for static cameras that leverages pseudo depth maps derived from object scale to improve multi-scale detection. By extending the anchor-free CornerNet to video with temporal modeling and geometry-guided feature fusion, GAST-Net achieves state-of-the-art performance on both synthetic (Carla) and real-world (DukeMTMC) datasets, outperforming existing one- and two-stage detectors by up to 6.2 points mAP.

ABSTRACT

In this paper we propose a geometry-aware model for video object detection. Specifically, we consider the setting that cameras can be well approximated as static, e.g. in video surveillance scenarios, and scene pseudo depth maps can therefore be inferred easily from the object scale on the image plane. We make the following contributions: First, we extend the recent anchor-free detector (CornerNet [17]) to video object detections. In order to exploit the spatial-temporal information while maintaining high efficiency, the proposed model accepts video clips as input, and only makes predictions for the starting and the ending frames, i.e. heatmaps of object bounding box corners and the corresponding embeddings for grouping. Second, to tackle the challenge from scale variations in object detection, scene geometry information, e.g. derived depth maps, is explicitly incorporated into deep networks for multi-scale feature selection and for the network prediction. Third, we validate the proposed architectures on an autonomous driving dataset generated from the Carla simulator [5], and on a real dataset for human detection (DukeMTMC dataset [28]). When comparing with the existing competitive single-stage or two-stage detectors, the proposed geometry-aware spatio-temporal network achieves significantly better results.

Motivation & Objective

• To address scale variation and occlusion in video object detection under static camera settings where scene geometry is exploitable.
• To extend single-stage anchor-free detection (CornerNet) to video by modeling spatio-temporal dependencies efficiently.
• To integrate scene geometry—derived from object scale on the image plane—into deep networks for improved multi-scale feature selection and prediction.
• To validate the effectiveness of geometry priors in video object detection on both synthetic and real-world datasets.
• To demonstrate that geometry-aware design leads to superior detection accuracy compared to standard detectors.

Proposed method

• The model processes video clips as input and predicts object corner heatmaps and embeddings only for the first and last frames, reducing computational cost while capturing long-range temporal context.
• A spatio-temporal convolutional backbone encodes appearance and motion features across frames, enabling joint modeling of spatial and temporal cues.
• Geometry-aware feature fusion is introduced, where pseudo depth maps (derived from object height in pixels) guide multi-scale feature selection and fusion in the network.
• The network uses a corner-based detection head that predicts top-left and bottom-right corners, followed by embedding-based grouping to form bounding boxes.
• Pseudo depth maps are computed from object scale on the image plane using known camera geometry, providing geometric priors without requiring explicit depth sensors.
• The architecture is end-to-end trainable and optimized using standard detection loss functions for keypoint regression and embedding contrastive learning.

Experimental results

Research questions

• RQ1 Can scene geometry derived from static cameras improve multi-scale video object detection?
• RQ2 How does incorporating pseudo depth maps as geometric priors affect detection accuracy and robustness to scale variation?
• RQ3 Can a corner-based, anchor-free detection framework be effectively extended to video with spatio-temporal modeling?
• RQ4 Does geometry-aware feature fusion outperform standard multi-scale feature aggregation in video object detection?
• RQ5 How does the proposed method compare to state-of-the-art one-stage and two-stage detectors on real-world and synthetic benchmarks?

Key findings

• On the DukeMTMC dataset, GAST-Net with ResNet-50 backbone achieves 74.42% mAP, outperforming Faster R-CNN by 3.74 points and Single-Frame CornerNet by 5.71 points.
• With VGG-11 backbone, GAST-Net achieves 68.26% mAP, surpassing SSD by 9.17 points and Single-Frame CornerNet by 6.77 points.
• The geometry-guided feature fusion module improves performance across all metrics, confirming that geometric priors enhance scale perception and detection accuracy.
• On the Carla synthetic dataset, GAST-Net demonstrates strong generalization, validating its effectiveness in autonomous driving scenarios with complex scale variations.
• The model achieves higher AP75 (68.21%) than Faster R-CNN (59.64%) on DukeMTMC, indicating superior localization accuracy due to dense corner prediction.
• Failure cases are primarily due to incorrect corner grouping in extreme scale or occluded scenarios, highlighting the need for improved grouping mechanisms.