[Paper Review] Video Object Detection with an Aligned Spatial-Temporal Memory

Fanyi Xiao, Yong Jae Lee|arXiv (Cornell University)|Dec 18, 2017

Advanced Image and Video Retrieval Techniques4 citations

TL;DR

This paper proposes a Spatial-Temporal Memory Network (STMN) for video object detection that uses a novel Spatial-Temporal Memory Module (STMM) to integrate pre-trained ImageNet weights and a MatchTrans module for spatial-temporal alignment across frames. The method achieves state-of-the-art performance on ImageNet VID, with a 50.7% mAP using Fast-RCNN and VGG-16, demonstrating the critical role of pre-trained weights and motion-aware alignment in long-term video understanding.

ABSTRACT

We introduce Spatial-Temporal Memory Networks for video object detection. At its core, a novel Spatial-Temporal Memory module (STMM) serves as the recurrent computation unit to model long-term temporal appearance and motion dynamics. The STMM's design enables full integration of pretrained backbone CNN weights, which we find to be critical for accurate detection. Furthermore, in order to tackle object motion in videos, we propose a novel MatchTrans module to align the spatial-temporal memory from frame to frame. Our method produces state-of-the-art results on the benchmark ImageNet VID dataset, and our ablative studies clearly demonstrate the contribution of our different design choices. We release our code and models at http://fanyix.cs.ucdavis.edu/project/stmn/project.html.

Motivation & Objective

• To address the limitations of existing video object detectors that treat temporal information as post-processing or use fixed-length windows.
• To model long-term appearance and motion dynamics in videos while preserving spatial localization and feature integrity.
• To enable end-to-end learning of temporal dependencies by integrating pre-trained ImageNet weights into both feature and prediction heads.
• To improve detection robustness under challenging conditions like occlusion, pose variation, and motion blur by leveraging memory from earlier, easier frames.

Proposed method

• The core of the method is the Spatial-Temporal Memory Module (STMM), a convolutional recurrent unit that maintains spatially-aware memory across frames.
• The STMM fully integrates pre-trained weights from ImageNet for both the feature extractor and the prediction head, improving generalization and convergence.
• A novel MatchTrans module explicitly models 2D motion displacement between frames to align features in the spatial memory, enabling accurate pixel-level temporal aggregation.
• The memory is updated frame-by-frame using a gated recurrent mechanism with ReLU activation, replacing standard Sigmoid and Tanh to better match pre-trained weights.
• Region-of-interest (ROI) features are extracted directly from the aligned memory via ROI pooling, enabling efficient detection inference.
• The model is trained end-to-end with a standard detection head, and inference supports variable-length test windows to capture long-range dependencies.

Experimental results

Research questions

• RQ1 How does integrating pre-trained ImageNet weights into both the feature and prediction heads affect video object detection performance?
• RQ2 To what extent does spatial-temporal alignment via MatchTrans improve detection accuracy under motion and viewpoint variation?
• RQ3 How does the performance of the proposed STMM compare to standard ConvGRU when using the same pre-trained weights?
• RQ4 What is the impact of test window length on detection accuracy, and how does memory utilization scale with sequence length?
• RQ5 Can the proposed method generalize across different backbone networks and base detectors while maintaining state-of-the-art performance?

Key findings

• The proposed STMN achieves a 50.7% mAP on the ImageNet VID validation set using Fast-RCNN and VGG-16, outperforming all ablated baselines.
• Removing the MatchTrans module (STMN-No-MatchTrans) leads to a 1.7% drop in mAP (to 49.0%), demonstrating the importance of spatial alignment.
• Using ConvGRU with pre-trained weights (ConvGRU-Pretrain) yields 48.0% mAP, but the full STMN with ReLU and pre-trained weights achieves 50.7%, showing the benefit of architectural compatibility.
• The model's performance improves with longer test windows: mAP increases by 1.0% at window size 15 compared to size 7, indicating effective long-range memory utilization.
• The computational overhead of the STMM is minimal, adding only 0.028 seconds per frame (0.31s for 11 frames) on a Titan X GPU.
• Ablation studies confirm that initializing both the feature and prediction heads with pre-trained weights is critical—randomly initialized heads (ConvGRU-FreshFC) yield only 44.8% mAP.