[Paper Review] MODETR: Moving Object Detection with Transformers

Eslam Mohamed, Ahmad El Sallab|arXiv (Cornell University)|Jun 21, 2021

Advanced Neural Network Applications8 references4 citations

TL;DR

MODETR proposes a two-stream Transformer-based architecture for Moving Object Detection (MOD) that integrates spatial (RGB) and motion (optical flow or stacked RGB) features using multi-head self-attention across both modalities. By introducing Temporal Positional Encoding (TPE) to extend DETR’s Spatial Positional Encoding, MODETR achieves a 5% mAP improvement over state-of-the-art methods and a 10% mAP gain over baseline RGB-only models on the KITTI MOD dataset.

ABSTRACT

Moving Object Detection (MOD) is a crucial task for the Autonomous Driving pipeline. MOD is usually handled via 2-stream convolutional architectures that incorporates both appearance and motion cues, without considering the inter-relations between the spatial or motion features. In this paper, we tackle this problem through multi-head attention mechanisms, both across the spatial and motion streams. We propose MODETR; a Moving Object DEtection TRansformer network, comprised of multi-stream transformer encoders for both spatial and motion modalities, and an object transformer decoder that produces the moving objects bounding boxes using set predictions. The whole architecture is trained end-to-end using bi-partite loss. Several methods of incorporating motion cues with the Transformer model are explored, including two-stream RGB and Optical Flow (OF) methods, and multi-stream architectures that take advantage of sequence information. To incorporate the temporal information, we propose a new Temporal Positional Encoding (TPE) approach to extend the Spatial Positional Encoding(SPE) in DETR. We explore two architectural choices for that, balancing between speed and time. To evaluate the our network, we perform the MOD task on the KITTI MOD [6] data set. Results show significant 5% mAP of the Transformer network for MOD over the state-of-the art methods. Moreover, the proposed TPE encoding provides 10% mAP improvement over the SPE baseline.

Motivation & Objective

• To address the limitation of 2D convolutional networks in modeling inter-relations between spatial and motion features in moving object detection.
• To extend the DETR architecture to handle spatio-temporal features by incorporating motion cues from optical flow or consecutive RGB frames.
• To design a temporal encoding mechanism that enables Transformers to model sequential motion patterns effectively.
• To evaluate the performance of different motion integration strategies (RGB+OF vs. RGB+RGB) and TPE variants in a unified Transformer framework.

Proposed method

• Adapt the DETR encoder-decoder architecture into a two-stream design, with separate encoders for spatial (RGB) and motion (OF or stacked RGB) features.
• Apply multi-head self-attention in both spatial and motion streams using Spatial Positional Encoding (SPE) to model spatial relationships.
• Introduce a novel Temporal Positional Encoding (TPE) to extend SPE and enable modeling of temporal dynamics across consecutive frames.
• Explore two TPE strategies: early TPE (applied before fusion) and late TPE (applied after individual frame encoding), balancing speed and performance.
• Fusion of streams is performed via concatenation or 1x1 convolution before the object query-based decoder.
• Train the entire network end-to-end using bipartite matching loss for set prediction, enabling end-to-end optimization without post-processing.

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture effectively model both spatial and motion features for moving object detection, outperforming 2D CNN-based methods?
• RQ2 How does the integration of optical flow versus stacked RGB frames as motion cues affect detection performance in a Transformer-based MOD system?
• RQ3 What is the impact of different temporal encoding strategies (early vs. late TPE) on modeling motion dynamics in a two-stream Transformer?
• RQ4 Can a modified DETR with TPE achieve state-of-the-art performance on the KITTI MOD benchmark?
• RQ5 Does the attention mechanism in MODETR effectively focus on moving objects, and how does it differ across input modalities?

Key findings

• The RGB+OF variant of MODETR achieves 42.9% mAP on the KITTI MOD dataset, representing a 5% improvement over the best existing state-of-the-art method.
• The proposed Temporal Positional Encoding (TPE) provides a 10% mAP improvement over the baseline Spatial Positional Encoding (SPE) when used with the RGB+OF setup.
• Among RGB+RGB configurations, the early TPE variant achieves 33.3% mAP, outperforming late TPE (24.4%) and demonstrating the effectiveness of early temporal integration.
• The attention maps in MODETR clearly focus on moving parts such as wheels and object boundaries, especially in the RGB+OF setup, indicating effective attention to motion cues.
• MODETR outperforms both MODNet and InstanceMotSeg on the same benchmark, even though those models perform multi-task learning, suggesting the advantages of the end-to-end Transformer design.
• The model with RGB+OF input achieves 66.3% mAP@50, significantly outperforming the RGB-only baseline of 48.3% mAP@50, highlighting the value of explicit motion features.