[Paper Review] Multiview Detection with Shadow Transformer (and View-Coherent Data Augmentation)

Yunzhong Hou, Liang Zheng|arXiv (Cornell University)|Aug 12, 2021

Video Surveillance and Tracking Methods43 references4 citations

TL;DR

This paper proposes MVDeTr, a multiview object detection system that uses a novel shadow transformer to adaptively aggregate features across multiple camera views, addressing view-specific distortions caused by projection. The method introduces view-coherent data augmentation to maintain multiview consistency during training, achieving state-of-the-art performance on Wildtrack and MultiviewX benchmarks.

ABSTRACT

Multiview detection incorporates multiple camera views to deal with occlusions, and its central problem is multiview aggregation. Given feature map projections from multiple views onto a common ground plane, the state-of-the-art method addresses this problem via convolution, which applies the same calculation regardless of object locations. However, such translation-invariant behaviors might not be the best choice, as object features undergo various projection distortions according to their positions and cameras. In this paper, we propose a novel multiview detector, MVDeTr, that adopts a newly introduced shadow transformer to aggregate multiview information. Unlike convolutions, shadow transformer attends differently at different positions and cameras to deal with various shadow-like distortions. We propose an effective training scheme that includes a new view-coherent data augmentation method, which applies random augmentations while maintaining multiview consistency. On two multiview detection benchmarks, we report new state-of-the-art accuracy with the proposed system. Code is available at https://github.com/hou-yz/MVDeTr.

Motivation & Objective

• To address the limitations of convolution-based multiview feature aggregation, which treats all positions uniformly despite varying projection distortions.
• To design a learnable attention mechanism that is aware of both position and camera view to better handle shadow-like distortions in projected feature maps.
• To develop a data augmentation strategy that preserves multiview consistency while applying random transformations, enabling effective training.
• To improve detection accuracy in occluded scenarios by leveraging joint multiview supervision through a unified detection head.
• To validate the effectiveness of view-coherent augmentation and per-view supervision in improving generalization and reducing overfitting.

Proposed method

• The core architecture uses a shadow transformer that attends across multiple camera views and positions simultaneously, using shared ground plane position embeddings and learned camera embeddings.
• The shadow transformer operates on projected feature maps from multiple views, enabling view- and position-aware feature fusion without translation invariance.
• A novel view-coherent data augmentation strategy applies different random affine transformations to each view while preserving multiview consistency by inverting transformations before projection.
• The system uses per-view supervision with adjusted ground truth to maintain consistency during augmentation and improve training stability.
• The detection head is a fully connected layer that predicts object locations and bounding boxes directly on the ground plane, with auxiliary losses for offset and bounding box regression.
• The training pipeline integrates standard detection losses (e.g., from RetinaNet) with view-specific supervision and the proposed augmentation scheme.

Experimental results

Research questions

• RQ1 Can a self-attention mechanism that is aware of both spatial position and camera view outperform translation-invariant convolutions in multiview feature aggregation?
• RQ2 Does view-coherent data augmentation improve generalization in multiview detection without breaking multiview consistency?
• RQ3 How do per-view supervision and auxiliary losses affect detection performance in multiview settings?
• RQ4 To what extent do offset and bounding box regression losses improve detection accuracy in multiview systems?
• RQ5 Can a transformer-based architecture achieve state-of-the-art performance on multiview detection benchmarks while handling projection distortions?

Key findings

• MVDeTr achieves new state-of-the-art performance on both Wildtrack and MultiviewX benchmarks, with a 1.7% and 0.7% improvement in MODA over the previous SOTA method, MVDet.
• Removing the view-coherent data augmentation leads to a 2.0% and 0.6% drop in MODA on Wildtrack and MultiviewX, respectively, demonstrating its critical role in performance.
• The removal of per-view supervision causes a 1.6% and 1.1% drop in MODA on Wildtrack and MultiviewX, indicating its effectiveness in reducing overfitting and improving generalization.
• The offset loss is more important for MultiviewX (0.6% MODA drop when removed), likely due to dense pedestrian arrangements and NMS-induced false negatives.
• Bounding box regression loss contributes equally to both datasets, with a 0.5% MODA drop when removed, showing its value in accurate localization.
• Replacing multiview deformable attention with standard deformable attention reduces performance by 1.7% and 0.7% on Wildtrack and MultiviewX, respectively, proving the superiority of the proposed attention mechanism.