[Paper Review] Diversity Regularized Spatiotemporal Attention for Video-based Person Re-identification

Shuang Li, Sławomir Bąk+2 more|arXiv (Cornell University)|Mar 27, 2018

Video Surveillance and Tracking Methods24 references51 citations

TL;DR

The paper introduces a spatiotemporal attention framework with multiple diverse spatial attention models and temporal attention, regularized by a diversity term, to improve video-based person re-identification. It outperforms state-of-the-art on PRID2011, iLIDS-VID, and MARS.

ABSTRACT

Video-based person re-identification matches video clips of people across non-overlapping cameras. Most existing methods tackle this problem by encoding each video frame in its entirety and computing an aggregate representation across all frames. In practice, people are often partially occluded, which can corrupt the extracted features. Instead, we propose a new spatiotemporal attention model that automatically discovers a diverse set of distinctive body parts. This allows useful information to be extracted from all frames without succumbing to occlusions and misalignments. The network learns multiple spatial attention models and employs a diversity regularization term to ensure multiple models do not discover the same body part. Features extracted from local image regions are organized by spatial attention model and are combined using temporal attention. As a result, the network learns latent representations of the face, torso and other body parts using the best available image patches from the entire video sequence. Extensive evaluations on three datasets show that our framework outperforms the state-of-the-art approaches by large margins on multiple metrics.

Motivation & Objective

• Motivate robust video-based person re-id despite occlusions and misalignment in frames.
• Develop a spatiotemporal attention network that automatically discovers diverse informative body parts.
• Regularize spatial attention to prevent redundancy and promote distinct part detectors.
• Aggregate region-level features across time with per-part temporal attention to form a compact video descriptor.

Proposed method

• Use restricted random sampling to select N frames from each video.
• Implement multiple spatial attention models to localize discriminative regions within each frame.
• Apply a diversity regularization term based on the Hellinger distance to encourage diverse, minimally overlapping receptive fields.
• Compute per-frame spatial features, then aggregate them across time with per-part temporal attention, producing x_k for each spatial part.
• Concatenate x_k across K parts and supervise with the Online Instance Matching (OIM) loss.

Experimental results

Research questions

• RQ1 Can a set of diverse, automatically discovered spatial attention models improve frame alignment and occlusion handling in video re-id?
• RQ2 Does diversity regularization (based on Hellinger distance) reduce redundancy among part detectors and improve recognition accuracy?
• RQ3 How does per-part temporal attention compare to global pooling for aggregating region features across a video?
• RQ4 What is the impact of the number of spatial attention models (K) on re-id performance?
• RQ5 How does the proposed method compare to state-of-the-art on standard video re-id datasets?

Key findings

• Outperforms state-of-the-art on PRID2011, iLIDS-VID, and MARS across rank-1 accuracy and mAP (for MARS).
• Diverse spatial attention (K>1) yields better performance, with maximum gains around K=6 in experiments.
• Diversity regularization using the Hellinger distance improves discriminability of detected parts and reduces overlap.
• Temporal attention at the per-part level yields stronger performance than per-frame or global pooling baselines.
• Final SpaAtn+Q+TemAtn+Ind setup achieves 93.2% (PRID2011), 80.2% (iLIDS-VID), 82.3% (MARS) in rank-1; MARS mAP: 65.8.
• Compared with Baseline and several variants, the proposed approach consistently improves accuracy across datasets.