[Paper Review] Densely Semantically Aligned Person Re-Identification

Zhizheng Zhang, Cuiling Lan+2 more|arXiv (Cornell University)|Dec 21, 2018

Video Surveillance and Tracking MethodsComputer Science54 references18 citations

TL;DR

This paper proposes a densely semantically aligned person re-identification framework that uses dense semantic mapping (from DensePose) to generate 24 densely aligned part images (DSAP-images), enabling fine-grained spatial and semantic alignment across varying poses and viewpoints. A two-stream network uses these DSAP-images in a guiding stream to regulate feature learning in a main full-image stream, achieving state-of-the-art performance: 95.7% rank-1 on Market1501, 90.4% on CUHK01, and 78.9% on CUHK03 under a new protocol, with inference requiring only the main stream for efficiency.

ABSTRACT

We propose a densely semantically aligned person re-identification framework. It fundamentally addresses the body misalignment problem caused by pose/viewpoint variations, imperfect person detection, occlusion, etc. By leveraging the estimation of the dense semantics of a person image, we construct a set of densely semantically aligned part images (DSAP-images), where the same spatial positions have the same semantics across different images. We design a two-stream network that consists of a main full image stream (MF-Stream) and a densely semantically-aligned guiding stream (DSAG-Stream). The DSAG-Stream, with the DSAP-images as input, acts as a regulator to guide the MF-Stream to learn densely semantically aligned features from the original image. In the inference, the DSAG-Stream is discarded and only the MF-Stream is needed, which makes the inference system computationally efficient and robust. To the best of our knowledge, we are the first to make use of fine grained semantics to address the misalignment problems for re-ID. Our method achieves rank-1 accuracy of 78.9% (new protocol) on the CUHK03 dataset, 90.4% on the CUHK01 dataset, and 95.7% on the Market1501 dataset, outperforming state-of-the-art methods.

Motivation & Objective

• To address persistent body misalignment in person re-identification caused by pose, viewpoint, occlusion, and detection errors.
• To enable fine-grained, pixel-level semantic alignment across person images using dense 3D surface correspondences (UV maps).
• To design a two-stream network where a guiding stream based on aligned part images improves feature learning in the main stream without affecting inference efficiency.
• To achieve state-of-the-art performance on standard benchmarks while maintaining computational efficiency in deployment.

Proposed method

• Leverages DensePose to estimate dense 2D-to-3D surface correspondences (UV coordinates) for each person image, enabling fine-grained semantic labeling at the pixel level.
• Generates 24 densely semantically aligned part images (DSAP-images) by warping the original image into a canonical UV space, ensuring spatially aligned semantics across images.
• Designs a two-stream network: a main full-image stream (MF-Stream) for primary feature learning, and a densely semantically-aligned guiding stream (DSAG-Stream) that uses DSAP-images to supervise and regulate the MF-Stream.
• Employs element-wise addition fusion between MF-Stream and DSAG-Stream features during training, with the DSAG-Stream discarded at inference to preserve efficiency.
• Uses joint training of global and part-aware branches in both streams to enhance complementary feature learning and robustness.
• Applies a multi-scale feature learning strategy with global and local branches to improve discriminative capacity.

Experimental results

Research questions

• RQ1 Can dense semantic alignment of person images via UV-space warping significantly reduce misalignment issues caused by pose and viewpoint variations?
• RQ2 Does using a guiding stream based on densely aligned part images improve the quality and robustness of features learned from the full image in person re-identification?
• RQ3 Can a two-stream architecture that discards the guiding stream at inference maintain high accuracy while improving computational efficiency?
• RQ4 How does dense semantic alignment compare to coarse part-based alignment in terms of re-ID performance?

Key findings

• The proposed method achieves 95.7% rank-1 accuracy on the Market1501 dataset, surpassing previous state-of-the-art methods.
• On the CUHK01 dataset, the method achieves 90.4% rank-1 accuracy, demonstrating strong performance on a challenging, diverse benchmark.
• On the CUHK03 dataset under a new protocol, the method achieves 78.9% rank-1 accuracy, outperforming prior methods by at least 10.9 percentage points in rank-1 accuracy.
• The ablation study confirms that dense semantic alignment provides a +1.6% gain in rank-1 accuracy over coarse alignment, highlighting the benefit of fine-grained alignment.
• Element-wise addition fusion of streams yields 87.6% mAP and 95.7% rank-1 on Market1501, significantly outperforming concatenation-based fusion (81.6% mAP).
• The full model with both global and part-aware branches achieves the highest performance, showing that complementary feature learning enhances re-ID accuracy.