[Paper Review] TransReID: Transformer-based Object Re-Identification

Shuting He, Hao Luo+4 more|arXiv (Cornell University)|Feb 8, 2021

Advanced Neural Network ApplicationsComputer Science48 references95 citations

TL;DR

TransReID presents a pure transformer framework for object ReID, introducing the jigsaw patch module (JPM) and side information embeddings (SIE) to achieve state-of-the-art results on person and vehicle ReID benchmarks.

ABSTRACT

Extracting robust feature representation is one of the key challenges in object re-identification (ReID). Although convolution neural network (CNN)-based methods have achieved great success, they only process one local neighborhood at a time and suffer from information loss on details caused by convolution and downsampling operators (e.g. pooling and strided convolution). To overcome these limitations, we propose a pure transformer-based object ReID framework named TransReID. Specifically, we first encode an image as a sequence of patches and build a transformer-based strong baseline with a few critical improvements, which achieves competitive results on several ReID benchmarks with CNN-based methods. To further enhance the robust feature learning in the context of transformers, two novel modules are carefully designed. (i) The jigsaw patch module (JPM) is proposed to rearrange the patch embeddings via shift and patch shuffle operations which generates robust features with improved discrimination ability and more diversified coverage. (ii) The side information embeddings (SIE) is introduced to mitigate feature bias towards camera/view variations by plugging in learnable embeddings to incorporate these non-visual clues. To the best of our knowledge, this is the first work to adopt a pure transformer for ReID research. Experimental results of TransReID are superior promising, which achieve state-of-the-art performance on both person and vehicle ReID benchmarks.

Motivation & Objective

• Motivation to overcome CNN limitations in global context modeling and fine-grained detail preservation for ReID.
• Develop a pure transformer-based ReID framework that matches or surpasses CNN-based methods.
• Introduce mechanisms to enhance robustness to occlusion, misalignment, and camera/viewpoint variation.
• Demonstrate effectiveness of two novel modules, JPM and SIE, within an end-to-end trainable system.

Proposed method

• Encode images as sequences of overlapping patches and process with a Transformer to capture global context.
• Use a two-branch setup: a global feature branch and a JPM-based local feature branch with a shared transformer layer.
• Introduce a Jigsaw Patch Module (JPM) that shifts and shuffles patch embeddings to generate diverse, robust local features.
• Integrate Side Information Embeddings (SIE) to encode camera and viewpoint information as learnable embeddings.
• Train with ID loss and triplet loss on both global and local features, and fuse them for inference.

Experimental results

Research questions

• RQ1 Can a pure Transformer backbone achieve competitive performance for person and vehicle ReID compared to CNN-based models?
• RQ2 Do JPM and SIE improve robustness to occlusion, misalignment, and cross-camera/viewpoint variations in ReID?
• RQ3 What is the impact of overlapping patches and joint global-local learning on ReID performance?
• RQ4 How much do camera/viewpoint side information embeddings contribute to performance on standard ReID benchmarks?

Key findings

• A pure Transformer baseline already achieves competitive ReID performance versus CNN backbones.
• Adding JPM yields notable gains (e.g., +2.6% mAP on MSMT17 and +1.0% mAP on VeRi-776 over Baseline).
• Incorporating SIE reduces inter-camera/viewpoint biases and further improves mAP (e.g., MSMT17 gains with camera and viewpoint encoding).
• TransReID with JPM and SIE achieves 64.9 mAP and 83.3 R1 on MSMT17, and 80.6 mAP and 96.9 R1 on VeRi-776.
• Compared to prior state-of-the-art, TransReID variants attain state-of-the-art performance on multiple benchmarks including MSMT17, Market-1501, DukeMTMC-reID, Occluded-Duke, VeRi-776 and VehicleID.