[Paper Review] HorNet: A Hierarchical Offshoot Recurrent Network for Improving Person Re-ID via Image Captioning

Shiyang Yan, Jun Xu+2 more|arXiv (Cornell University)|Aug 14, 2019

Video Surveillance and Tracking Methods19 references4 citations

TL;DR

This paper proposes HorNet, a hierarchical offshoot recurrent network that jointly learns visual and language representations from images and automatically generated captions to improve person re-identification (re-ID). By using domain transfer via SPGAN and a captioner to generate high-quality descriptions, HorNet leverages discrete binary gates in a two-layer LSTM to filter relevant linguistic features, achieving state-of-the-art performance on CUHK03, Market-1501, and Duke-MTMC, with 97.1% top-1 CMC accuracy and 79.2% mAP on Duke-MTMC after reranking.

ABSTRACT

Person re-identification (re-ID) aims to recognize a person-of-interest across different cameras with notable appearance variance. Existing research works focused on the capability and robustness of visual representation. In this paper, instead, we propose a novel hierarchical offshoot recurrent network (HorNet) for improving person re-ID via image captioning. Image captions are semantically richer and more consistent than visual attributes, which could significantly alleviate the variance. We use the similarity preserving generative adversarial network (SPGAN) and an image captioner to fulfill domain transfer and language descriptions generation. Then the proposed HorNet can learn the visual and language representation from both the images and captions jointly, and thus enhance the performance of person re-ID. Extensive experiments are conducted on several benchmark datasets with or without image captions, i.e., CUHK03, Market-1501, and Duke-MTMC, demonstrating the superiority of the proposed method. Our method can generate and extract meaningful image captions while achieving state-of-the-art performance.

Motivation & Objective

• To address appearance variance in person re-identification caused by pose, viewpoint, and lighting changes.
• To overcome the scarcity of annotated language descriptions in real-world person re-ID datasets.
• To develop a method that generates high-quality, discriminative image captions for unseen person images.
• To jointly learn visual and language representations using a novel hierarchical recurrent architecture.
• To achieve state-of-the-art re-ID performance using both annotated and automatically generated captions.

Proposed method

• A similarity preserving generative adversarial network (SPGAN) is used to transfer real-world images into a unified domain, improving caption quality.
• An image captioner generates semantic descriptions for person images, especially for datasets without annotations.
• HorNet employs a two-branch architecture: a CNN (ResNet-50) for visual feature extraction and a two-layer LSTM with Gumbel-Softmax gates for language representation.
• The discrete binary gates in the LSTM dynamically select relevant words from the caption, ensuring alignment with visual features.
• The model jointly optimizes visual and language representations, enhancing feature discriminability for re-ID.
• Reranking is applied post-feature extraction to further improve mAP and CMC scores.

Experimental results

Research questions

• RQ1 Can automatically generated image captions significantly reduce appearance variance in person re-identification?
• RQ2 How effective is a hierarchical offshoot recurrent network in fusing visual and language representations for re-ID?
• RQ3 Does domain adaptation via SPGAN improve the quality of generated captions for downstream re-ID tasks?
• RQ4 Can HorNet outperform existing methods that use auxiliary language or attribute information?
• RQ5 What is the impact of caption quality on re-ID performance, especially when transferring across domains?

Key findings

• HorNet achieves 97.1% top-1 CMC accuracy on CUHK03 with labeled captions, a 4.6% improvement over the ILA method.
• On Market-1501, the model achieves 85.8% mAP, demonstrating strong generalization across datasets.
• On Duke-MTMC, which lacks captions, the model reaches 60.4% mAP after domain transfer and caption generation, outperforming baseline visual features.
• With reranking, the mAP on Duke-MTMC increases to 79.2%, showing the effectiveness of the joint representation learning.
• Replacing HorNet with BERT yields poorer performance, confirming the superiority of the proposed gate-controlled architecture.
• Ablation studies confirm that both the captioning module and the hierarchical offshoot mechanism are essential for optimal performance.