[Paper Review] Segmentation Mask Guided End-to-End Person Search

Dingyuan Zheng, Jimin Xiao+2 more|arXiv (Cornell University)|Aug 27, 2019

Video Surveillance and Tracking Methods46 references4 citations

TL;DR

This paper proposes a novel end-to-end person search framework that leverages segmentation masks to guide feature learning, reducing the negative impact of background clutters in pedestrian bounding boxes. By jointly optimizing detection, re-identification, and segmentation in a unified network, the method achieves state-of-the-art performance with 86.3% mAP and 86.5% top-1 accuracy on the CUHK-SYSU benchmark.

ABSTRACT

Person search aims to search for a target person among multiple images recorded by multiple surveillance cameras, which faces various challenges from both pedestrian detection and person re-identification. Besides the large intra-class variations owing to various illumination conditions, occlusions and varying poses, background clutters in the detected pedestrian bounding boxes further deteriorate the extracted features for each person, making them less discriminative. To tackle these problems, we develop a novel approach which guides the network with segmentation masks so that discriminative features can be learned invariant to the background clutters. We demonstrate that joint optimization of pedestrian detection, person re-identification and pedestrian segmentation enables to produce more discriminative features for pedestrian, and consequently leads to better person search performance. Extensive experiments on benchmark dataset CUHK-SYSU, show that our proposed model achieves the state-of-the-art performance with 86.3% mAP and 86.5 top-1 accuracy respectively.

Motivation & Objective

• Address the challenge of background clutters in pedestrian bounding boxes that degrade feature discriminability in person search.
• Overcome the limitations of existing methods that treat detection, re-identification, and segmentation as separate stages.
• Improve person search performance by jointly optimizing pedestrian detection, person re-identification, and segmentation in an end-to-end manner.
• Demonstrate the effectiveness of using partially labeled segmentation masks to guide feature learning in a unified framework.
• Establish a new dataset with 1,833 images annotated with segmentation masks to support training and evaluation of the proposed method.

Proposed method

• Introduce a parallel mask branch in the network to guide feature learning using segmentation masks, enhancing foreground feature extraction.
• Train the model end-to-end to jointly optimize pedestrian detection, person re-identification, and pedestrian segmentation.
• Use a subset of 16% of training images (1,833 out of 11,206) with ground-truth segmentation masks to supervise the mask-guided feature learning.
• Leverage the segmentation mask as a soft attention mechanism to suppress background features and emphasize discriminative human body regions.
• Employ a ResNet-50 or ResNet-101 backbone for feature extraction and adopt a multi-task learning strategy to optimize all components simultaneously.
• Apply a mask-aware feature aggregation module that combines global image features with mask-conditional local features to improve representation robustness.

Experimental results

Research questions

• RQ1 Can segmentation masks be effectively used to guide feature learning in person search to reduce the impact of background clutters?
• RQ2 Does end-to-end joint optimization of detection, re-identification, and segmentation lead to better performance than stage-wise approaches?
• RQ3 How does the proportion of training images with segmentation masks affect the final person search performance?
• RQ4 Can the proposed method maintain superior performance under challenging conditions such as occlusion and low resolution?
• RQ5 Is the proposed framework robust across varying gallery sizes and real-world surveillance conditions?

Key findings

• The proposed method achieves 86.3% mAP and 86.5% top-1 accuracy on the CUHK-SYSU dataset, setting a new state-of-the-art performance.
• Performance remains consistently superior across different gallery sizes, from 50 to 4,000, demonstrating robustness to scale variations.
• On low-resolution and occlusion subsets, the model achieves 66.7% mAP and 66.8% top-1 accuracy (low-res) and 70.8% mAP and 71.3% top-1 accuracy (occlusion), outperforming prior methods.
• The ablation study shows that performance improves significantly when 12% of training images include segmentation masks, stabilizing at 15% and plateauing at 16%.
• Visual comparisons show that the proposed model correctly retrieves the target person more frequently than baseline models like OIM and E2E-PS, especially in complex backgrounds.
• The integration of segmentation masks as a guidance signal leads to more discriminative features, as evidenced by improved ranking results in qualitative comparisons.