[Paper Review] Rethinking of Pedestrian Attribute Recognition: Realistic Datasets with Efficient Method

Jian Jia, Houjing Huang+3 more|arXiv (Cornell University)|May 25, 2020

Video Surveillance and Tracking MethodsComputer Science36 references30 citations

TL;DR

The paper reveals that existing pedestrian attribute datasets allow train-test identity overlap, leading to inflated results, and proposes zero-shot datasets PETA zs and RAPv2 zs, plus a strong baseline that outperforms recent SOTA methods without localization tweaks.

ABSTRACT

Despite various methods are proposed to make progress in pedestrian attribute recognition, a crucial problem on existing datasets is often neglected, namely, a large number of identical pedestrian identities in train and test set, which is not consistent with practical application. Thus, images of the same pedestrian identity in train set and test set are extremely similar, leading to overestimated performance of state-of-the-art methods on existing datasets. To address this problem, we propose two realistic datasets PETA extsubscript{$zs$} and RAPv2 extsubscript{$zs$} following zero-shot setting of pedestrian identities based on PETA and RAPv2 datasets. Furthermore, compared to our strong baseline method, we have observed that recent state-of-the-art methods can not make performance improvement on PETA, RAPv2, PETA extsubscript{$zs$} and RAPv2 extsubscript{$zs$}. Thus, through solving the inherent attribute imbalance in pedestrian attribute recognition, an efficient method is proposed to further improve the performance. Experiments on existing and proposed datasets verify the superiority of our method by achieving state-of-the-art performance.

Motivation & Objective

• Highlight the impractical identity overlap in current pedestrian attribute datasets and its impact on evaluation.
• Propose zero-shot datasets (PETA zs and RAPv2 zs) to reflect realistic deployment scenarios.
• Introduce a strong baseline that challenges the notion that SOTA improvements rely on attribute localization modules.

Proposed method

• Identify the dataset problem of train-test identity overlap and demonstrate performance gaps between common-identity and unique-identity test images.
• Construct two zero-shot datasets PETA zs and RAPv2 zs by repartitioning identities so test identities do not appear in training.
• Propose a strong baseline with a linear classifier on top of a ResNet50 backbone and a task-specific weight normalization, and evaluate using mA, Accuracy, Precision, Recall, and F1.
• Reimplement SOTA methods (MsVAA, VAC, ALM) under the same baseline and backbone for fair comparison.
• Use Grad-CAM to analyze attribute localization and show the baseline can implicitly localize attribute regions without explicit localization modules.

Experimental results

Research questions

• RQ1 Do current pedestrian attribute datasets overestimate model performance due to overlapping identities between train and test sets?
• RQ2 What is the impact on performance when test identities are strictly unseen (zero-shot) during evaluation?
• RQ3 Can a strong baseline achieve competitive results without explicit localization modules for attribute-specific regions?
• RQ4 Do state-of-the-art methods provide consistent gains on zero-shot datasets compared to traditional baselines?
• RQ5 How do common-identity vs unique-identity image subsets affect evaluation metrics across datasets?

Key findings

• Existing datasets show large overlap of identities between train and test sets, leading to inflated performance estimates.
• Zero-shot datasets PETA zs and RAPv2 zs reduce overlap and reveal significant performance drops for current methods.
• A strong baseline with a ResNet50 backbone outperforms several SOTA methods when evaluated under zero-shot settings.
• Reimplemented SOTA methods under the same baseline yield comparable or inferior gains, indicating the baseline strength rather than novel modules drives performance.
• Localization-focused attention modules offer limited or no additional gains on the strong baseline, suggesting localization may not be the key factor for improvement.
• The proposed datasets and baseline together reveal that improvements claimed by SOTA methods on existing datasets may be artifacts of data leakage and inadequate baselines.