[Paper Review] A Richly Annotated Dataset for Pedestrian Attribute Recognition

Dangwei Li, Zhang Zhang+3 more|arXiv (Cornell University)|Mar 23, 2016

Video Surveillance and Tracking MethodsComputer Science22 references180 citations

TL;DR

The paper presents the RAP dataset with 41,585 pedestrian samples and 72 attributes, plus viewpoint, occlusion, and body-part annotations, and analyzes environmental factors on attribute recognition using multi-label baselines and evaluation metrics.

ABSTRACT

In this paper, we aim to improve the dataset foundation for pedestrian attribute recognition in real surveillance scenarios. Recognition of human attributes, such as gender, and clothes types, has great prospects in real applications. However, the development of suitable benchmark datasets for attribute recognition remains lagged behind. Existing human attribute datasets are collected from various sources or an integration of pedestrian re-identification datasets. Such heterogeneous collection poses a big challenge on developing high quality fine-grained attribute recognition algorithms. Furthermore, human attribute recognition are generally severely affected by environmental or contextual factors, such as viewpoints, occlusions and body parts, while existing attribute datasets barely care about them. To tackle these problems, we build a Richly Annotated Pedestrian (RAP) dataset from real multi-camera surveillance scenarios with long term collection, where data samples are annotated with not only fine-grained human attributes but also environmental and contextual factors. RAP has in total 41,585 pedestrian samples, each of which is annotated with 72 attributes as well as viewpoints, occlusions, body parts information. To our knowledge, the RAP dataset is the largest pedestrian attribute dataset, which is expected to greatly promote the study of large-scale attribute recognition systems. Furthermore, we empirically analyze the effects of different environmental and contextual factors on pedestrian attribute recognition. Experimental results demonstrate that viewpoints, occlusions and body parts information could assist attribute recognition a lot in real applications.

Motivation & Objective

• Create a large-scale, richly annotated pedestrian attribute dataset from real surveillance scenes.
• Annotate samples with 72 fine-grained attributes plus environmental/contextual factors (viewpoints, occlusions, body parts).
• Evaluate baseline and multi-label models to understand how context affects attribute recognition.
• Introduce multi-label evaluation metrics to better capture inter-attribute dependencies in real-world scenarios.

Proposed method

• Collect real surveillance footage from 26 camera scenes over three months.
• Annotate 41,585 pedestrian samples with 72 attributes and contextual factors (viewpoints, occlusions, parts).
• Evaluate baselines using SVM with ELF and CNN features (FC6/FC7) and two multi-label CNN models (ACN, DeepMAR).
• Use two feature types (ELF and CNN-based features from CaffeNet) and compare single-attribute vs. multi-attribute joint learning.
• Propose and apply multi-label evaluation metrics (accuracy, precision, recall, F1) alongside traditional mean Accuracy (mA).
• Investigate the influence of parts by analyzing head-shoulder, upper-body, and lower-body regions on attribute recognition.

Experimental results

Research questions

• RQ1 How do viewpoints, occlusions, and body-part visibility influence pedestrian attribute recognition performance?
• RQ2 Do multi-label learning approaches (ACN, DeepMAR) outperform single-attribute classifiers on RAP?
• RQ3 Can part-based representations improve attribute recognition under real surveillance conditions?
• RQ4 What evaluation metrics best capture the dependencies among multiple attributes in this context?

Key findings

• RAP is the largest pedestrian attribute dataset to date, with 41,585 samples and 72 attributes plus contextual annotations.
• Viewpoints, occlusions, and body-part information significantly affect attribute recognition performance.
• CNN-based features (FC6/FC7) generally outperform ELF features for this task, with FC6 showing strong generalization.
• Example-based (multi-label) evaluation reveals meaningful inter-attribute dependencies and substantial gains for multi-attribute joint learning over single-attribute SVM approaches.
• Part-based analysis shows that attributes linked to specific body regions benefit from using head-shoulder, upper-body, or lower-body features, and incorporating parts can improve recognition.