[Paper Review] Understanding Humans in Crowded Scenes: Deep Nested Adversarial Learning and A New Benchmark for Multi-Human Parsing

Jian Zhao, Jianshu Li+5 more|arXiv (Cornell University)|Apr 10, 2018

Anomaly Detection Techniques and Applications40 references42 citations

TL;DR

The paper introduces a large-scale Multi-Human Parsing (MHP v2.0) dataset with 25,403 images and 58 fine-grained categories, and a novel Deep Nested Adversarial Network (NAN) for end-to-end multi-human parsing. NAN comprises three GAN-like sub-nets for semantic saliency, instance-agnostic parsing, and instance-aware clustering, trained in a nested adversarial framework.

ABSTRACT

Despite the noticeable progress in perceptual tasks like detection, instance segmentation and human parsing, computers still perform unsatisfactorily on visually understanding humans in crowded scenes, such as group behavior analysis, person re-identification and autonomous driving, etc. To this end, models need to comprehensively perceive the semantic information and the differences between instances in a multi-human image, which is recently defined as the multi-human parsing task. In this paper, we present a new large-scale database "Multi-Human Parsing (MHP)" for algorithm development and evaluation, and advances the state-of-the-art in understanding humans in crowded scenes. MHP contains 25,403 elaborately annotated images with 58 fine-grained semantic category labels, involving 2-26 persons per image and captured in real-world scenes from various viewpoints, poses, occlusion, interactions and background. We further propose a novel deep Nested Adversarial Network (NAN) model for multi-human parsing. NAN consists of three Generative Adversarial Network (GAN)-like sub-nets, respectively performing semantic saliency prediction, instance-agnostic parsing and instance-aware clustering. These sub-nets form a nested structure and are carefully designed to learn jointly in an end-to-end way. NAN consistently outperforms existing state-of-the-art solutions on our MHP and several other datasets, and serves as a strong baseline to drive the future research for multi-human parsing.

Motivation & Objective

• Motivate holistic understanding of humans in crowded scenes beyond detection and instance segmentation.
• Provide a large-scale, richly annotated benchmark for multi-human parsing with fine-grained semantic categories.
• Develop a unified end-to-end model that learns parsing and instance differentiation simultaneously in a nested adversarial setting.
• Enable efficient, single-pass multi-human parsing suitable for real-world applications.

Proposed method

• Propose MHP v2.0 as a large-scale dataset with 25,403 images and 58 semantic categories for body parts, clothes, and accessories.
• Introduce NAN, a three-branch GAN-like framework for semantic saliency prediction, instance-agnostic parsing, and instance-aware clustering.
• Each sub-net is trained with adversarial and task-specific losses, with a nested, mutually boosting structure to enable end-to-end backpropagation.
• Use semantic saliency as a prior to facilitate parsing, combine with instance-agnostic parsing, and finally perform instance-aware clustering without region proposals.
• Provide training details including network initialization, loss terms, and end-to-end optimization objective.
• Report evaluation on MHP v2.0 and other datasets showing NAN’s superiority over state-of-the-art methods.

Experimental results

Research questions

• RQ1 Can a nested adversarial learning framework improve holistic multi-human parsing in crowded scenes?
• RQ2 Does a large-scale, fine-grained dataset (MHP v2.0) better support learning of instance-level body-part and fashion-item parsing in the presence of occlusion and interactions?
• RQ3 Can end-to-end NAN deliver accurate parsing and instance differentiation in a single forward pass without heavy pre/post-processing?
• RQ4 What are the effects of incorporating semantic saliency priors and instance-agnostic parsing on instance-aware clustering performance?

Key findings

• NAN achieves superior performance on multi-human parsing compared to state-of-the-art methods across MHP v2.0 and other benchmarks.
• The model enables single-pass multi-human parsing with competitive speed, avoiding expensive region proposals.
• NAN demonstrates effective end-to-end training through its nested adversarial structure with joint optimization of multiple losses.
• The MHP v2.0 dataset provides extensive annotations (58 categories) and real-world variability including viewpoints, occlusions, and interactions.
• Experiments include evaluations on MHP v2.0, MHP v1.0, PASCAL-Person-Part, and Buffy to validate NAN’s versatility.