[Paper Review] Multiple-Human Parsing in the Wild

Jianshu Li, Jian Zhao+6 more|arXiv (Cornell University)|May 19, 2017

Multimodal Machine Learning Applications47 references61 citations

TL;DR

The paper introduces the Multi-Human Parsing (MHP) dataset and a novel MH-Parser model that performs global parsing and instance-aware parsing for multiple humans in unconstrained real-world scenes, via a Graph-GAN based affinity learning approach.

ABSTRACT

Human parsing is attracting increasing research attention. In this work, we aim to push the frontier of human parsing by introducing the problem of multi-human parsing in the wild. Existing works on human parsing mainly tackle single-person scenarios, which deviates from real-world applications where multiple persons are present simultaneously with interaction and occlusion. To address the multi-human parsing problem, we introduce a new multi-human parsing (MHP) dataset and a novel multi-human parsing model named MH-Parser. The MHP dataset contains multiple persons captured in real-world scenes with pixel-level fine-grained semantic annotations in an instance-aware setting. The MH-Parser generates global parsing maps and person instance masks simultaneously in a bottom-up fashion with the help of a new Graph-GAN model. We envision that the MHP dataset will serve as a valuable data resource to develop new multi-human parsing models, and the MH-Parser offers a strong baseline to drive future research for multi-human parsing in the wild.

Motivation & Objective

• Define the multi-human parsing problem to reflect real-world scenarios with multiple interacting persons.
• Create a large-scale MHP dataset with pixel-level, instance-aware 18-part annotations.
• Propose MH-Parser to generate global parsing maps and instance masks without relying on external detectors.
• Leverage Graph-GAN to learn high-order relations and improve parsing of entangled persons.

Proposed method

• Use ResNet-101-based representation learning to produce a global instance-agnostic parsing map G_seg.
• Define a pairwise affinity graph over superpixels and predict an affinity map A via an affinity net.
• Introduce Graph-GAN with a GCN-based discriminator to refine the affinity graph and capture high-order relations.
• Compute a global accordance map M to distinguish instances and derive clusters via spectral clustering on predicted A.
• Refine instance masks with a CRF that incorporates unary and pairwise terms informed by the affinity graph.
• Train with a combination of segmentation loss, L2 affinity loss, and GAN loss, and perform testing to obtain pixel-level instance-aware parsing.

Experimental results

Research questions

• RQ1 How can multi-human parsing be formulated to operate in the wild with multiple interacting and occluded persons?
• RQ2 Can a bottom-up approach using graph-structured affinity learning outperform detector-based methods for separating closely entangled human instances?
• RQ3 Does a Graph-GAN trained on graph-structured affinities improve high-order relationship modeling for body parts and clothing across instances?
• RQ4 What is the effectiveness of joint global parsing and instance clustering followed by CRF refinement on the MHP dataset?

Key findings

• MH-Parser achieves competitive performance with Mask R-CNN and Discriminative Loss on the MHP dataset in terms of AP_p and PCP metrics.
• On challenging subsets with high instance proximity, MH-Parser outperforms Mask R-CNN and DL by better handling entangled persons.
• On Buffy dataset evaluation, MH-Parser achieves an average forward score of 71.11% and backward score of 71.94% (outperforming prior methods).
• Baseline ablations show gains from incorporating GAN loss and the refinement step, with GT-based components yielding higher scores (e.g., GT Global Segmentation leading to 91.75 AP_p_0.5).
• The MHP dataset contains 4,980 images with 14,969 person instances and 18 part labels, demonstrating substantial real-world complexity for multi-human parsing.