[Paper Review] Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing

Ke Gong, Xiaodan Liang+3 more|arXiv (Cornell University)|Mar 16, 2017

Multimodal Machine Learning Applications31 references51 citations

TL;DR

Introduces the Look into Person (LIP) large-scale human parsing benchmark and a self-supervised structure-sensitive learning (SSL) method that enforces parsing results to be consistent with inferred body joint structures. SSL improves parsing accuracy on LIP and PASCAL-Person-Part datasets.

ABSTRACT

Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.

Motivation & Objective

• Create a large-scale, diverse benchmark for human parsing to cover real-world appearance variability and challenging scenarios.
• Analyze leading human parsing approaches to identify strengths and failure modes under diverse conditions.
• Propose a self-supervised structure-sensitive learning framework that enforces semantic consistency with human body structure without extra joint annotations.

Proposed method

• Annotate a new Look into Person (LIP) dataset with 50,462 images and 19 semantic part labels plus a background label.
• Analyze state-of-the-art parsing methods on LIP to understand performance gaps and structure-related failures.
• Introduce a self-supervised structure-sensitive loss that uses joints inferred from parsing maps (head, upper body, lower body, limbs, shoes) to weight parsing loss.
• Compute joint structure heatmaps from parsing results and ground truth, then minimize L2 loss between predicted and ground-truth joint heatmaps as a structure term.
• Derive the final loss as Structure = JointLoss × ParsingLoss, enabling end-to-end integration into existing networks (e.g., Attention to Scale, DeepLabV2).
• Evaluate SSL on LIP and the public PASCAL-Person-Part dataset to demonstrate improved mean IoU and per-class improvements, especially for small or visually ambiguous parts.

Experimental results

Research questions

• RQ1 How large-scale and diverse should a human parsing dataset be to capture real-world appearance variation, occlusion, and viewpoints?
• RQ2 Do current state-of-the-art parsing models suffer from structural inconsistencies with human body layout, and can a structure-aware, self-supervised signal improve predictions without extra annotations?
• RQ3 Can a joint-structure-based weighting scheme improve pixel-wise parsing accuracy, particularly for small parts and ambiguous left/right distinctions?
• RQ4 Is the proposed SSL approach transferable across datasets (LIP and PASCAL-Person-Part) and network backbones?

Key findings

• LIP is a 50,462-image benchmark with 19 part labels plus background, offering higher diversity and difficulty than prior datasets.
• State-of-the-art parsing methods show meaningful performance gaps on LIP, with structure priors and multi-scale features improving results.
• The proposed self-supervised structure-sensitive learning (SSL) consistently improves parsing performance on both LIP and PASCAL-Person-Part, outperforming baselines by notable margins.
• Per-class IoU gains are especially pronounced for small or highly ambiguous parts (e.g., sunglasses, gloves, socks) and for left/right limb distinctions.
• SSL helps parsing outputs align better with plausible human body configurations, addressing unreasonable results observed in structure-agnostic methods.
• The SSL signal can be injected into existing architectures (e.g., Attention to Scale, DeepLabV2) with minimal architectural changes and without extra joint annotations.