[Paper Review] Keypoint Based Weakly Supervised Human Parsing

Zhonghua Wu, Guosheng Lin+1 more|arXiv (Cornell University)|Sep 14, 2018

Advanced Neural Network Applications17 references3 citations

TL;DR

This paper proposes a weakly supervised human parsing method that achieves competitive performance using only object keypoint annotations instead of costly pixel-level labels. It employs an iterative pseudo mask generation process and a correlation network for joint part-object segmentation, reducing labeling effort while matching fully supervised results with only a ~10% performance drop.

ABSTRACT

Fully convolutional networks (FCN) have achieved great success in human parsing in recent years. In conventional human parsing tasks, pixel-level labeling is required for guiding the training, which usually involves enormous human labeling efforts. To ease the labeling efforts, we propose a novel weakly supervised human parsing method which only requires simple object keypoint annotations for learning. We develop an iterative learning method to generate pseudo part segmentation masks from keypoint labels. With these pseudo masks, we train an FCN network to output pixel-level human parsing predictions. Furthermore, we develop a correlation network to perform joint prediction of part and object segmentation masks and improve the segmentation performance. The experiment results show that our weakly supervised method is able to achieve very competitive human parsing results. Despite our method only uses simple keypoint annotations for learning, we are able to achieve comparable performance with fully supervised methods which use the expensive pixel-level annotations.

Motivation & Objective

• Reduce the high human labeling cost of pixel-level annotations in human parsing by using only object keypoint annotations as supervision.
• Develop a method to generate high-quality pseudo segmentation masks from sparse keypoint annotations to train a fully convolutional network (FCN).
• Improve part segmentation performance by modeling the strong correlation between part and object segmentation through joint learning.
• Enable the use of both human-annotated and model-predicted keypoints (e.g., from Mask R-CNN or AlphaPose) for training, increasing practical applicability.
• Demonstrate that weakly supervised learning with keypoint supervision can achieve performance close to fully supervised methods.

Proposed method

• Use keypoint annotations to define initial foreground and background regions for graph cut-based pseudo mask generation, with foreground regions derived from keypoint labeling and connections.
• Apply an iterative refinement process to improve pseudo mask quality by alternating between FCN predictions and graph cut optimization using keypoint constraints.
• Train a DeepLab-based FCN with VGG-16 backbone using the generated pseudo masks, optimizing for object, part, and refined part segmentation losses with equal weights.
• Introduce a correlation block in the network to model interactions between part and object predictions, enhancing segmentation consistency and accuracy.
• Incorporate test-time keypoint predictions (from Mask R-CNN or AlphaPose) into the graph cut optimization to further refine segmentation results.
• Use super-pixel segmentation with a minimum component size of 60 to improve graph cut granularity and stability.

Experimental results

Research questions

• RQ1 Can human parsing be effectively trained using only object keypoint annotations instead of pixel-level masks?
• RQ2 How effective is an iterative pseudo mask generation method in transforming sparse keypoint annotations into high-quality supervision for semantic segmentation?
• RQ3 To what extent does joint learning of part and object segmentation improve parsing performance compared to independent prediction?
• RQ4 Can the proposed method generalize to low-quality keypoint predictions from pre-trained detectors like Mask R-CNN and AlphaPose?
• RQ5 Does incorporating test-time keypoint predictions further improve segmentation accuracy?

Key findings

• The proposed weakly supervised method achieves a mean IoU of 67.8% on the PASCAL VOC Human Part dataset, which is within ~10% of the fully supervised baseline using the same VGG-16 backbone.
• Iterative refinement of pseudo masks improves IoU performance, with convergence achieved after 3 iterations, indicating effective mask refinement.
• The correlation block for joint part-object learning improves part segmentation IoU, demonstrating that modeling interdependencies enhances segmentation accuracy.
• The method successfully leverages keypoint annotations from both human-labeled datasets (e.g., PASCAL VOC Human Pose) and pre-trained detectors like Mask R-CNN and AlphaPose, achieving competitive results even with less accurate keypoint inputs.
• Incorporating test-time keypoint predictions via graph cut optimization leads to a significant performance boost, confirming the value of dynamic keypoint integration during inference.