[Paper Review] Boosting Semantic Human Matting with Coarse Annotations

Jinlin Liu, Yuan Yao+5 more|arXiv (Cornell University)|Apr 10, 2020

Image Enhancement Techniques33 references9 citations

TL;DR

This paper proposes a novel framework that leverages both coarse and fine human annotations to boost end-to-end semantic human matting without requiring trimaps. By training a mask prediction network on hybrid data, applying a quality unification network to standardize mask quality, and refining results with a matting refinement network, the method achieves state-of-the-art performance while significantly reducing the need for expensive high-quality annotations.

ABSTRACT

Semantic human matting aims to estimate the per-pixel opacity of the foreground human regions. It is quite challenging and usually requires user interactive trimaps and plenty of high quality annotated data. Annotating such kind of data is labor intensive and requires great skills beyond normal users, especially considering the very detailed hair part of humans. In contrast, coarse annotated human dataset is much easier to acquire and collect from the public dataset. In this paper, we propose to use coarse annotated data coupled with fine annotated data to boost end-to-end semantic human matting without trimaps as extra input. Specifically, we train a mask prediction network to estimate the coarse semantic mask using the hybrid data, and then propose a quality unification network to unify the quality of the previous coarse mask outputs. A matting refinement network takes in the unified mask and the input image to predict the final alpha matte. The collected coarse annotated dataset enriches our dataset significantly, allows generating high quality alpha matte for real images. Experimental results show that the proposed method performs comparably against state-of-the-art methods. Moreover, the proposed method can be used for refining coarse annotated public dataset, as well as semantic segmentation methods, which reduces the cost of annotating high quality human data to a great extent.

Motivation & Objective

• To address the high cost and difficulty of annotating high-quality human alpha mattes, especially for fine details like hair.
• To reduce reliance on expensive, user-interactive trimaps in semantic human matting.
• To leverage easily collectible coarse-annotation datasets to improve matting performance.
• To develop a framework that enables high-quality matting from low-quality or coarse masks without explicit trimap supervision.
• To provide a practical solution for refining public coarse-annotation datasets and semantic segmentation outputs into accurate alpha mattes.

Proposed method

• A mask prediction network (MPN) is trained on a hybrid dataset of fine and coarse human annotations to predict low-resolution semantic masks.
• A quality unification network (QUN) is introduced to normalize the quality of MPN outputs across diverse input images, ensuring consistent input for the final refinement.
• A matting refinement network (MRN) takes the unified coarse mask and original image as input to predict the final high-accuracy alpha matte.
• The MPN and QUN are trained on both coarse and fine data, while the MRN is trained exclusively on fine-annotated data to preserve detail fidelity.
• The framework is designed to be compatible with existing coarse-annotation datasets and semantic segmentation outputs, enabling end-to-end refinement.
• The method uses a three-stage pipeline: coarse mask prediction → quality standardization → high-resolution matting refinement, all without trimap input.

Experimental results

Research questions

• RQ1 Can coarse-annotation data be effectively leveraged to improve semantic human matting performance without requiring trimaps?
• RQ2 How can inconsistent quality across coarse masks be unified to enable robust matting refinement?
• RQ3 Can a model trained on limited fine data generalize well when enhanced with abundant coarse data?
• RQ4 To what extent can the proposed method refine low-quality public annotations or segmentation outputs into high-quality alpha mattes?
• RQ5 Does the quality unification network significantly improve matting performance compared to direct refinement of raw coarse masks?

Key findings

• The proposed method achieves state-of-the-art performance in semantic human matting without requiring trimaps as input.
• The inclusion of coarse-annotation data significantly improves performance compared to training only on fine-annotated data, as shown by quantitative metrics in Table 2.
• The quality unification network (QUN) is essential for robust matting, as removing it leads to a noticeable drop in performance, as confirmed by metrics in Table 1.
• The method successfully refines coarse masks from public datasets like PASCAL and COCO, generating high-quality alpha mattes even for challenging details like hair.
• The framework effectively enhances outputs from semantic segmentation models such as DeepLab, recovering fine-grained details that were missing in the original predictions.
• The collected hybrid dataset, combining fine and coarse annotations, enables the model to generalize well across diverse real-world images, as demonstrated in Figure 7.