[Paper Review] Mask Guided Matting via Progressive Refinement Network

Qihang Yu, Jianming Zhang+6 more|arXiv (Cornell University)|Dec 12, 2020

Image Enhancement Techniques44 references4 citations

TL;DR

This paper proposes Mask Guided (MG) Matting, a robust deep learning framework that uses a Progressive Refinement Network (PRN) to iteratively refine alpha matte predictions from coarse guidance masks—such as trimaps, binary segmentations, or low-quality alpha mattes—while maintaining high accuracy. The method achieves state-of-the-art performance across real and synthetic benchmarks by combining self-guided refinement with a novel data augmentation strategy, Random Alpha Blending, to improve foreground color estimation.

ABSTRACT

We propose Mask Guided (MG) Matting, a robust matting framework that takes a general coarse mask as guidance. MG Matting leverages a network (PRN) design which encourages the matting model to provide self-guidance to progressively refine the uncertain regions through the decoding process. A series of guidance mask perturbation operations are also introduced in the training to further enhance its robustness to external guidance. We show that PRN can generalize to unseen types of guidance masks such as trimap and low-quality alpha matte, making it suitable for various application pipelines. In addition, we revisit the foreground color prediction problem for matting and propose a surprisingly simple improvement to address the dataset issue. Evaluation on real and synthetic benchmarks shows that MG Matting achieves state-of-the-art performance using various types of guidance inputs. Code and models are available at https://github.com/yucornetto/MGMatting.

Motivation & Objective

• To address the limitation of existing matting methods that require high-quality, specific guidance inputs like precise trimaps.
• To develop a generalizable matting framework that works robustly with various types and qualities of coarse masks, including trimaps, binary segmentations, and low-quality alpha mattes.
• To improve foreground color prediction in matting, which is critical for reducing fringing in compositing, by addressing dataset limitations through synthetic data generation.
• To release a new high-quality real-world matting benchmark dataset to better evaluate matting models in realistic scenarios.

Proposed method

• Proposes a Progressive Refinement Network (PRN) that enables self-guided, iterative refinement of uncertain regions during decoding, improving prediction accuracy over multiple stages.
• Introduces a guidance mask perturbation pipeline during training, including random binarization, morphological operations, and CutMask, to enhance robustness to diverse and imperfect external masks.
• Employs Random Alpha Blending (RAB) to generate synthetic training data by blending random alpha mattes with images, improving foreground color estimation without manual annotations.
• Trains the model on a modified Composition-1k dataset with data augmentation (re-JPEGing, blur, noise) to improve robustness to real-world image corruptions.
• Uses a two-branch network architecture with shared encoder and separate decoders for alpha matte and foreground color prediction, enabling joint optimization.
• Replaces the standard composition loss during training when using augmented data to avoid label mismatch due to color changes from data augmentation.

Experimental results

Research questions

• RQ1 Can a deep learning matting model generalize across diverse and low-quality guidance masks, such as trimaps, binary segmentations, and soft alpha mattes, without requiring task-specific fine-tuning?
• RQ2 How effective is self-guided progressive refinement in improving matting quality, especially in detail-rich regions like hair and fur?
• RQ3 To what extent can synthetic data generation via Random Alpha Blending improve foreground color prediction and reduce fringing in compositing?
• RQ4 How does the proposed method compare to state-of-the-art trimap-based and trimap-free matting models on real-world and synthetic benchmarks?
• RQ5 Can a robust matting framework be trained to maintain consistent performance under perturbations to the guidance mask, such as erosion and dilation?

Key findings

• MG Matting achieves state-of-the-art performance on both synthetic (Composition-1k) and real-world benchmarks, with a mean SAD of 8.01 and MSE of 0.0095 on the whole image, and SAD of 5.94 and MSE of 0.0637 on detail regions.
• The model maintains consistent performance under guidance mask perturbations: SAD increases only slightly from 26.8 to 27.4 when the mask is eroded by 30 pixels, demonstrating strong robustness.
• Compared to the trimap-free method BSHM [27], MG Matting achieves significantly better results, with lower MSE (0.0095 vs. 0.0155) and SAD (8.01 vs. 10.66) on the whole image.
• The proposed Random Alpha Blending (RAB) method improves foreground color prediction accuracy, reducing fringing artifacts in composited images without requiring additional annotations.
• The model generalizes well to unseen guidance types, including low-quality alpha mattes and rough binary masks, outperforming methods trained specifically for one type of guidance.
• The released high-quality real-world matting benchmark dataset enables more reliable evaluation of matting models in realistic conditions, addressing the lack of such benchmarks in prior work.