[Paper Review] Context-Aware Image Matting for Simultaneous Foreground and Alpha Estimation

Qiqi Hou, Feng Liu|arXiv (Cornell University)|Sep 20, 2019

Image Enhancement Techniques51 references11 citations

TL;DR

This paper proposes a context-aware deep learning framework for simultaneous foreground and alpha matte estimation from a single natural image, using a two-encoder-two-decoder architecture that fuses local features and global context. The method achieves state-of-the-art performance through a combination of Laplacian and perceptual feature loss, along with effective data augmentation, producing high-quality, visually plausible matting results even on real-world images trained on synthetic data.

ABSTRACT

Natural image matting is an important problem in computer vision and graphics. It is an ill-posed problem when only an input image is available without any external information. While the recent deep learning approaches have shown promising results, they only estimate the alpha matte. This paper presents a context-aware natural image matting method for simultaneous foreground and alpha matte estimation. Our method employs two encoder networks to extract essential information for matting. Particularly, we use a matting encoder to learn local features and a context encoder to obtain more global context information. We concatenate the outputs from these two encoders and feed them into decoder networks to simultaneously estimate the foreground and alpha matte. To train this whole deep neural network, we employ both the standard Laplacian loss and the feature loss: the former helps to achieve high numerical performance while the latter leads to more perceptually plausible results. We also report several data augmentation strategies that greatly improve the network's generalization performance. Our qualitative and quantitative experiments show that our method enables high-quality matting for a single natural image. Our inference codes and models have been made publicly available at https://github.com/hqqxyy/Context-Aware-Matting.

Motivation & Objective

• To address the ill-posed nature of single-image natural image matting by jointly estimating foreground and alpha matte.
• To improve matting quality by integrating both local visual details and global contextual information.
• To overcome the limitation of existing deep learning methods that only predict alpha matte without reconstructing the foreground.
• To enhance generalization to real-world images despite training on synthetic datasets.
• To achieve perceptually superior and numerically accurate matting results through loss design and data augmentation.

Proposed method

• The method employs a two-encoder-two-decoder fully convolutional neural network: a matting encoder captures local features, and a context encoder captures global context.
• Features from both encoders are concatenated and fed into separate decoders for foreground image and alpha matte prediction.
• The network is trained using a combination of Laplacian loss for numerical accuracy and feature loss for perceptual quality.
• Data augmentation strategies such as Re-JPEGING and Gaussian blurring are applied to improve robustness to real-world image variations.
• The model is trained on the synthetic dataset from Xu et al. [52] but generalizes effectively to real-world test images.
• Inference codes and trained models are publicly released on GitHub for reproducibility and further research.

Figure 1: Real-world image matting. Our method is able to simultaneously estimate high-quality foreground images and alpha maps from real-world images although trained on a synthetic dataset. Our results keep final structures (the top example) while being free from the common color bleeding problem

Experimental results

Research questions

• RQ1 Can a deep learning model simultaneously estimate both the foreground image and alpha matte from a single natural image?
• RQ2 How does integrating local and global features improve matting performance compared to methods relying only on local features?
• RQ3 To what extent does combining Laplacian and feature loss improve both numerical and perceptual quality of matting results?
• RQ4 Can data augmentation strategies trained on synthetic data effectively generalize to real-world matting challenges?
• RQ5 How does the proposed method compare to state-of-the-art non-deep learning and deep learning matting methods in terms of visual fidelity and user preference?

Key findings

• The proposed method achieves a mean preference rate of 85.48% in user studies against Global Matting [19], indicating strong perceptual superiority.
• It achieves a preference rate of 84.11% against Closed-form Matting [27], demonstrating consistent user preference over non-deep learning SOTA methods.
• The method outperforms Deep Matting [52] with a 77.67% preference rate, indicating that joint foreground and alpha estimation leads to better visual results.
• The combination of Laplacian and feature loss leads to both high numerical accuracy and improved perceptual quality, especially in handling fine structures like hair and lace.
• Data augmentation techniques such as Re-JPEGING and Gaussian blurring significantly improve generalization to real-world images despite training on synthetic data.
• The method successfully mitigates common issues like color bleeding and preserves fine details, as demonstrated in qualitative comparisons on challenging real-world examples.

Figure 2: The architecture of our matting network. We design a two-encoder-two-decoder network. The matting encoder and the context encoder capture both visual features and more global context information. The features from these two encoders are concatenated and feed to the foreground and the alpha