[Paper Review] Boosting General Trimap-free Matting in the Real-World Image

Leo Shan Wenzhang Zhou Grace Zhao|arXiv (Cornell University)|May 28, 2024

Textile materials and evaluationsMaterials Science3 citations

TL;DR

This paper proposes MFC-Net, a coarse-to-fine trimap-free matting network that uses saliency-defined foregrounds and multi-feature fusion to improve foreground completeness and generalization. It introduces image harmony in data synthesis and establishes the Real-19k dataset, achieving state-of-the-art performance on both synthetic and real-world images with SAD of 35.4 and MSE of 9.4 on Real-19k.

ABSTRACT

Image matting aims to obtain an alpha matte that separates foreground objects from the background accurately. Recently, trimap-free matting has been well studied because it requires only the original image without any extra input. Such methods usually extract a rough foreground by itself to take place trimap as further guidance. However, the definition of 'foreground' lacks a unified standard and thus ambiguities arise. Besides, the extracted foreground is sometimes incomplete due to inadequate network design. Most importantly, there is not a large-scale real-world matting dataset, and current trimap-free methods trained with synthetic images suffer from large domain shift problems in practice. In this paper, we define the salient object as foreground, which is consistent with human cognition and annotations of the current matting dataset. Meanwhile, data and technologies in salient object detection can be transferred to matting in a breeze. To obtain a more accurate and complete alpha matte, we propose a network called extbf{M}ulti- extbf{F}eature fusion-based extbf{C}oarse-to-fine Network extbf{(MFC-Net)}, which fully integrates multiple features for an accurate and complete alpha matte. Furthermore, we introduce image harmony in data composition to bridge the gap between synthetic and real images. More importantly, we establish the largest general matting dataset extbf{(Real-19k)} in the real world to date. Experiments show that our method is significantly effective on both synthetic and real-world images, and the performance in the real-world dataset is far better than existing matting-free methods. Our code and data will be released soon.

Motivation & Objective

• To address the ambiguity in foreground definition in trimap-free matting by defining the foreground as the salient object, aligning with human cognition and existing matting datasets.
• To improve foreground completeness and alpha matte accuracy through a coarse-to-fine network with multi-feature fusion of low-level, high-level, and global context features.
• To reduce domain shift between synthetic and real-world images by introducing image harmony in data generation.
• To establish a large-scale, real-world matting benchmark (Real-19k) for reliable evaluation of trimap-free methods.
• To achieve state-of-the-art performance on both synthetic and real-world matting benchmarks through integrated methodological and data innovations.

Proposed method

• Defines the foreground as the salient object using saliency detection, ensuring consistency with human perception and existing dataset annotations.
• Proposes MFC-Net, a coarse-to-fine network that uses low-resolution feature extraction in the coarse module to enhance foreground integrity and reduce false negatives.
• Introduces an Interweave Attention (IA) module that fuses low-level appearance, high-level semantic, and global context features through learnable filters rather than simple concatenation or addition.
• Incorporates a Global Context Feature (GCF) module that normalizes foreground-background differences across diverse images, improving generalization.
• Applies image harmony during synthetic data generation to align synthetic image textures and brightness with real-world images, reducing domain shift.
• Uses a multi-scale training strategy with downsampling to 512×512 as optimal trade-off between global context and edge detail preservation.

Experimental results

Research questions

• RQ1 How can a consistent and cognition-aligned foreground definition be established for trimap-free matting to reduce ambiguity in model learning?
• RQ2 To what extent can multi-level feature fusion with attention mechanisms improve foreground completeness and alpha matte accuracy?
• RQ3 Can image harmony in data generation significantly reduce domain shift and improve real-world generalization of matting models?
• RQ4 What is the optimal trade-off between resolution and feature representation in coarse-to-fine matting networks?
• RQ5 How does a large-scale real-world matting dataset like Real-19k improve the evaluation and benchmarking of trimap-free matting methods?

Key findings

• The proposed MFC-Net achieves SOTA performance on the Real-19k dataset with a SAD of 35.4, MSE of 9.4, Grad of 19.6, and Conn of 33.5.
• Ablation studies show that removing the Interweave Attention (IA) module increases MSE by 2.2, and removing the Global Context Feature (GCF) increases MSE by 6.8, demonstrating their critical roles.
• Image harmony improves generalization: MSE drops from 15.5 to 9.4 when combined with 512×512 downsampling, showing significant domain adaptation gains.
• The coarse module with 1/4 downsampling (512×512) achieves optimal balance, as full-resolution training leads to poor performance due to local feature learning.
• The Real-19k dataset, with 19,000 real-world images, enables reliable evaluation of real-world matting generalization, outperforming existing benchmarks.
• The method significantly outperforms prior trimap-free methods like LFM and MGMatting on real-world images, with a 30%+ reduction in SAD and MSE.