[Paper Review] Deep Image Matting with Flexible Guidance Input

Hang Cheng, Shugong Xu+2 more|arXiv (Cornell University)|Oct 21, 2021

Image Enhancement Techniques4 citations

TL;DR

This paper proposes a deep image matting method that supports flexible guidance input—trimap, scribblemap, clickmap, or no guidance—using a Progressive Trimap Deformation (PTD) scheme and a Semantic Fusion Module (SFM). The method achieves state-of-the-art performance across all guidance levels, including zero guidance, by progressively distorting trimaps during training and fusing multi-scale features for robust foreground-background separation.

ABSTRACT

Image matting is an important computer vision problem. Many existing matting methods require a hand-made trimap to provide auxiliary information, which is very expensive and limits the real world usage. Recently, some trimap-free methods have been proposed, which completely get rid of any user input. However, their performance lag far behind trimap-based methods due to the lack of guidance information. In this paper, we propose a matting method that use Flexible Guidance Input as user hint, which means our method can use trimap, scribblemap or clickmap as guidance information or even work without any guidance input. To achieve this, we propose Progressive Trimap Deformation(PTD) scheme that gradually shrink the area of the foreground and background of the trimap with the training step increases and finally become a scribblemap. To make our network robust to any user scribble and click, we randomly sample points on foreground and background and perform curve fitting. Moreover, we propose Semantic Fusion Module(SFM) which utilize the Feature Pyramid Enhancement Module(FPEM) and Joint Pyramid Upsampling(JPU) in matting task for the first time. The experiments show that our method can achieve state-of-the-art results comparing with existing trimap-based and trimap-free methods.

Motivation & Objective

• To address the high user burden of manually creating trimaps in image matting.
• To improve performance of trimap-free methods by incorporating flexible user guidance inputs.
• To enable a single model to handle varying levels of user input complexity—trimap, scribbles, clicks, or no input.
• To enhance feature learning for accurate alpha matte prediction using multi-scale semantic fusion.
• To develop a training scheme that generalizes well across diverse guidance inputs.

Proposed method

• Introduces Progressive Trimap Deformation (PTD), a data augmentation scheme that gradually shrinks foreground and background regions in trimaps during training, transforming them into scribblemaps.
• Employs random point sampling and curve fitting to simulate realistic user scribbles and clicks, improving robustness to diverse user inputs.
• Proposes a Semantic Fusion Module (SFM) combining Feature Pyramid Enhancement Module (FPEM) and Joint Pyramid Upsampling (JPU) for multi-scale semantic feature fusion.
• Trains the model end-to-end with multiple guidance types, enabling zero-shot generalization to unseen guidance levels.
• Uses the composition equation $ I_i = \alpha_i F_i + (1 - \alpha_i) B_i $ as the underlying matting formulation, with $ \alpha \in [0,1] $.
• Applies PTD not only to improve performance on reduced guidance but also to enhance generalization to clickmaps and no-guidance scenarios.

Experimental results

Research questions

• RQ1 Can a deep image matting model achieve SOTA performance across multiple guidance input types, including no guidance?
• RQ2 How does progressive deformation of trimaps during training improve robustness to sparse user inputs like scribbles and clicks?
• RQ3 To what extent can a single model trained with PTD generalize to trimap-free matting scenarios?
• RQ4 Does the proposed Semantic Fusion Module (SFM) effectively enhance feature representation for accurate alpha matte prediction?
• RQ5 How does the model’s performance degrade with decreasing guidance quality, and can it remain robust without any guidance?

Key findings

• The proposed method achieves a SAD of 36.36 and MSE of 0.0141 on the no-guidance test set, demonstrating strong performance without any user input.
• With clickmap guidance, the method achieves SAD of 34.67 and MSE of 0.0112, outperforming existing trimap-free methods.
• The PTD scheme improves performance on trimap by 2.32 SAD points compared to standard training, and boosts performance on clickmaps and no guidance by over 10 SAD points.
• Applying PTD to GCA and DIM models improves their no-guidance SAD from 60.15 and 90.10 to 41.05 and 65.67, respectively.
• The SFM module enhances performance on all guidance types, with ablation studies showing it outperforms using FPEM or JPU alone.
• The method shows low standard deviation across different scribblemap and clickmap test sets, proving robustness to shape variations at the same guidance level.