[Paper Review] TransMatting: Tri-token Equipped Transformer Model for Image Matting

Huanqia Cai, Fanglei Xue+2 more|arXiv (Cornell University)|Mar 11, 2023

Image Enhancement Techniques4 citations

TL;DR

This paper proposes TransMatting, a Vision Transformer-based image matting model that introduces tri-token, a learnable token representation of trimap semantics (foreground, background, unknown), to enhance long-range feature learning and improve performance on transparent and non-salient objects. By replacing traditional trimap concatenation with tri-token injection, the method achieves state-of-the-art results on benchmarks including Composition-1k, AIM-500, and the new Transparent-460 dataset, with up to 20% reduction in MSE and 10% in SAD on existing methods.

ABSTRACT

Image matting aims to predict alpha values of elaborate uncertainty areas of natural images, like hairs, smoke, and spider web. However, existing methods perform poorly when faced with highly transparent foreground objects due to the large area of uncertainty to predict and the small receptive field of convolutional networks. To address this issue, we propose a Transformer-based network (TransMatting) to model transparent objects with long-range features and collect a high-resolution matting dataset of transparent objects (Transparent-460) for performance evaluation. Specifically, to utilize semantic information in the trimap flexibly and effectively, we also redesign the trimap as three learnable tokens, named tri-token. Both Transformer and convolution matting models could benefit from our proposed tri-token design. By replacing the traditional trimap concatenation strategy with our tri-token, existing matting methods could achieve about 10% improvement in SAD and 20% in MSE. Equipped with the new tri-token design, our proposed TransMatting outperforms current state-of-the-art methods on several popular matting benchmarks and our newly collected Transparent-460.

Motivation & Objective

• To address the poor performance of existing image matting methods on highly transparent and non-salient objects like glass, smoke, and hair, which suffer from limited receptive fields and sparse known regions.
• To overcome the limitations of traditional trimap concatenation in deep networks, which fails to effectively capture long-range relationships and semantic context.
• To design a new, unified representation of trimap information—tri-token—that enables better integration of foreground, background, and unknown region semantics into the self-attention mechanism.
• To collect and release a high-resolution, transparent-object-specific dataset (Transparent-460) to better evaluate and benchmark matting models on challenging real-world transparent cases.

Proposed method

• Propose tri-token, a learnable representation of the trimap using three distinct tokens for foreground, background, and unknown regions, replacing traditional concatenation with RGB input.
• Integrate tri-token into the self-attention mechanism of a Vision Transformer to allow global, attention-driven modeling of long-range dependencies across the image.
• Design a Multi-scale Global-guided Fusion (MGF) module that uses high-level features to guide the fusion of multi-scale features, ensuring only foreground-relevant features are passed to the decoder.
• Train the model end-to-end using a combination of L1, L2, and structural losses to optimize alpha matte prediction.
• Adapt the tri-token mechanism to convolutional networks by injecting it at multiple stages, enabling performance gains across both Transformer and CNN-based matting architectures.
• Collect and release Transparent-460, a high-resolution dataset of 460 images with highly transparent and non-salient objects for evaluation.

Experimental results

Research questions

• RQ1 Can a Vision Transformer-based architecture with global self-attention improve image matting performance on transparent and non-salient objects compared to CNNs with limited receptive fields?
• RQ2 Does replacing traditional trimap concatenation with a learnable tri-token representation enhance feature learning and model generalization in image matting?
• RQ3 Can the tri-token mechanism be effectively applied to both Transformer and convolutional neural network architectures to boost performance?
• RQ4 How does the model generalize to real-world transparent objects not seen during training, especially in complex scenes like glass or fire?
• RQ5 To what extent does the proposed MGF module reduce background noise influence and improve feature fusion in uncertainty-prone regions?

Key findings

• TransMattingV2 achieves SOTA performance on the Composition-1k benchmark, with 15.70 SAD, 12.20 MSE, 11.36 Grad, and 15.05 Conn. on the AIM-500 real-world dataset, outperforming all prior methods.
• On the new Transparent-460 test set, TransMattingV2 achieves 181.94 SAD, 18.37 MSE, 40.99 Grad, and 159.53 Conn., demonstrating strong generalization on highly transparent objects.
• Replacing traditional trimap concatenation with tri-token improves existing methods by approximately 10% in SAD and 20% in MSE across multiple SOTA baselines.
• The tri-token mechanism enables the model to focus on foreground regions even in complex, transparent scenes, as shown by attention map visualizations that suppress background responses.
• The MGF module effectively reduces background noise by guiding feature fusion with high-level semantic cues, improving feature representation in uncertain regions.
• Ablation studies confirm that tri-token injection significantly enhances performance, especially in regions with minimal known pixels, validating its role in long-range feature learning.