[Paper Review] Segmenting Transparent Object in the Wild with Transformer

Enze Xie, Wenjia Wang+5 more|arXiv (Cornell University)|Jan 21, 2021

Advanced Neural Network ApplicationsComputer Science41 references25 citations

TL;DR

The paper introduces Trans10K-v2, a fine‑grained transparent object segmentation dataset with 11 categories, and a transformer‑based segmentation model Trans2Seg that achieves state‑of‑the‑art results on Trans10K-v2 and transfers to ADE20K.

ABSTRACT

This work presents a new fine-grained transparent object segmentation dataset, termed Trans10K-v2, extending Trans10K-v1, the first large-scale transparent object segmentation dataset. Unlike Trans10K-v1 that only has two limited categories, our new dataset has several appealing benefits. (1) It has 11 fine-grained categories of transparent objects, commonly occurring in the human domestic environment, making it more practical for real-world application. (2) Trans10K-v2 brings more challenges for the current advanced segmentation methods than its former version. Furthermore, a novel transformer-based segmentation pipeline termed Trans2Seg is proposed. Firstly, the transformer encoder of Trans2Seg provides the global receptive field in contrast to CNN's local receptive field, which shows excellent advantages over pure CNN architectures. Secondly, by formulating semantic segmentation as a problem of dictionary look-up, we design a set of learnable prototypes as the query of Trans2Seg's transformer decoder, where each prototype learns the statistics of one category in the whole dataset. We benchmark more than 20 recent semantic segmentation methods, demonstrating that Trans2Seg significantly outperforms all the CNN-based methods, showing the proposed algorithm's potential ability to solve transparent object segmentation.

Motivation & Objective

• Motivate robust segmentation of highly transparent objects in real-world scenes for robotics and vision systems.
• Provide a large, diverse, fine-grained dataset (Trans10K-v2) with high-quality masks and functional categories.
• Propose a transformer-based segmentation architecture (Trans2Seg) that leverages global context and learnable category prototypes for accurate mask prediction.

Proposed method

• Propose Trans2Seg, a CNN‑backbone plus encoder–decoder Transformer architecture.
• Use a CNN backbone (ResNet-50 with last-stage dilation) to extract features and provide a feature map for the Transformer encoder.
• Employ a Transformer decoder that uses a set of learnable class prototypes as queries to attend to encoder features, enabling dictionary-like category lookup.
• Upsample and fuse decoder attention maps with high-resolution CNN features through a small convolutional head for final per-pixel classification via argmax.
• Introduce positional embeddings for the Transformer encoder to recover spatial information after flattening feature maps.
• Compare Transformer encoder–decoder design to SETR and DETR, highlighting the category-prototype query as a key difference for semantic segmentation.

Experimental results

Research questions

• RQ1 Can a transformer-based pipeline improve fine-grained transparent object segmentation over CNN-based methods?
• RQ2 Does modeling segmentation as a dictionary lookup with learnable category prototypes enhance mask quality and category discrimination?
• RQ3 How does Trans2Seg perform on a large, fine-grained transparent object dataset and on general segmentation benchmarks like ADE20K?

Key findings

• Trans10K-v2 contains 10,428 images with 11 fine-grained categories (shelf, jar, freezer, window, glass door, eyeglass, cup, glass wall, glass bowl, water bottle, storage box).
• Trans2Seg significantly outperforms CNN-based methods on Trans10K-v2, achieving 72.15% mIoU and 94.14% pixel accuracy (vs. 69.00 mIoU for the previous SOTA TransLab).
• Transformer encoder provides a larger global receptive field than CNNs, improving segmentation of transparent objects.
• Replacing a CNN decoder with a Transformer decoder that uses learnable category prototypes as queries yields a further mIoU improvement (up to 72.1% in ablations).
• On ADE20K, Trans2Seg reaches 39.7 mIoU, demonstrating transferability to general segmentation tasks.