[Paper Review] Segmentation Transformer: Object-Contextual Representations for Semantic Segmentation

Yuhui Yuan, Xiaokang Chen+2 more|arXiv (Cornell University)|Sep 24, 2019

Advanced Neural Network ApplicationsComputer Science78 references473 citations

TL;DR

The paper introduces object-contextual representations (OCR) for semantic segmentation, aggregating pixel features by learned object regions and integrating them via a Transformer-like encoder-decoder framework to boost segmentation accuracy across benchmarks.

ABSTRACT

In this paper, we address the semantic segmentation problem with a focus on the context aggregation strategy. Our motivation is that the label of a pixel is the category of the object that the pixel belongs to. We present a simple yet effective approach, object-contextual representations, characterizing a pixel by exploiting the representation of the corresponding object class. First, we learn object regions under the supervision of ground-truth segmentation. Second, we compute the object region representation by aggregating the representations of the pixels lying in the object region. Last, % the representation similarity we compute the relation between each pixel and each object region and augment the representation of each pixel with the object-contextual representation which is a weighted aggregation of all the object region representations according to their relations with the pixel. We empirically demonstrate that the proposed approach achieves competitive performance on various challenging semantic segmentation benchmarks: Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff. Our submission "HRNet + OCR + SegFix" achieves 1-st place on the Cityscapes leaderboard by the time of submission. Code is available at: https://git.io/openseg and https://git.io/HRNet.OCR. We rephrase the object-contextual representation scheme using the Transformer encoder-decoder framework. The details are presented in~Section3.3.

Motivation & Objective

• Motivate context aggregation for pixel labeling by treating the label as the corresponding object class.
• Propose object-contextual representations that learn soft object regions and their region representations.
• Augment pixel representations with weighted object-region representations via relations between pixels and object regions.
• Demonstrate strong performance on Cityscapes, ADE20K, LIP, PASCAL-Context, COCO-Stuff, and COCO panoptic tasks.

Proposed method

• Form soft object regions M1,...,MK corresponding to each class by a coarse soft segmentation learned from backbone features.
• Compute object region representations fk by aggregating pixel features x_i weighted by normalized region memberships tilde{m}_{ki}.
• Compute pixel-object region relations w_{ik} as softmax over a bilinear function kappa(x_i, f_k) to obtain an object-contextual representation y_i.
• Fuse the original pixel feature x_i with the object-contextual representation y_i to form augmented pixel features z_i via a small neural transform.
• Rephrase OCR within a Segmentation Transformer: use K category queries as object-region selectors in the decoder cross-attention to produce M_k and f_k, with encoder cross-attention integrating object-region representations into per-pixel predictions.
• Backbone choices include dilated ResNet-101 or HRNet-W48, with the OCR module trained end-to-end using pixel-wise cross-entropy losses on both object-region supervision and final segmentation.

Experimental results

Research questions

• RQ1 Can object-contextual representations improve semantic segmentation by explicitly modeling pixel-to-object-region relations?
• RQ2 How do soft object regions and their regional representations affect per-pixel classification accuracy?
• RQ3 Do Transformer-like cross-attention mechanisms effectively implement the OCR concept in segmentation tasks?
• RQ4 What are the efficiency and accuracy trade-offs of OCR compared to multi-scale and relational context methods?

Key findings

• OCR improves over multi-scale (PPM/ASPP) and relational context baselines across Cityscapes, ADE20K, LIP, PASCAL-Context, and COCO-Stuff.
• Object-region supervision and pixel–region relation estimation both contribute to performance gains.
• The approach achieves competitive or state-of-the-art results on several benchmarks, and demonstrates favorable efficiency (memory, FLOPs, runtime) compared to several relational and multi-scale context methods.