[Paper Review] SOLQ: Segmenting Objects by Learning Queries

Bin Dong, Fangao Zeng+3 more|arXiv (Cornell University)|Jun 4, 2021

Advanced Image and Video Retrieval TechniquesComputer Science54 references56 citations

TL;DR

SOLQ presents an end-to-end instance segmentation framework built on DETR that learns a Unified Query Representation to predict class, bbox, and masks via mask vectors encoded with compression techniques. It achieves state-of-the-art results on COCO and improves DETR’s detection through joint multi-task learning.

ABSTRACT

In this paper, we propose an end-to-end framework for instance segmentation. Based on the recently introduced DETR [1], our method, termed SOLQ, segments objects by learning unified queries. In SOLQ, each query represents one object and has multiple representations: class, location and mask. The object queries learned perform classification, box regression and mask encoding simultaneously in an unified vector form. During training phase, the mask vectors encoded are supervised by the compression coding of raw spatial masks. In inference time, mask vectors produced can be directly transformed to spatial masks by the inverse process of compression coding. Experimental results show that SOLQ can achieve state-of-the-art performance, surpassing most of existing approaches. Moreover, the joint learning of unified query representation can greatly improve the detection performance of DETR. We hope our SOLQ can serve as a strong baseline for the Transformer-based instance segmentation. Code is available at https://github.com/megvii-research/SOLQ.

Motivation & Objective

• Motivate an end-to-end instance segmentation framework that avoids post-processing like NMS.
• Unify classification, localization, and segmentation into a single learnable query representation.
• Encode high-resolution masks into compact mask vectors for efficient supervision and reconstruction.
• Demonstrate competitive or state-of-the-art performance on COCO while improving DETR's detection performance through joint learning.

Proposed method

• Build on DETR with learnable object queries.
• Introduce Unified Query Representation (UQR) to parallelize three sub-tasks (classification, localization, segmentation).
• Encode spatial masks into low-dimensional mask vectors via compression coding (DCT, PCA, Sparse Coding) for supervision and reconstruction.
• Train with joint losses including detection loss and a mask-vector regression loss that is not part of the bipartite matching.

Experimental results

Research questions

• RQ1 Can a unified query representation enable end-to-end instance segmentation without ROI-based post-processing?
• RQ2 Does joint learning of detection and segmentation via UQR improve both tasks compared to DETR?
• RQ3 Which mask compression scheme yields the best mask reconstruction quality and segmentation performance?
• RQ4 What is the impact of multi-decoder auxiliary supervision on both mask and detection performance?

Key findings

• SOLQ achieves state-of-the-art performance on COCO test-dev for both instance segmentation (AP seg) and object detection (AP box).
• Using Unified Query Representation significantly improves DETR’s box AP by about 2 percentage points across tested backbones.
• Mask compression with DCT provides the best balance of segmentation and detection performance among tested methods.
• Auxiliary mask-vector loss across multiple decoders boosts both segmentation and detection metrics.
• SOLQ with Swin-L backbone reaches high performance, e.g., AP seg of 46.7 and AP box of 60.9 on COCO test-dev.