[Paper Review] Focus on Query: Adversarial Mining Transformer for Few-Shot Segmentation

Yuan Wang, Naisong Luo+1 more|arXiv (Cornell University)|Nov 29, 2023

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes Adversarial Mining Transformer (AMFormer), a query-centric few-shot segmentation model that achieves state-of-the-art performance by focusing on refining query features rather than extracting fine-grained support information. It uses an adversarial training scheme between a global expansion network (G) and a detail mining network (D) to progressively refine coarse predictions using intra-object similarity, achieving 76.8% mIoU on Pascal-5i with only rough support labels.

ABSTRACT

Few-shot segmentation (FSS) aims to segment objects of new categories given only a handful of annotated samples. Previous works focus their efforts on exploring the support information while paying less attention to the mining of the critical query branch. In this paper, we rethink the importance of support information and propose a new query-centric FSS model Adversarial Mining Transformer (AMFormer), which achieves accurate query image segmentation with only rough support guidance or even weak support labels. The proposed AMFormer enjoys several merits. First, we design an object mining transformer (G) that can achieve the expansion of incomplete region activated by support clue, and a detail mining transformer (D) to discriminate the detailed local difference between the expanded mask and the ground truth. Second, we propose to train G and D via an adversarial process, where G is optimized to generate more accurate masks approaching ground truth to fool D. We conduct extensive experiments on commonly used Pascal-5i and COCO-20i benchmarks and achieve state-of-the-art results across all settings. In addition, the decent performance with weak support labels in our query-centric paradigm may inspire the development of more general FSS models. Code will be available at https://github.com/Wyxdm/AMNet.

Motivation & Objective

• To address the limitation of current few-shot segmentation methods that over-rely on fine-grained support information, especially under high intra-class diversity.
• To investigate whether rough category-level support guidance is sufficient for accurate query image segmentation.
• To develop a query-centric paradigm that leverages intra-object similarity to recover complete object masks from incomplete activations.
• To reduce dependency on precise support annotations by designing a model robust to weak or incomplete support labels.
• To improve segmentation accuracy through adversarial refinement of coarse predictions using local detail discrimination.

Proposed method

• Introduces an object mining transformer (G) that expands incomplete, support-activated regions in the query image using intra-object feature similarity.
• Proposes a detail mining transformer (D) that identifies local discrepancies between the expanded mask and ground truth to guide refinement.
• Employs an adversarial training objective where G generates increasingly accurate masks to fool D, enabling coarse-to-fine refinement.
• Uses learnable local proxies in D to focus on ambiguous regions such as object boundaries, improving localization accuracy.
• Applies multi-scale feature aggregation in G to handle objects of varying scales in the query image.
• Integrates a diversity loss ($\mathcal{L}_{div}$) in D to prevent proxies from collapsing to the same region, ensuring comprehensive feature coverage.

Experimental results

Research questions

• RQ1 Can a few-shot segmentation model achieve high performance using only rough category-level support guidance instead of precise pixel-level support features?
• RQ2 How does intra-object similarity compare to inter-object or intra-category similarity in guiding mask refinement under high intra-class diversity?
• RQ3 Can adversarial training between a global expansion network and a local detail discriminator improve segmentation accuracy without relying on fine-grained support supervision?
• RQ4 What is the impact of weak support annotations (e.g., bounding boxes, scribbles) on the performance of query-centric FSS models?
• RQ5 How does multi-scale feature aggregation enhance the robustness of mask prediction across varying object scales?

Key findings

• AMFormer achieves 76.8% mIoU on the Pascal-5i benchmark in the 1-shot setting, setting a new state-of-the-art performance.
• The model maintains strong performance (69.3% mIoU) even when 80% of the support features are erased, demonstrating robustness to incomplete support information.
• The addition of the detail mining transformer (D) improves performance by 1.2% mIoU over G alone, with the diversity loss ($\mathcal{L}_{div}$) being critical to this gain.
• Using 10 local proxies in D achieves optimal performance, as more proxies yield diminishing returns due to redundancy.
• The model achieves 73.1% mIoU with bounding box support and 71.5% with scribble support on Pascal-5i, proving feasibility under weak supervision.
• Visualizations confirm that the pseudo support image generated from rough guidance better captures full object attributes than real support, due to reduced misalignment with query features.