[Paper Review] CamoFormer: Masked Separable Attention for Camouflaged Object Detection

Bowen Yin, Xuying Zhang+4 more|arXiv (Cornell University)|Dec 10, 2022

Visual Attention and Saliency Detection27 citations

TL;DR

CamoFormer introduces Masked Separable Attention (MSA) within an encoder–decoder Transformer framework to separately model foreground and background cues, achieving state-of-the-art camouflaged object detection on major COD benchmarks.

ABSTRACT

How to identify and segment camouflaged objects from the background is challenging. Inspired by the multi-head self-attention in Transformers, we present a simple masked separable attention (MSA) for camouflaged object detection. We first separate the multi-head self-attention into three parts, which are responsible for distinguishing the camouflaged objects from the background using different mask strategies. Furthermore, we propose to capture high-resolution semantic representations progressively based on a simple top-down decoder with the proposed MSA to attain precise segmentation results. These structures plus a backbone encoder form a new model, dubbed CamoFormer. Extensive experiments show that CamoFormer surpasses all existing state-of-the-art methods on three widely-used camouflaged object detection benchmarks. There are on average around 5% relative improvements over previous methods in terms of S-measure and weighted F-measure.

Motivation & Objective

• Motivate and address the challenge of segmenting camouflaged objects that are highly similar to their surroundings.
• Propose a novel Masked Separable Attention (MSA) that splits attention heads to handle foreground, background, and global relations.
• Leverage a progressive top-down decoder to refine segmentation maps using MSA at multiple feature levels.
• Demonstrate state-of-the-art performance on three COD benchmarks and analyze the contributions of MSA components.

Proposed method

• Adopt an encoder–decoder architecture with a Transformer-based backbone (PVTv2) to extract multi-scale features.
• Introduce Masked Separable Attention (MSA) that groups attention heads into Foreground-TA (F-TA), Background-TA (B-TA), and normal TA, each using predicted foreground masks to compute masked queries/keys.
• Generate foreground masks from intermediate predictions and use them to drive F-TA and B-TA computations, while concatenating with normal TA outputs (Z) and passing through a 3x3 conv to 128 channels.
• Progressively fuse features from top to bottom in the decoder by computing an element-wise product before summing with the corresponding encoder features (D_i = up(MSA(D_{i+1})) * E_i + E_i).
• Supervise multi-stage predictions with BCE and IoU losses, summing losses across stages to train the model end-to-end.

Experimental results

Research questions

• RQ1 Can foreground-background separated attention improve the segmentation of camouflaged objects compared to standard self-attention in COD?
• RQ2 Does progressive refinement with MSA at multiple decoder levels yield higher quality CAMO masks and borders?
• RQ3 How do the individual MSA components (F-TA, B-TA, and TA) contribute to COD performance?
• RQ4 What are the trade-offs in decoder width (C_d) and how do they affect accuracy vs. computation?

Key findings

• CamoFormer achieves new state-of-the-art results on NC4K, COD10K, and CAMO COD benchmarks.
• On COD10K-test, it reaches 0.786 weighted F-measure and 0.023 MAE (surpassing the second-best FDNet with 0.731 and 0.030).
• MSA provides consistent improvements over baselines and standalone TA, with all three branches (F-TA, B-TA, TA) contributing to performance gains.
• Progressive refinement with MSA yields larger performance gaps in later decoder levels, indicating strong complementarity between MSA and the top-down fusion.
• Ablation shows that increasing decoder channels (C_d) improves performance up to a point (128 channels), balancing accuracy and computation.
• Qualitative visualizations show sharper borders and more complete camouflaged object segmentation compared to prior methods.