[Paper Review] Towards Complex Backgrounds: A Unified Difference-Aware Decoder for Binary Segmentation

Jiepan Li, Wei He+1 more|arXiv (Cornell University)|Oct 27, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes a unified dual-branch difference-aware decoder (DAD) for binary segmentation that mimics human visual attention to enhance foreground-background distinction in complex backgrounds. By leveraging multi-level features through three stages—guide map generation, background-aware feature fusion, and difference-guided enhancement—DAD achieves state-of-the-art performance across salient object detection, camouflaged object detection, polyp segmentation, and mirror detection tasks.

ABSTRACT

Binary segmentation is used to distinguish objects of interest from background, and is an active area of convolutional encoder-decoder network research. The current decoders are designed for specific objects based on the common backbones as the encoders, but cannot deal with complex backgrounds. Inspired by the way human eyes detect objects of interest, a new unified dual-branch decoder paradigm named the difference-aware decoder is proposed in this paper to explore the difference between the foreground and the background and separate the objects of interest in optical images. The difference-aware decoder imitates the human eye in three stages using the multi-level features output by the encoder. In Stage A, the first branch decoder of the difference-aware decoder is used to obtain a guide map. The highest-level features are enhanced with a novel field expansion module and a dual residual attention module, and are combined with the lowest-level features to obtain the guide map. In Stage B, the other branch decoder adopts a middle feature fusion module to make trade-offs between textural details and semantic information and generate background-aware features. In Stage C, the proposed difference-aware extractor, consisting of a difference guidance model and a difference enhancement module, fuses the guide map from Stage A and the background-aware features from Stage B, to enlarge the differences between the foreground and the background and output a final detection result. The results demonstrate that the difference-aware decoder can achieve a higher accuracy than the other state-of-the-art binary segmentation methods for these tasks.

Motivation & Objective

• Address the challenge of accurate binary segmentation in complex backgrounds where foreground objects are visually similar to or indistinct from the background.
• Overcome the limitations of task-specific decoders by designing a unified decoder architecture applicable across diverse binary segmentation tasks.
• Improve feature representation by emulating the human visual system’s three-stage detection process: coarse detection, background attention, and difference amplification.
• Enhance feature discrimination using novel modules such as the field expansion module, dual residual attention, and difference-aware fusion mechanisms.
• Demonstrate generalization and robustness across multiple binary segmentation tasks using standard backbones like VGG, ResNet, Res2Net, and PVT.

Proposed method

• Stage A uses the highest- and lowest-level encoder features with a field expansion module (FEM) and dual residual attention to generate a guide map for object-of-interest localization.
• Stage B employs a middle feature fusion (MFF) module to balance semantic and textural information from mid-level features, producing background-aware features.
• Stage C fuses the guide map and background-aware features via a difference-aware extractor (DAE) combining a difference guidance model (DGM) and difference enhancement module (DEM).
• The DGM uses a cross-attention-like mechanism to refine features using prior knowledge from the guide map, enhancing foreground relevance.
• The DEM performs feature subtraction (foreground - background) to amplify differences between object and background, improving segmentation precision.
• The DAE module is designed as a plug-and-play component and can be repeated twice for optimal performance, avoiding degradation from overuse.

Experimental results

Research questions

• RQ1 Can a unified decoder architecture effectively handle diverse binary segmentation tasks with complex backgrounds?
• RQ2 How can human visual attention mechanisms be effectively modeled in a deep learning-based segmentation framework?
• RQ3 What is the optimal design for a decoder that enhances both foreground localization and background discrimination simultaneously?
• RQ4 How do individual components—FEM, MFF, DGM, and DEM—contribute to the overall performance gain in complex background scenarios?
• RQ5 Does the proposed difference-aware fusion strategy outperform standard feature fusion or attention mechanisms in binary segmentation tasks?

Key findings

• The difference-aware decoder (DAD) achieves state-of-the-art performance across four binary segmentation tasks: salient object detection (SOD), camouflaged object detection (COD), polyp segmentation, and mirror detection.
• On the CAMO dataset, DAD achieves an $E_{ ho}$ of 0.895 and $M$ of 0.063, outperforming the SOTA method BASNet (0.887, 0.068) in both metrics.
• On the COD10K dataset, DAD achieves $E_{ ho}$ of 0.905 and $M$ of 0.032, surpassing SINet V2 (0.898, 0.038) in both metrics.
• On ECSSD, DAD achieves $E_{ ho}$ of 0.963 and $M$ of 0.028, exceeding Polyp-PVT’s performance (0.950, 0.032).
• On PASCAL-S, DAD achieves $E_{ ho}$ of 0.909 and $M$ of 0.058, outperforming MirrorNet (0.901, 0.060).
• Ablation studies confirm that the field expansion module (FEM) outperforms ASPP and RFB, and repeating the DAE module twice yields optimal performance, while three repetitions degrade results.