[Paper Review] Pyramid Grafting Network for One-Stage High Resolution Saliency Detection

C. Xie, Changqun Xia+4 more|arXiv (Cornell University)|Apr 11, 2022

Visual Attention and Saliency Detection4 citations

TL;DR

This paper proposes Pyramid Grafting Network (PGNet), a one-stage framework for high-resolution saliency detection that fuses features from a CNN and a transformer backbone via a Cross-Model Grafting Module (CMGM) to preserve both global context and fine details. It achieves state-of-the-art performance on high-resolution benchmarks, including a new 4K–8K dataset, UHRSD, which is the largest such dataset to date.

ABSTRACT

Recent salient object detection (SOD) methods based on deep neural network have achieved remarkable performance. However, most of existing SOD models designed for low-resolution input perform poorly on high-resolution images due to the contradiction between the sampling depth and the receptive field size. Aiming at resolving this contradiction, we propose a novel one-stage framework called Pyramid Grafting Network (PGNet), using transformer and CNN backbone to extract features from different resolution images independently and then graft the features from transformer branch to CNN branch. An attention-based Cross-Model Grafting Module (CMGM) is proposed to enable CNN branch to combine broken detailed information more holistically, guided by different source feature during decoding process. Moreover, we design an Attention Guided Loss (AGL) to explicitly supervise the attention matrix generated by CMGM to help the network better interact with the attention from different models. We contribute a new Ultra-High-Resolution Saliency Detection dataset UHRSD, containing 5,920 images at 4K-8K resolutions. To our knowledge, it is the largest dataset in both quantity and resolution for high-resolution SOD task, which can be used for training and testing in future research. Sufficient experiments on UHRSD and widely-used SOD datasets demonstrate that our method achieves superior performance compared to the state-of-the-art methods.

Motivation & Objective

• To address the performance drop of existing saliency detection models on high-resolution images due to limited receptive fields and loss of fine details.
• To overcome the limitations of multi-stage high-resolution SOD methods, including error propagation and high inference cost.
• To design a one-stage, efficient, and accurate framework that effectively fuses CNN and transformer features for high-resolution saliency detection.
• To provide a new large-scale, high-resolution saliency detection dataset (UHRSD) to support future research.

Proposed method

• PGNet uses a dual-branch encoder: a ResNet-based CNN and a Swin Transformer, each processing input images at full resolution independently.
• A Cross-Model Grafting Module (CMGM) enables feature fusion by attending to corresponding features from the transformer branch and injecting them into the CNN branch at matching spatial scales.
• The CMGM employs an attention mechanism to guide the grafting process, allowing the CNN branch to integrate global semantic cues from the transformer while preserving spatial detail.
• An Attention Guided Loss (AGL) is introduced to supervise the attention matrix in CMGM, improving alignment and feature interaction between the two branches.
• The network is trained end-to-end using a combination of standard loss and AGL, with feature maps from the final decoder stage used for prediction.
• The method is evaluated on both low- and high-resolution benchmarks, including the newly introduced UHRSD dataset with 5,920 images at 4K–8K resolution.

Experimental results

Research questions

• RQ1 Can a one-stage deep learning framework effectively handle high-resolution saliency detection by fusing CNN and transformer features?
• RQ2 How does cross-branch feature grafting using attention improve detail preservation and global context integration in high-resolution SOD?
• RQ3 To what extent does training on a large-scale, high-resolution dataset (UHRSD) improve generalization and performance on unseen high-res images?
• RQ4 Does the proposed Attention Guided Loss enhance attention alignment and feature fusion quality in cross-model grafting?

Key findings

• PGNet achieves state-of-the-art performance on both low-resolution (DUTS-TE) and high-resolution (UHRSD-TE) benchmarks, with an Fβ^Max of 0.949 and MAE of 0.026 on UHRSD-TE.
• Training on the mixed dataset UHRSD-TR + HRSOD-TR yields superior performance on UHRSD-TE, indicating that high-resolution annotations help improve generalization.
• The ablation study shows that grafting at the R5-S2 feature pair (residual block 5 of CNN and stage 2 of transformer) yields the best performance, with Fβ^Max of 0.946 on UHRSD-TE.
• The Attention Guided Loss (AGL) significantly improves performance, with a 0.004 increase in Fβ^Max on UHRSD-TE compared to baseline without AGL.
• Visual results demonstrate that PGNet produces sharper boundaries and better captures small, salient objects often missed by prior methods.
• The UHRSD dataset, containing 5,920 images at 4K–8K resolution, is the largest and highest-resolution SOD dataset available to date.