[Paper Review] PiCANet: Learning Pixel-wise Contextual Attention for Saliency Detection

Nian Liu, Junwei Han+1 more|arXiv (Cornell University)|Aug 21, 2017

Visual Attention and Saliency DetectionComputer Science11 references51 citations

TL;DR

PiCANet learns pixel-level contextual attention to selectively weight contextual information for each pixel, improving saliency detection when integrated with CNNs like U-Net, using global and local context forms.

ABSTRACT

Contexts play an important role in the saliency detection task. However, given a context region, not all contextual information is helpful for the final task. In this paper, we propose a novel pixel-wise contextual attention network, i.e., the PiCANet, to learn to selectively attend to informative context locations for each pixel. Specifically, for each pixel, it can generate an attention map in which each attention weight corresponds to the contextual relevance at each context location. An attended contextual feature can then be constructed by selectively aggregating the contextual information. We formulate the proposed PiCANet in both global and local forms to attend to global and local contexts, respectively. Both models are fully differentiable and can be embedded into CNNs for joint training. We also incorporate the proposed models with the U-Net architecture to detect salient objects. Extensive experiments show that the proposed PiCANets can consistently improve saliency detection performance. The global and local PiCANets facilitate learning global contrast and homogeneousness, respectively. As a result, our saliency model can detect salient objects more accurately and uniformly, thus performing favorably against the state-of-the-art methods.

Motivation & Objective

• Motivate the use of contextual information for saliency detection and acknowledge that not all context is equally informative.
• Introduce a pixel-wise contextual attention mechanism that learns to attend to informative context locations per pixel.
• Propose global and local PiCANet variants to capture global and local context respectively.
• Enable end-to-end training by embedding PiCANet into CNN architectures such as U-Net.
• Demonstrate consistent performance gains over state-of-the-art saliency methods.

Proposed method

• Define PiCANet as a module that outputs an attention map for each pixel over all context locations.
• Compute attended contextual features by weighted aggregation using the pixel-wise attention weights.
• Formulate two variants: global PiCANet for global context and local PiCANet for local context.
• Ensure the modules are fully differentiable and integrateable into CNNs for joint training.
• Incorporate PiCANet with a U-Net architecture to detect salient objects.
• Show through extensive experiments that PiCANet improves saliency detection and aids learning of global contrast and homogeneousness.

Experimental results

Research questions

• RQ1 Can pixel-wise contextual attention improve saliency detection by selectively attending to informative context locations?
• RQ2 How do global and local PiCANet variants contribute to saliency performance?
• RQ3 Does integrating PiCANet with U-Net enhance saliency accuracy and saliency map uniformity?
• RQ4 Are PiCANets differentiable and trainable end-to-end within standard CNN backbones?

Key findings

• PiCANet learns attention weights that reflect contextual relevance at each pixel.
• Global PiCANet facilitates learning global contrast, while local PiCANet aids homogeneousness in saliency maps.
• The PiCANet-augmented model consistently improves saliency detection performance.
• Integrating PiCANet with U-Net yields favorable results against state-of-the-art methods.
• Both global and local PiCANets are fully differentiable and trainable within end-to-end CNN frameworks.