[Paper Review] A Deep Spatial Contextual Long-term Recurrent Convolutional Network for Saliency Detection

Nian Liu, Junwei Han|arXiv (Cornell University)|Oct 6, 2016

Visual Attention and Saliency Detection56 references22 citations

TL;DR

This paper proposes a deep spatial contextual long-term recurrent convolutional network (DSCLRCN) that leverages a novel deep spatial contextual LSTM (DSCLSTM) to model both global spatial context and scene-level modulation for saliency detection. By integrating long short-term memory units with spatial contextual modeling, the network captures long-range dependencies and improves saliency prediction accuracy, achieving state-of-the-art performance on benchmark datasets through end-to-end training.

ABSTRACT

Traditional saliency models usually adopt hand-crafted image features and human-designed mechanisms to calculate local or global contrast. In this paper, we propose a novel computational saliency model, i.e., deep spatial contextual long-term recurrent convolutional network (DSCLRCN) to predict where people looks in natural scenes. DSCLRCN first automatically learns saliency related local features on each image location in parallel. Then, in contrast with most other deep network based saliency models which infer saliency in local contexts, DSCLRCN can mimic the cortical lateral inhibition mechanisms in human visual system to incorporate global contexts to assess the saliency of each image location by leveraging the deep spatial long short-term memory (DSLSTM) model. Moreover, we also integrate scene context modulation in DSLSTM for saliency inference, leading to a novel deep spatial contextual LSTM (DSCLSTM) model. The whole network can be trained end-to-end and works efficiently when testing. Experimental results on two benchmark datasets show that DSCLRCN can achieve state-of-the-art performance on saliency detection. Furthermore, the proposed DSCLSTM model can significantly boost the saliency detection performance by incorporating both global spatial interconnections and scene context modulation, which may uncover novel inspirations for studies on them in computational saliency models.

Motivation & Objective

• To overcome limitations of traditional saliency models that rely on hand-crafted features and local contrast mechanisms.
• To model long-range spatial dependencies in images for improved saliency prediction by mimicking human visual cortical lateral inhibition.
• To integrate scene context modulation into recurrent spatial modeling to enhance global context awareness.
• To develop an end-to-end trainable deep network that captures both local features and global contextual relationships.
• To achieve superior performance on benchmark saliency detection datasets through novel architectural design.

Proposed method

• Proposes a deep spatial long short-term memory (DSLSTM) module to model long-range spatial dependencies across image locations.
• Introduces a deep spatial contextual LSTM (DSCLSTM) that combines spatial context modeling with scene-level modulation for improved saliency inference.
• Uses parallel convolutional feature learning to extract saliency-related local features at each image location.
• Employs long-term recurrent connections to propagate contextual information across spatial positions, emulating lateral inhibition in the human visual system.
• Integrates scene context modulation into the LSTM gates to dynamically adjust feature representation based on global scene semantics.
• Trains the entire network end-to-end using supervised learning with ground-truth saliency maps as supervision.

Experimental results

Research questions

• RQ1 Can a deep recurrent network effectively model long-range spatial dependencies in saliency detection beyond local context?
• RQ2 How does incorporating scene context modulation improve saliency prediction performance?
• RQ3 Can a recurrent architecture with spatial long short-term memory capture global visual context comparable to human visual processing?
• RQ4 Does end-to-end training of a unified network outperform traditional two-stage or hand-designed feature approaches?
• RQ5 To what extent does the proposed DSCLSTM improve state-of-the-art performance on standard saliency detection benchmarks?

Key findings

• The proposed DSCLRCN achieves state-of-the-art performance on two benchmark saliency detection datasets, outperforming existing methods.
• The integration of scene context modulation into the DSCLSTM significantly boosts detection accuracy by enhancing global context awareness.
• The DSCLSTM model effectively captures long-range spatial interconnections, improving saliency prediction beyond local context modeling.
• End-to-end training of the full network leads to efficient inference and robust feature learning without hand-designed components.
• The model demonstrates strong generalization across diverse natural scenes, indicating effective modeling of human-like visual attention mechanisms.
• Ablation studies confirm that both the long-term recurrent structure and scene context modulation contribute significantly to performance gains.