[Paper Review] TASED-Net: Temporally-Aggregating Spatial Encoder-Decoder Network for Video Saliency Detection

Kyle Min, Jason J. Corso|arXiv (Cornell University)|Aug 15, 2019

Visual Attention and Saliency DetectionComputer Science40 references17 citations

TL;DR

TASED-Net is a 3D fully-convolutional encoder-decoder network for video saliency detection that jointly performs spatial decoding and temporal aggregation using a novel Auxiliary pooling mechanism to enable effective max-unpooling. It significantly outperforms prior state-of-the-art methods on DHF1K, Hollywood2, and UCFSports, especially in attending to salient moving objects.

ABSTRACT

TASED-Net is a 3D fully-convolutional network architecture for video saliency detection. It consists of two building blocks: first, the encoder network extracts low-resolution spatiotemporal features from an input clip of several consecutive frames, and then the following prediction network decodes the encoded features spatially while aggregating all the temporal information. As a result, a single prediction map is produced from an input clip of multiple frames. Frame-wise saliency maps can be predicted by applying TASED-Net in a sliding-window fashion to a video. The proposed approach assumes that the saliency map of any frame can be predicted by considering a limited number of past frames. The results of our extensive experiments on video saliency detection validate this assumption and demonstrate that our fully-convolutional model with temporal aggregation method is effective. TASED-Net significantly outperforms previous state-of-the-art approaches on all three major large-scale datasets of video saliency detection: DHF1K, Hollywood2, and UCFSports. After analyzing the results qualitatively, we observe that our model is especially better at attending to salient moving objects.

Motivation & Objective

• To address the limitation of existing video saliency models that process spatial and temporal information separately or via recurrent units like LSTMs.
• To develop a fully-convolutional architecture that jointly models spatial and temporal features for improved saliency prediction.
• To overcome the challenge of mismatched temporal receptive fields in max-unpooling layers during decoder upscaling.
• To validate that conditioning on a limited number of past frames is sufficient and effective for video saliency prediction.
• To demonstrate the superiority of joint spatiotemporal modeling over sequential processing with LSTMs in video saliency detection.

Proposed method

• The network uses S3D pre-trained on Kinetics as the encoder to extract low-resolution spatiotemporal features from a clip of T consecutive frames.
• The prediction network performs spatial upscaling via transposed convolutions and max-unpooling layers, with Auxiliary pooling providing properly-sized switches for temporal reconstruction.
• Auxiliary pooling introduces additional max-poolings with reduced temporal dimension to generate valid switches for max-unpooling layers, overcoming the mismatch between pooling and unpooling temporal extents.
• The model predicts a single saliency map per input clip using a sliding-window inference strategy, enabling frame-wise saliency prediction across the entire video.
• The architecture avoids recurrent units like LSTMs, instead using fully-convolutional operations to jointly decode spatial and aggregate temporal information.
• The method is evaluated on three large-scale datasets using standard metrics including NSS, CC, SIM, AUC-J, and s-AUC.

Experimental results

Research questions

• RQ1 Can a fully-convolutional network jointly perform spatial decoding and temporal aggregation to improve video saliency detection beyond LSTM-based methods?
• RQ2 Is it effective to condition saliency prediction on a fixed number of past frames, and what is the optimal window size?
• RQ3 Can max-unpooling layers be effectively used in a 3D decoder when temporal receptive fields are mismatched?
• RQ4 Does the proposed Auxiliary pooling mechanism outperform standard upsampling methods like interpolation and transposed convolution?
• RQ5 Does joint spatiotemporal modeling lead to better attention to moving salient objects compared to prior approaches?

Key findings

• TASED-Net achieves state-of-the-art performance on all three benchmark datasets: DHF1K, Hollywood2, and UCFSports, with a mean NSS of 2.706 on DHF1K.
• The model significantly outperforms previous SOTA methods, achieving a 0.122 improvement in NSS over the next best method on DHF1K.
• TASED-Net with T=32 frames (approximately one second of video) achieves the best performance, indicating that a fixed window of past frames is sufficient for optimal saliency prediction.
• Auxiliary pooling outperforms both trilinear interpolation and transposed convolution in the ablation study, demonstrating its necessity and effectiveness.
• Adding deeper spatial decoding (more transposed convolution layers) slightly degrades performance, suggesting that precise spatial reconstruction is less critical for saliency than for segmentation tasks.
• Predicting multiple saliency maps per window reduces performance, confirming that single-map prediction with temporal aggregation is more effective.