[Paper Review] Recurrent Mixture Density Network for Spatiotemporal Visual Attention

Loris Bazzani, Hugo Larochelle+1 more|arXiv (Cornell University)|Mar 27, 2016

Visual Attention and Saliency DetectionComputer Science60 references115 citations

TL;DR

The paper introduces RMDN, a spatiotemporal saliency model that uses 3D-CNN features, an LSTM backbone, and a Gaussian Mixture Network to predict human-like video saliency, improving both saliency prediction and action recognition performance.

ABSTRACT

In many computer vision tasks, the relevant information to solve the problem at hand is mixed to irrelevant, distracting information. This has motivated researchers to design attentional models that can dynamically focus on parts of images or videos that are salient, e.g., by down-weighting irrelevant pixels. In this work, we propose a spatiotemporal attentional model that learns where to look in a video directly from human fixation data. We model visual attention with a mixture of Gaussians at each frame. This distribution is used to express the probability of saliency for each pixel. Time consistency in videos is modeled hierarchically by: 1) deep 3D convolutional features to represent spatial and short-term time relations and 2) a long short-term memory network on top that aggregates the clip-level representation of sequential clips and therefore expands the temporal domain from few frames to seconds. The parameters of the proposed model are optimized via maximum likelihood estimation using human fixations as training data, without knowledge of the action in each video. Our experiments on Hollywood2 show state-of-the-art performance on saliency prediction for video. We also show that our attentional model trained on Hollywood2 generalizes well to UCF101 and it can be leveraged to improve action classification accuracy on both datasets.

Motivation & Objective

• Motivate and model visual attention in videos to filter out distracting information in automatic analysis.
• Develop a spatiotemporal saliency predictor trained directly from human fixations without action labels.
• Leverage clip-level 3D CNN features and LSTM-based temporal aggregation to produce frame-wise saliency maps.
• Show that predicted saliency maps improve downstream action recognition performance.
• Demonstrate state-of-the-art saliency prediction on Hollywood2 and assess generalization to UCF101.

Proposed method

• Represent each video clip (16 frames) with 3D CNN features (C3D).
• Use an LSTM to aggregate clip representations over time for long-term temporal consistency.
• Predict saliency as a Gaussian Mixture Model, with GMM parameters produced by a dense layer from the LSTM (MDN).
• Train the network by maximum likelihood using human fixation data, with backpropagation through time.
• At test time, normalize the GMM-based saliency map to form a pixel-wise probability distribution.
• Utilize saliency as soft attention by weighting video pixels (or features) to improve action recognition via concatenation of context and saliency-weighted representations and a linear SVM classifier.

Experimental results

Research questions

• RQ1 Can a deep recurrent model predict spatiotemporal human saliency in videos using fixation data without action labels?
• RQ2 Does integrating 3D CNN clip representations with an LSTM and MDN yield state-of-the-art saliency prediction on Hollywood2?
• RQ3 Can predicted saliency maps improve action recognition performance on Hollywood2 and generalize to UCF101?

Key findings

• RMDN outperforms the trained central bias baseline across AUC, NSS, CC, and similarity metrics on Hollywood2.
• Using an LSTM-based recurrent MDN yields higher saliency prediction accuracy than an RNN baseline and benefits from more fixations per frame.
• The full RMDN with 20 GMM components achieves state-of-the-art saliency results on Hollywood2 (AUC 0.9037, NSS 2.6455, CC 0.6129, Sim 0.5349).
• Prediction-time saliency inference is fast (0.08s per 16-frame clip on GPU).
• Ground-truth saliency maps and predicted maps yield similar improvements when used as soft attention for action recognition on Hollywood2 (mean mAP improvements when concatenating saliency-weighted features).
• On UCF101, combining C3D features with saliency-weighted features yields performance gains (C3D + RMDN + linear SVM improves over context alone by about 1.1%), with further gains when adding a central bias.