[Paper Review] Attentioned Convolutional LSTM InpaintingNetwork for Anomaly Detection in Videos

Itamar Ben-Ari, Ravid Shwartz-Ziv|arXiv (Cornell University)|Nov 26, 2018

Anomaly Detection Techniques and Applications8 references4 citations

TL;DR

This paper proposes an attentioned Convolutional-LSTM inpainting network for unsupervised video anomaly detection, where a partially masked frame and prior frames are used as input to reconstruct the current frame. The model uses a convolutional attention mechanism to enhance spatial-temporal context modeling and leverages reconstruction error (log-likelihood) as an anomaly score, achieving 87% Equal Error Rate on a corrupted Moving MNIST benchmark, outperforming both VPN and Conv-LSTM baselines.

ABSTRACT

We propose a semi-supervised model for detecting anomalies in videos inspiredby the Video Pixel Network [van den Oord et al., 2016]. VPN is a probabilisticgenerative model based on a deep neural network that estimates the discrete jointdistribution of raw pixels in video frames. Our model extends the Convolutional-LSTM video encoder part of the VPN with a novel convolutional based attentionmechanism. We also modify the Pixel-CNN decoder part of the VPN to a frameinpainting task where a partially masked version of the frame to predict is given asinput. The frame reconstruction error is used as an anomaly indicator. We test ourmodel on a modified version of the moving mnist dataset [Srivastava et al., 2015]. Our model is shown to be effective in detecting anomalies in videos. This approachcould be a component in applications requiring visual common sense.

Motivation & Objective

• To develop a real-time, unsupervised video anomaly detection system that operates efficiently on edge devices.
• To improve spatial and temporal context modeling in video reconstruction for anomaly detection by integrating a dynamic convolutional attention mechanism.
• To enable parallel inference by using a partially masked input frame, eliminating the need for sequential masked convolutions.
• To evaluate the effectiveness of the proposed architecture on detecting both spatial (pixel corruption) and temporal (frame order reversal) anomalies in video sequences.

Proposed method

• The model uses a Convolutional-LSTM encoder to extract spatiotemporal features from a sequence of video frames.
• A novel convolutional-based attention mechanism generates input-dependent filters that enhance local spatial structure awareness in the context window.
• The input to the decoder includes the partially masked current frame (with ~95% masked pixels via a grid mask), enabling the model to reconstruct missing regions using unmasked neighborhood context.
• The decoder is a modified Pixel-CNN that performs frame inpainting by predicting pixel values in parallel, using masked convolutions to maintain autoregressive dependency flow.
• The loss function is the negative log-likelihood of the true pixel values given the predicted distribution, used as an anomaly score.
• During inference, low log-likelihood values indicate anomalies, as the model assigns low probability to unexpected or corrupted pixel values.

Experimental results

Research questions

• RQ1 Can a convolutional attention mechanism improve spatial context modeling in video reconstruction-based anomaly detection compared to standard global attention or no attention?
• RQ2 Does using a partially masked frame as input enable effective parallel inference while maintaining reconstruction quality for anomaly detection?
• RQ3 How does the proposed model compare to baseline models (VPN and Conv-LSTM) in detecting both spatial and temporal anomalies in video sequences?
• RQ4 To what extent does the combination of attention and masked inpainting improve anomaly detection performance on a benchmark dataset?

Key findings

• The proposed model achieves an Equal Error Rate (EER) of 87% on the corrupted Moving MNIST dataset, outperforming the baseline Conv-LSTM (70.3%) and VPN (82.1%) models.
• Removing the masked frame input reduces performance to 84.6% EER, demonstrating the critical role of masked input in enabling effective inpainting and anomaly detection.
• Omitting the convolutional attention mechanism results in 85.7% EER, indicating that attention significantly enhances the model’s ability to capture local spatial structures.
• The model successfully detects both spatial anomalies (e.g., 3×3 black pixel corruption) and temporal anomalies (e.g., frame order reversal), as shown by high loss values in the corresponding regions of the loss map.
• The reconstruction error (log-likelihood) effectively identifies anomalous pixels, with low probability predictions for corrupted pixels such as zero-valued black squares.
• The use of a grid mask with ~95% masked pixels forces the model to rely on temporal context and unmasked neighbors, improving robustness to spatial corruption.