[Paper Review] Learning Memory-guided Normality for Anomaly Detection

Hyunjong Park, Jongyoun Noh+1 more|arXiv (Cornell University)|Mar 30, 2020

Anomaly Detection Techniques and Applications50 references46 citations

TL;DR

An unsupervised video anomaly detection method that uses a memory module to store multiple prototypical normal patterns and learns with feature compactness and separateness losses, achieving state-of-the-art results. It updates memory with a weighted scheme to prevent learning from anomalies.

ABSTRACT

We address the problem of anomaly detection, that is, detecting anomalous events in a video sequence. Anomaly detection methods based on convolutional neural networks (CNNs) typically leverage proxy tasks, such as reconstructing input video frames, to learn models describing normality without seeing anomalous samples at training time, and quantify the extent of abnormalities using the reconstruction error at test time. The main drawbacks of these approaches are that they do not consider the diversity of normal patterns explicitly, and the powerful representation capacity of CNNs allows to reconstruct abnormal video frames. To address this problem, we present an unsupervised learning approach to anomaly detection that considers the diversity of normal patterns explicitly, while lessening the representation capacity of CNNs. To this end, we propose to use a memory module with a new update scheme where items in the memory record prototypical patterns of normal data. We also present novel feature compactness and separateness losses to train the memory, boosting the discriminative power of both memory items and deeply learned features from normal data. Experimental results on standard benchmarks demonstrate the effectiveness and efficiency of our approach, which outperforms the state of the art.

Motivation & Objective

• Address anomaly detection in videos without anomalous training data.
• Explicitly model the diversity of normal patterns using a memory of prototypical features.
• Limit CNNs' reconstruction/prediction capacity to focus on normal patterns via memory.
• Propose stable memory update rules that avoid learning from anomalies.
• Demonstrate state-of-the-art performance on standard benchmarks.

Proposed method

• Introduce a memory module with M items where each item records a prototypical normal pattern.
• Use an encoder (U-Net) to produce per-pixel queries, and read memory via cosine similarity to obtain updated features for reconstruction/prediction.
• Compute a reading step to form hat{p}_t^k as a weighted sum of memory items; concatenate with q_t for decoding.
• Update memory items using queries assigned to each item, with v_t^k,m guiding weighted updates; apply normalization to focus updates on normal frames.
• Train with a reconstruction loss, a feature compactness loss (q_t^k close to its nearest memory item), and a feature separateness loss (push second-nearest item away with a margin).
• At test time, compute a weighted regular score E_t to prevent memory updates on abnormal frames, and derive an abnormality score S_t as a combination of PSNR-based reconstruction quality and memory-based discrepancy.

Experimental results

Research questions

• RQ1 Can a memory-based representation capture the diversity of normal patterns in video frames to improve anomaly detection?
• RQ2 Does enforcing both compactness and separateness among memory items and queries lead to more discriminative normal-pattern prototypes?
• RQ3 Can memory updates be conditioned to avoid absorbing anomalous frames during testing while maintaining performance?
• RQ4 How does memory-guided anomaly detection perform against state-of-the-art methods on standard benchmarks (Ped2, Avenue, ShanghaiTech) in terms of AUC?
• RQ5 What is the trade-off between reconstruction-based and memory-based cues for abnormality scoring?

Key findings

• The memory-augmented model with a prediction task achieves top AUC on Ped2 (97.0) and Avenue (88.5) when using memory, surpassing baselines.
• Memory usage yields consistent gains across Ped2, Avenue, and ShanghaiTech (e.g., Ours-P w/ Mem. achieves 97.0, 88.5, 70.5 respectively).
• Feature separateness loss substantially improves performance (e.g., ablation shows a 3.8 percentage point AUC gain when adding separateness).
• Updating memory only with normal frames (via the weighted regular score) improves anomaly detection performance.
• The method runs at high speed (approx. 67 fps) and offers a favorable accuracy-runtime trade-off compared to Flow-based or adversarial approaches.