[Paper Review] Learning Not to Reconstruct Anomalies

Marcella Astrid, Muhammad Zaigham Zaheer+2 more|arXiv (Cornell University)|Oct 19, 2021

Anomaly Detection Techniques and Applications18 citations

TL;DR

This paper proposes a novel training method for autoencoders in one-class video anomaly detection that explicitly prevents reconstruction of anomalies by leveraging pseudo-anomalies generated via patch and skip-frame manipulations. The approach improves discrimination between normal and anomalous data, achieving state-of-the-art performance on three benchmark datasets without degrading normal reconstruction quality.

ABSTRACT

Video anomaly detection is often seen as one-class classification (OCC) problem due to the limited availability of anomaly examples. Typically, to tackle this problem, an autoencoder (AE) is trained to reconstruct the input with training set consisting only of normal data. At test time, the AE is then expected to well reconstruct the normal data while poorly reconstructing the anomalous data. However, several studies have shown that, even with only normal data training, AEs can often start reconstructing anomalies as well which depletes the anomaly detection performance. To mitigate this problem, we propose a novel methodology to train AEs with the objective of reconstructing only normal data, regardless of the input (i.e., normal or abnormal). Since no real anomalies are available in the OCC settings, the training is assisted by pseudo anomalies that are generated by manipulating normal data to simulate the out-of-normal-data distribution. We additionally propose two ways to generate pseudo anomalies: patch and skip frame based. Extensive experiments on three challenging video anomaly datasets demonstrate the effectiveness of our method in improving conventional AEs, achieving state-of-the-art performance.

Motivation & Objective

• Address the limitation of standard autoencoders reconstructing anomalous inputs despite being trained only on normal data.
• Overcome the poor discrimination between normal and anomalous reconstructions in conventional autoencoders under one-class classification.
• Develop a training mechanism that ensures autoencoders reconstruct only normal data, even when presented with anomalous inputs.
• Introduce two effective pseudo-anomaly generation strategies—patch-based and skip-frame-based—to simulate out-of-distribution anomalies during training.
• Improve anomaly detection performance without compromising the quality of normal data reconstruction.

Proposed method

• Train autoencoders with the objective of reconstructing only normal data, regardless of input type (normal or anomalous).
• Generate pseudo-anomalies by applying patch-based occlusions or skip-frame manipulations to normal video frames to simulate abnormal distributions.
• Use the generated pseudo-anomalies during training to explicitly penalize the model for reconstructing anomalous content.
• Integrate the pseudo-anomaly data into the training loss to encourage the model to learn a reconstruction boundary tightly constrained around normal data.
• Maintain the original autoencoder architecture while modifying the training objective to enforce reconstruction invariance to input type.
• Leverage min-max normalized reconstruction error heatmaps to visualize and validate improved anomaly localization.

Experimental results

Research questions

• RQ1 Can an autoencoder be trained to reconstruct only normal data even when presented with anomalous inputs, thereby improving anomaly detection performance?
• RQ2 How effective are patch-based and skip-frame-based pseudo-anomaly generation methods in simulating out-of-distribution anomalies for one-class training?
• RQ3 Does the proposed method improve discrimination between normal and anomalous reconstructions compared to conventional autoencoders and memory-augmented models?
• RQ4 To what extent does the method preserve high-quality reconstruction of normal data while distorting anomalous inputs?
• RQ5 Can the proposed training strategy achieve state-of-the-art performance on standard video anomaly detection benchmarks?

Key findings

• The proposed method achieves state-of-the-art performance on three challenging video anomaly detection datasets: ShanghaiTech, Avenue, and Ped2.
• The patch-based pseudo-anomaly method achieves optimal performance with a patch probability of p=0.2, while the skip-frame method performs best with skip intervals s=2,3,4,5.
• Qualitative results show that the model successfully reconstructs normal background and suppresses anomalous objects (e.g., a tossed backpack or moving bicycle), producing higher reconstruction errors on anomalies.
• Compared to MemAE, the proposed model maintains better normal reconstruction quality and exhibits superior anomaly discrimination, especially in cases where memory size limits performance.
• Failure cases reveal that thin or visually inconspicuous anomalies (e.g., skateboards, strollers) remain challenging, indicating the need for more sophisticated pseudo-anomaly design.
• The method demonstrates robustness across diverse anomaly types and consistently outperforms the baseline autoencoder and MemAE in both quantitative metrics and visual reconstruction quality.