[Paper Review] Improving unsupervised anomaly localization by applying multi-scale memories to autoencoders

Yifei Yang, Shibing Xiang+1 more|arXiv (Cornell University)|Dec 21, 2020

Anomaly Detection Techniques and Applications29 references4 citations

TL;DR

This paper proposes Multi-Scale Memory Autoencoders (MMAE), which enhance unsupervised anomaly localization by introducing scale-specific memory modules and a multi-scale attention fuser to improve feature representation across resolutions. By replacing encoded features at each scale with the most relevant prototype features from dedicated memories and fusing them via attention, MMAE achieves superior reconstruction fidelity and anomaly detection performance over baseline autoencoders and MemAE on multiple datasets, with AUC improvements up to 0.949 on UCSD-Ped2.

ABSTRACT

Autoencoder and its variants have been widely applicated in anomaly detection.The previous work memory-augmented deep autoencoder proposed memorizing normality to detect anomaly, however it neglects the feature discrepancy between different resolution scales, therefore we introduce multi-scale memories to record scale-specific features and multi-scale attention fuser between the encoding and decoding module of the autoencoder for anomaly detection, namely MMAE.MMAE updates slots at corresponding resolution scale as prototype features during unsupervised learning. For anomaly detection, we accomplish anomaly removal by replacing the original encoded image features at each scale with most relevant prototype features,and fuse these features before feeding to the decoding module to reconstruct image. Experimental results on various datasets testify that our MMAE successfully removes anomalies at different scales and performs favorably on several datasets compared to similar reconstruction-based methods.

Motivation & Objective

• To address the limitation of MemAE in handling feature discrepancies across different resolution scales during unsupervised anomaly detection.
• To improve reconstruction fidelity of normal regions by preserving scale-specific normal patterns through dedicated memory modules.
• To enhance anomaly localization precision by fusing multi-scale, anomaly-free features using an attention mechanism.
• To maintain the end-to-end training paradigm of autoencoders while strengthening representation capacity without increasing model complexity.

Proposed method

• Introduces multiple memory modules, each storing prototype features at a specific resolution scale during unsupervised pre-training.
• Uses a multi-scale attention fuser to dynamically weigh and combine the most relevant prototype features from each scale before decoding.
• During inference, replaces the original encoded features at each scale with the most similar prototype features from the corresponding scale-specific memory.
• Employs a 3D convolutional encoder-decoder architecture for video data to preserve spatio-temporal coherence.
• Applies a threshold-independent per-region-overlap (PRO) metric to evaluate anomaly segmentation performance across varying false positive rates.
• Trains the model end-to-end using reconstruction loss, with memory updates performed via differentiable addressing during backpropagation.

Experimental results

Research questions

• RQ1 Can scale-specific memory modules improve the representation of normal patterns across multiple resolutions in autoencoder-based anomaly detection?
• RQ2 Does fusing multi-scale, memory-derived features through an attention mechanism lead to better reconstruction of normal regions and improved anomaly localization?
• RQ3 How does the proposed MMAE method compare to MemAE and other SOTA reconstruction-based methods in terms of anomaly detection performance on diverse datasets?
• RQ4 To what extent does the use of multi-scale memories mitigate the degradation of representation capacity caused by linear subspace constraints in memory-augmented autoencoders?

Key findings

• MMAE achieves an AUC of 0.949 on the UCSD-Ped2 video dataset, outperforming MemAE (0.941) and other SOTA methods.
• On the CUHK Avenue dataset, MMAE achieves an AUC of 0.846, surpassing MemAE (0.833) and other reconstruction-based baselines.
• On the ShanghaiTech dataset, MMAE achieves an AUC of 0.731, showing consistent improvement over MemAE (0.712) and other methods.
• The proposed method successfully removes anomalies at multiple scales, as evidenced by improved reconstruction quality on test images with anomalies.
• The use of multi-scale memories enables a larger, more expressive anomaly-free latent space, satisfying both reconstruction fidelity and anomaly non-reconstruction assumptions.
• The attention-based fusion mechanism effectively combines scale-specific, clean features, leading to more accurate and robust reconstruction of normal image regions.