[Paper Review] Discriminative-Generative Dual Memory Video Anomaly Detection

Xin Guo, Zhongming Jin+6 more|arXiv (Cornell University)|Apr 29, 2021

Anomaly Detection Techniques and Applications35 references4 citations

TL;DR

This paper proposes DREAM, a discriminative-generative dual memory model for video anomaly detection that addresses data imbalance by using two discriminators to tighten normal feature boundaries and a generator for next-frame prediction. A novel dual memory module separately stores normality and abnormality prototypes, enabling effective learning from few anomalies and achieving state-of-the-art performance across four benchmark datasets without extra parameters.

ABSTRACT

Recently, people tried to use a few anomalies for video anomaly detection (VAD) instead of only normal data during the training process. A side effect of data imbalance occurs when a few abnormal data face a vast number of normal data. The latest VAD works use triplet loss or data re-sampling strategy to lessen this problem. However, there is still no elaborately designed structure for discriminative VAD with a few anomalies. In this paper, we propose a DiscRiminative-gEnerative duAl Memory (DREAM) anomaly detection model to take advantage of a few anomalies and solve data imbalance. We use two shallow discriminators to tighten the normal feature distribution boundary along with a generator for the next frame prediction. Further, we propose a dual memory module to obtain a sparse feature representation in both normality and abnormality space. As a result, DREAM not only solves the data imbalance problem but also learn a reasonable feature space. Further theoretical analysis shows that our DREAM also works for the unknown anomalies. Comparing with the previous methods on UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech, our model outperforms all the baselines with no extra parameters. The ablation study demonstrates the effectiveness of our dual memory module and discriminative-generative network.

Motivation & Objective

• To address the data imbalance problem in video anomaly detection where a few anomalies are present among vast amounts of normal data.
• To design a dedicated architecture that explicitly leverages the uniqueness of few abnormal samples for better feature discrimination.
• To learn a more robust and disentangled feature space for both normal and abnormal patterns through dual memory mechanisms.
• To improve generalization to unknown anomaly types by enhancing feature discriminability.

Proposed method

• The model employs two shallow discriminators to tighten the boundary of the normal feature distribution, improving discriminative power.
• A frame prediction generator is used to learn temporal dynamics, with reconstruction error used as an anomaly indicator.
• A dual memory module maintains separate memory banks for normality and abnormality, updating them independently to prevent dominance by normal data.
• Query vectors from input clips are compared to both memory banks using L2 distance to compute anomaly scores.
• The model uses a ratio-based scoring mechanism: r_normal = sum(d(U^N, M^N)) / sum(d(U^N, M^A)) and r_abnormal = sum(d(U^A, M^A)) / sum(d(U^A, M^N)) to distinguish normal and abnormal clips.
• The memory update rule ensures that both normal and abnormal patterns are memorized effectively, even when anomalies are rare.

Experimental results

Research questions

• RQ1 Can a dual memory structure effectively mitigate data imbalance in video anomaly detection with only a few abnormal samples?
• RQ2 How does separating normality and abnormality memory banks improve feature discrimination and model generalization?
• RQ3 Can the proposed model detect both known and unknown anomaly types effectively?
• RQ4 Does the combination of discriminative and generative components enhance anomaly detection performance compared to existing methods?

Key findings

• DREAM outperforms all baseline methods on UCSD Ped1, UCSD Ped2, CUHK Avenue, and ShanghaiTech datasets without adding extra parameters.
• The model achieves a ratio of r_normal = 15.48 on Ped1 and r_abnormal = 12.72 on Ped1, indicating strong separation between normal and abnormal features.
• The ablation study confirms that both the dual memory module and the discriminative-generative network design significantly contribute to performance gains.
• t-SNE visualization on the Avenue dataset shows clear clustering of normal (blue) and abnormal (red) features in the dual memory space.
• Case studies demonstrate that DREAM produces higher reconstruction errors on anomalous regions, confirming sensitivity to dynamic and static anomalies.
• Theoretical and empirical analysis shows that DREAM generalizes well to unknown anomaly types, even without seeing them during training.