[Paper Review] MIST: Multiple Instance Self-Training Framework for Video Anomaly Detection

Jia-Chang Feng, Fa-Ting Hong+1 more|arXiv (Cornell University)|Apr 4, 2021

Anomaly Detection Techniques and Applications37 references19 citations

TL;DR

MIST proposes a multiple instance self-training framework for weakly supervised video anomaly detection that improves discriminative feature learning using a sparse continuous sampling strategy for pseudo label generation and a self-guided attention module to highlight anomalous regions. It achieves state-of-the-art performance, attaining 94.83% frame-level AUC on ShanghaiTech with faster inference and lower FLOPs than prior methods.

ABSTRACT

Weakly supervised video anomaly detection (WS-VAD) is to distinguish anomalies from normal events based on discriminative representations. Most existing works are limited in insufficient video representations. In this work, we develop a multiple instance self-training framework (MIST)to efficiently refine task-specific discriminative representations with only video-level annotations. In particular, MIST is composed of 1) a multiple instance pseudo label generator, which adapts a sparse continuous sampling strategy to produce more reliable clip-level pseudo labels, and 2) a self-guided attention boosted feature encoder that aims to automatically focus on anomalous regions in frames while extracting task-specific representations. Moreover, we adopt a self-training scheme to optimize both components and finally obtain a task-specific feature encoder. Extensive experiments on two public datasets demonstrate the efficacy of our method, and our method performs comparably to or even better than existing supervised and weakly supervised methods, specifically obtaining a frame-level AUC 94.83% on ShanghaiTech.

Motivation & Objective

• Address the challenge of weakly supervised video anomaly detection (WS-VAD) with only video-level annotations, which limits representation learning.
• Overcome the issue of noisy pseudo labels in existing methods by refining clip-level labels using a multiple instance learning (MIL) framework.
• Develop a task-specific feature encoder that adaptively focuses on anomalous regions without external annotations.
• Improve efficiency and accuracy by co-optimizing pseudo label generation and feature encoding through a two-stage self-training scheme.
• Reduce domain gap and computational cost while maintaining high detection performance on real-time streaming video.

Proposed method

• Proposes a multiple instance pseudo label generator that uses sparse continuous sampling to generate more reliable clip-level pseudo labels for anomalous videos.
• Introduces a self-guided attention boosted feature encoder (E_SGA) that dynamically highlights anomalous regions using clip-level annotations and pseudo labels.
• Employs a deep MIL ranking loss with sparse-continuous sampling to improve pseudo label quality by focusing on contextual cues around anomalous clips.
• Applies a two-stage self-training scheme: first generating pseudo labels from a pre-trained encoder, then refining the feature encoder using both normal and pseudo-labeled anomalous videos.
• Uses post-processing techniques like min-max normalization and temporal smoothing to reduce label noise, avoiding iterative optimization.
• Leverages gradient accumulation with large batch sizes to further suppress label noise during training.

Figure 1: Our proposed MIST first assign clip-level pseudo labels $\hat{Y}^{a}=\{\hat{y}^{a}_{i}\}$ to anomaly videos with the help of a pseudo label generator $G$ . Then, MIST leverages information from all videos to refine a self-guided attention boosted feature encoder $E_{SGA}$ .

Experimental results

Research questions

• RQ1 Can a self-training framework improve feature representation learning in weakly supervised video anomaly detection using only video-level annotations?
• RQ2 How can pseudo label generation be made more robust to reduce noise from video-level supervision?
• RQ3 Can a self-guided attention mechanism effectively localize anomalous regions without human-annotated bounding boxes?
• RQ4 Does a two-stage self-training process outperform existing iterative or end-to-end training strategies in terms of accuracy and efficiency?
• RQ5 Can the proposed framework achieve state-of-the-art performance while maintaining low computational cost and fast inference speed?

Key findings

• MIST achieves a frame-level AUC of 94.83% on the ShanghaiTech dataset, outperforming existing supervised and weakly supervised methods.
• The MIST-I3D model runs at 324.46 FPS with only 45.68G FLOPs, significantly faster and more efficient than Zhong et al. [35], which requires 10-crop testing and 386.2G FLOPs.
• The self-guided attention module successfully highlights anomalous regions in 5 out of 7 visualized cases, demonstrating effective spatial attention.
• The method improves AUC by +8.48 percentage points (from 58.66% to 67.14%) on ShanghaiTech after refinement, showing the effectiveness of noise reduction techniques.
• The MIST-C3D model achieves 73.37% AUC after refinement, indicating strong performance even with a lighter backbone.
• The framework outperforms the state-of-the-art method Zhong et al. [35] in both accuracy and inference speed, especially on the ShanghaiTech dataset.

Figure 2: Illustration of our proposed MIST framework. MIST includes a multiple instance pseudo label generator $G$ and self-guided attention boosted feature encoder $E_{SGA}$ followed by a weighted-classification head $H_{c}$ . We first train a $G$ and then generate pseudo labels for $E_{SGA}$ fine