[Paper Review] Video Anomaly Detection and Explanation via Large Language Models

Hui Lv, Qianru Sun|arXiv (Cornell University)|Jan 11, 2024

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes VAD-LLaMA, a video anomaly detection framework that leverages video-based large language models (VLLMs) to eliminate threshold dependency and provide textual explanations for detected anomalies. By introducing a Long-Term Context (LTC) module to enhance long-range video understanding and a three-phase training method to reduce data and annotation costs, the approach achieves state-of-the-art AUC scores of 88.13% on UCF-Crime and 91.77% on TAD, with explainable anomaly detection capabilities.

ABSTRACT

Video Anomaly Detection (VAD) aims to localize abnormal events on the timeline of long-range surveillance videos. Anomaly-scoring-based methods have been prevailing for years but suffer from the high complexity of thresholding and low explanability of detection results. In this paper, we conduct pioneer research on equipping video-based large language models (VLLMs) in the framework of VAD, making the VAD model free from thresholds and able to explain the reasons for the detected anomalies. We introduce a novel network module Long-Term Context (LTC) to mitigate the incapability of VLLMs in long-range context modeling. We design a three-phase training method to improve the efficiency of fine-tuning VLLMs by substantially minimizing the requirements for VAD data and lowering the costs of annotating instruction-tuning data. Our trained model achieves the top performance on the anomaly videos of the UCF-Crime and TAD benchmarks, with the AUC improvements of +3.86\% and +4.96\%, respectively. More impressively, our approach can provide textual explanations for detected anomalies.

Motivation & Objective

• To develop a video anomaly detection (VAD) system that is both threshold-free and inherently explainable, overcoming limitations of score-based methods.
• To address the poor long-range context modeling capability of existing VLLMs in long-duration surveillance videos.
• To reduce dependency on large-scale, domain-specific VAD annotations by designing an efficient three-phase training strategy.
• To enable VLLMs to detect and explain anomalies in a way that aligns with human perception, rather than identifying irrelevant objects as anomalies.
• To achieve state-of-the-art performance on standard weakly-supervised VAD benchmarks while providing multi-turn dialogue-based anomaly explanations.

Proposed method

• Introduces a Long-Term Context (LTC) module that maintains online lists of the top-K most normal and abnormal clip features, integrating them via cross-attention and weighted-sum operations to enrich video representations.
• Employs a three-phase training pipeline: (1) pre-train a baseline VADor, (2) co-train VADor and LTC on self-generated anomaly scores, and (3) fine-tune the VLLM using synthetic instruction templates.
• Uses video clip features extracted by Video-LLaMA’s video encoder as input to the VADor, enabling end-to-end training with minimal human-annotated data.
• Leverages self-supervised anomaly score generation from the baseline VADor to create a synthetic instruction-tuning dataset, minimizing manual annotation effort.
• Applies a dynamic list update mechanism where each new clip is evaluated and the normal/abnormal feature lists are updated in real time during inference.
• Utilizes textual templates to generate instruction-tuning data for VLLMs, enabling them to answer questions like 'Is there an anomaly?' and 'What is the reason for the anomaly?'

Figure 1 : Prediction scores from a baseline VAD model, and clip descriptions by using VLLMs, for a car accident video (as shown in the middle of the figure). On the score curve, the red dashed lines denote anomaly thresholds. The bottom shows the answers from Video-LLaMA [ 24 ] by feeding it with t

Experimental results

Research questions

• RQ1 Can VLLMs be effectively integrated into VAD to produce threshold-free, explainable anomaly detection?
• RQ2 How can VLLMs be adapted to long-range video contexts when they are typically pre-trained on short videos?
• RQ3 Can a three-phase training strategy significantly reduce the need for expensive VAD annotations while maintaining high performance?
• RQ4 Does incorporating long-term normal and abnormal context improve anomaly detection robustness and explainability?
• RQ5 Can the model generate human-aligned explanations that correctly identify the true anomaly (e.g., a car accident) rather than irrelevant objects?

Key findings

• The proposed VAD-LLaMA achieves a new state-of-the-art AUC of 88.13% on the UCF-Crime benchmark, outperforming previous SOTA by +2.26% on TAD and +0.85% on UCF-Crime.
• The model improves AUC by +3.86% on UCF-Crime and +4.96% on TAD compared to the baseline, demonstrating the effectiveness of the LTC and training strategy.
• The ablation study confirms that using both normal and abnormal context lists in the LTC module leads to the highest performance, with K=4 being the optimal list length.
• Class-wise analysis shows that the model significantly outperforms baselines on long-range anomaly classes like 'Arson' and 'Shoplifting', which require contextual reasoning.
• The model generates accurate, human-aligned explanations in multi-turn dialogues, correctly identifying the cause of anomalies such as 'car accident' while avoiding false positives on irrelevant objects.
• The three-phase training method reduces the need for manual annotation of instruction-tuning data, enabling efficient fine-tuning of VLLMs on limited VAD datasets.

Figure 2 : The network architecture of the proposed VAD-LLaMA. It consists of a Video Anomaly Detector (VADor) with the Long-Term Context (LTC) module and a simple Anomaly Predictor (AP), a projection layer (called Adaptor), and the pre-trained Video-LLaMA [ 24 ] (composed by a Video Encoder (VE) an