[Paper Review] Self-Supervised Representation Learning for Visual Anomaly Detection

Rabia Ali, Muhammad Umar Karim Khan+1 more|arXiv (Cornell University)|Jun 17, 2020

Anomaly Detection Techniques and Applications63 references4 citations

TL;DR

This paper proposes a self-supervised frame permutation prediction task for visual anomaly detection in videos, learning spatiotemporal features without optical flow. By training a model to predict the original order of randomly shuffled video frames, it captures rich semantic and structural representations of normal video content, achieving state-of-the-art AUROC performance on UCF101 and ILSVRC2015 datasets, with M=5 frames and skip=2 yielding optimal results.

ABSTRACT

Self-supervised learning allows for better utilization of unlabelled data. The feature representation obtained by self-supervision can be used in downstream tasks such as classification, object detection, segmentation, and anomaly detection. While classification, object detection, and segmentation have been investigated with self-supervised learning, anomaly detection needs more attention. We consider the problem of anomaly detection in images and videos, and present a new visual anomaly detection technique for videos. Numerous seminal and state-of-the-art self-supervised methods are evaluated for anomaly detection on a variety of image datasets. The best performing image-based self-supervised representation learning method is then used for video anomaly detection to see the importance of spatial features in visual anomaly detection in videos. We also propose a simple self-supervision approach for learning temporal coherence across video frames without the use of any optical flow information. At its core, our method identifies the frame indices of a jumbled video sequence allowing it to learn the spatiotemporal features of the video. This intuitive approach shows superior performance of visual anomaly detection compared to numerous methods for images and videos on UCF101 and ILSVRC2015 video datasets.

Motivation & Objective

• To evaluate self-supervised representation learning methods for visual anomaly detection in images and videos, a task underexplored in prior work.
• To investigate the importance of spatial versus temporal features in video anomaly detection using self-supervised learning.
• To develop a simple, effective self-supervision method that learns spatiotemporal coherence in videos without relying on optical flow or external supervision.
• To demonstrate that learning from frame permutation improves detection of out-of-distribution (anomalous) video instances.

Proposed method

• The method introduces a frame permutation prediction task: given a shuffled sequence of M video frames, the model predicts the original frame order.
• The model is trained exclusively on normal video segments using a standard classification head to predict the permutation index.
• Spatial features are extracted using a convolutional neural network (e.g., ResNet) applied to each frame independently.
• Temporal reasoning emerges implicitly as the model learns to reconstruct the correct sequence, capturing object position, shape, and motion dynamics.
• The approach uses only RGB frames and does not require optical flow, bounding boxes, or human-annotated temporal signals.
• Hyperparameters such as the number of frames M and frame skip are tuned empirically to maximize AUROC.

Experimental results

Research questions

• RQ1 How do existing self-supervised visual representation learning methods perform when fine-tuned for image-based visual anomaly detection?
• RQ2 What is the relative contribution of spatial versus temporal features in video anomaly detection using self-supervised learning?
• RQ3 Can a simple frame permutation prediction task effectively learn spatiotemporal representations without optical flow or additional supervision?
• RQ4 How do hyperparameters like frame count M and frame skip affect the performance of the proposed method?

Key findings

• The frame permutation prediction task achieved state-of-the-art AUROC of 76.4% on UCF101 and 75.5% on ILSVRC2015, outperforming prior self-supervised and SOTA methods.
• Using M=5 frames and skipping two frames between selected frames (i.e., {f1, f4, f7, f10, f13}) yielded optimal performance, suggesting non-consecutive frames improve feature diversity.
• The method significantly outperformed models relying only on temporal features (e.g., AOT), indicating that spatial structure is more critical than motion for anomaly detection.
• Image-based self-supervised methods like RotNet showed competitive performance on images, but the proposed video-specific method surpassed them on video anomaly detection.
• The model learned robust representations of in-distribution (normal) video content, enabling accurate detection of out-of-distribution (anomalous) instances.
• Performance plateaued beyond M=5, indicating diminishing returns from longer sequences, and consecutive frames provided insufficient variation for effective learning.