[Paper Review] Hybrid Classifiers for Spatio-temporal Real-time Abnormal Behaviors Detection, Tracking, and Recognition in Massive Hajj Crowds

Tarik Alafif, Anas A. Hadi+5 more|arXiv (Cornell University)|Jul 25, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes a hybrid CNN-RF framework for real-time detection, tracking, and recognition of abnormal behaviors in small- and large-scale Hajj crowd videos. It uses fine-tuned ResNet-50 for spatial anomaly detection, Horn-Schunck optical flow for motion-based localization, Kalman filtering for tracking, and random forests on statistical motion features for temporal classification, achieving 99.77% AUC on UMN and 76.08% on the new HAJJv2 dataset.

ABSTRACT

Individual abnormal behaviors vary depending on crowd sizes, contexts, and scenes. Challenges such as partial occlusions, blurring, large-number abnormal behavior, and camera viewing occur in large-scale crowds when detecting, tracking, and recognizing individuals with abnormal behaviors. In this paper, our contribution is twofold. First, we introduce an annotated and labeled large-scale crowd abnormal behaviors Hajj dataset (HAJJv2). Second, we propose two methods of hybrid Convolutional Neural Networks (CNNs) and Random Forests (RFs) to detect and recognize Spatio-temporal abnormal behaviors in small and large-scales crowd videos. In small-scale crowd videos, a ResNet-50 pre-trained CNN model is fine-tuned to verify whether every frame is normal or abnormal in the spatial domain. If anomalous behaviors are observed, a motion-based individuals detection method based on the magnitudes and orientations of Horn-Schunck optical flow is used to locate and track individuals with abnormal behaviors. A Kalman filter is employed in large-scale crowd videos to predict and track the detected individuals in the subsequent frames. Then, means, variances, and standard deviations statistical features are computed and fed to the RF to classify individuals with abnormal behaviors in the temporal domain. In large-scale crowds, we fine-tune the ResNet-50 model using YOLOv2 object detection technique to detect individuals with abnormal behaviors in the spatial domain.

Motivation & Objective

• To address the challenge of detecting, tracking, and recognizing abnormal behaviors in massive Hajj crowds, where high density, occlusions, and camera limitations hinder performance.
• To develop a robust, real-time system capable of distinguishing multiple types of abnormal behaviors (e.g., running, walking against flow) in both small- and large-scale crowd scenarios.
• To introduce a new, large-scale, annotated, and labeled dataset—HAJJv2—specifically for abnormal behavior detection in Hajj crowds.
• To combine deep learning (CNNs) with traditional machine learning (Random Forests) for spatio-temporal anomaly recognition, improving accuracy and generalization.
• To evaluate the method on both public benchmarks (UMN, UCSD) and a newly collected, highly challenging large-scale dataset (HAJJv2).

Proposed method

• Fine-tunes a pre-trained ResNet-50 CNN on small-scale videos to classify each frame as normal or abnormal in the spatial domain.
• Applies Horn-Schunck optical flow to extract motion magnitude and orientation features, generating a binary mask to localize and track individuals with abnormal behaviors.
• Employs a Kalman filter to predict and track individuals across frames in large-scale videos, improving robustness to occlusions and motion blur.
• Computes temporal statistical features—mean, variance, and standard deviation—of motion trajectories for each tracked individual.
• Trains a Random Forest classifier on these temporal features to classify individuals into specific abnormal behavior categories.
• In large-scale settings, integrates YOLOv2 with fine-tuned ResNet-50 to improve object detection accuracy in high-density scenes.

Experimental results

Research questions

• RQ1 Can a hybrid CNN-RF model effectively detect and recognize multiple types of abnormal behaviors in real-time within small-scale crowd videos?
• RQ2 How does the integration of optical flow and Kalman filtering improve tracking robustness in large-scale, high-density crowds with occlusions?
• RQ3 To what extent does the proposed method outperform state-of-the-art approaches on public benchmarks like UMN and UCSD?
• RQ4 How effective is the model on a newly collected, highly challenging large-scale crowd dataset (HAJJv2) with extreme occlusions and distant camera views?
• RQ5 What role do statistical motion features (mean, variance, standard deviation) play in improving temporal classification of abnormal behaviors?

Key findings

• The method achieves 99.77% average AUC on the UMN benchmark dataset, outperforming state-of-the-art methods by 1.67 percentage points.
• On the UCSD Ped1 dataset, the method achieves 93.71% average AUC, surpassing SOTA by 6.06 percentage points.
• On the UCSD Ped2 dataset, the method records 98.55% average AUC, exceeding SOTA by 2.85 percentage points.
• In large-scale crowds, the method achieves 76.08% average AUC on the HAJJv2 dataset, demonstrating satisfactory performance despite extreme challenges like heavy occlusions and distant views.
• The model shows lower performance on UCSD Ped1 due to low resolution, camera angle, shadows, and poor lighting, indicating sensitivity to image quality.
• The Masaa scenes in HAJJv2 were easier to classify due to closer camera views and moderate occlusions, while Tawaf and Jamarat scenes remained the most challenging due to high density and far viewing angles.