[Paper Review] SSIVD-Net: A Novel Salient Super Image Classification & Detection Technique for Weaponized Violence

Toluwani Aremu, Zhiyuan Li+3 more|arXiv (Cornell University)|Jul 26, 2022

Anomaly Detection Techniques and Applications4 citations

TL;DR

This paper proposes SSIVD-Net, a novel salient-super-image representation technique that transforms 3D surveillance video into 2D salient-super-images to simplify violence detection while reducing computational load and information loss. The method achieves state-of-the-art performance, attaining 100% accuracy on MovieFight and HockeyFight datasets and 90.3% on RWF-2000, with a compact, efficient Salient-Classifier architecture designed for real-world smart city deployment.

ABSTRACT

Detection of violence and weaponized violence in closed-circuit television (CCTV) footage requires a comprehensive approach. In this work, we introduce the \emph{Smart-City CCTV Violence Detection (SCVD)} dataset, specifically designed to facilitate the learning of weapon distribution in surveillance videos. To tackle the complexities of analyzing 3D surveillance video for violence recognition tasks, we propose a novel technique called \emph{SSIVD-Net} ( extbf{S}alient- extbf{S}uper- extbf{I}mage for extbf{V}iolence extbf{D}etection). Our method reduces 3D video data complexity, dimensionality, and information loss while improving inference, performance, and explainability through salient-super-Image representations. Considering the scalability and sustainability requirements of futuristic smart cities, the authors introduce the \emph{Salient-Classifier}, a novel architecture combining a kernelized approach with a residual learning strategy. We evaluate variations of SSIVD-Net and Salient Classifier on our SCVD dataset and benchmark against state-of-the-art (SOTA) models commonly employed in violence detection. Our approach exhibits significant improvements in detecting both weaponized and non-weaponized violence instances. By advancing the SOTA in violence detection, our work offers a practical and scalable solution suitable for real-world applications. The proposed methodology not only addresses the challenges of violence detection in CCTV footage but also contributes to the understanding of weapon distribution in smart surveillance. Ultimately, our research findings should enable smarter and more secure cities, as well as enhance public safety measures.

Motivation & Objective

• Address the lack of specialized datasets for weaponized violence detection in surveillance systems.
• Reduce the computational complexity and dimensionality of 3D video analysis for violence detection without sacrificing performance.
• Improve detection accuracy and explainability for both weaponized and non-weaponized violence in real-time smart city surveillance.
• Develop a scalable, low-carbon, and efficient deep learning model suitable for deployment in resource-constrained urban environments.
• Enable open-world weapon detection by capturing diverse objects used as weapons beyond predefined categories.

Proposed method

• Propose a data-centric approach that transforms 3D video clips into 2D Salient-Super-Images by selecting and aggregating the most discriminative frames based on spatial and temporal saliency.
• Introduce the Salient-Classifier, a novel CNN architecture combining kernelized feature extraction with residual learning to enhance feature representation and model efficiency.
• Use a grid-based sampling strategy (e.g., 6×5 for 30fps SCVD videos, 4×3 for 12fps benchmark videos) to construct the Salient-Super-Image from key frames.
• Apply uniform sampling to ensure consistent frame rates across videos with variable frame rates, enabling stable model training.
• Train the Salient-Classifier on the SCVD dataset and benchmark datasets (MovieFight, HockeyFight, RWF-2000) using 30–50 epochs for convergence.
• Optimize model capacity by comparing variants (SaliNet-2m, 2b, 2n), identifying SaliNet-2m as optimal due to best trade-off between performance and complexity.

Experimental results

Research questions

• RQ1 Can a 2D image-based representation of 3D video data preserve critical spatiotemporal features for violence detection while reducing computational complexity?
• RQ2 How does the Salient-Super-Image transformation compare to traditional Super-Image methods in preserving information and reducing data dimensionality?
• RQ3 To what extent does the Salient-Classifier outperform state-of-the-art 3D CNNs and hybrid models (e.g., I3D, Conv-LSTM) in detecting weaponized and non-weaponized violence?
• RQ4 What is the optimal model capacity for the Salient-Classifier in balancing accuracy, efficiency, and scalability in real-world deployment?
• RQ5 Can the proposed method generalize across diverse datasets with varying lighting, occlusion, and frame rate conditions?

Key findings

• The Salient-Super-Image representation significantly reduces information loss compared to standard Super-Image methods, as validated through ablation studies.
• The Salient-Classifier achieved 100% accuracy on the MovieFight and HockeyFight datasets, demonstrating robustness in detecting fight scenes across diverse video content.
• On the more challenging RWF-2000 benchmark, the best-performing variant (SaliNet-2n) achieved 90.3% accuracy, outperforming existing SOTA models.
• SaliNet-2m showed the best balance between model capacity and inference efficiency, outperforming larger variants (SaliNet-2b and SaliNet-2n) in terms of performance-to-complexity ratio.
• The proposed method reduces computational load and carbon footprint compared to 3D-CNNs and hybrid models, making it suitable for sustainable smart city applications.
• The approach enables effective detection of weaponized violence even in complex scenarios with occlusions and variable lighting, highlighting its practical viability.