[Paper Review] Monitoring COVID-19 social distancing with person detection and tracking via fine-tuned YOLO v3 and Deepsort techniques

Narinder Singh Punn, Sanjay Kumar Sonbhadra+2 more|arXiv (Cornell University)|May 4, 2020

Video Surveillance and Tracking MethodsComputer Science68 references245 citations

TL;DR

The paper proposes a deep-learning framework using fine-tuned YOLO v3 for person detection and Deepsort for tracking to monitor social distancing in surveillance video, and compares it against Faster R-CNN and SSD, introducing a violation index and 3D-centric distance measures for real-time analysis.

ABSTRACT

The rampant coronavirus disease 2019 (COVID-19) has brought global crisis with its deadly spread to more than 180 countries, and about 3,519,901 confirmed cases along with 247,630 deaths globally as on May 4, 2020. The absence of any active therapeutic agents and the lack of immunity against COVID-19 increases the vulnerability of the population. Since there are no vaccines available, social distancing is the only feasible approach to fight against this pandemic. Motivated by this notion, this article proposes a deep learning based framework for automating the task of monitoring social distancing using surveillance video. The proposed framework utilizes the YOLO v3 object detection model to segregate humans from the background and Deepsort approach to track the identified people with the help of bounding boxes and assigned IDs. The results of the YOLO v3 model are further compared with other popular state-of-the-art models, e.g. faster region-based CNN (convolution neural network) and single shot detector (SSD) in terms of mean average precision (mAP), frames per second (FPS) and loss values defined by object classification and localization. Later, the pairwise vectorized L2 norm is computed based on the three-dimensional feature space obtained by using the centroid coordinates and dimensions of the bounding box. The violation index term is proposed to quantize the non adoption of social distancing protocol. From the experimental analysis, it is observed that the YOLO v3 with Deepsort tracking scheme displayed best results with balanced mAP and FPS score to monitor the social distancing in real-time.

Motivation & Objective

• Motivate and enable automated monitoring of social distancing using surveillance footage during COVID-19.
• Evaluate popular object detection and tracking models (YOLO v3, Faster RCNN, SSD) for real-time social distancing measurement.
• Propose a 3D feature-space approach and a violation index to quantify non-adherence to distancing guidelines.
• Visualize clusters and provide real-time indicators of crowding and rule violations in video streams.

Proposed method

• Fine-tune YOLO v3 for person detection and apply Deepsort for tracking with assigned IDs.
• Compare YOLO v3+Deepsort against Faster RCNN and SSD in terms of mean average precision (mAP), FPS, and loss.
• Compute a three-dimensional feature space (centroid coordinates and depth) and derive a pairwise 3D L2 norm to assess social distancing violations.
• Propose a violation index to quantify non-adoption of distancing protocols and color-code bounding boxes to visualize groups.
• Use Kalman filtering and Hungarian algorithm within Deepsort for robust multi-object tracking under occlusion and view changes.

Experimental results

Research questions

• RQ1 How does YOLO v3 with Deepsort perform for real-time social distancing monitoring compared to Faster RCNN and SSD in terms of mAP and FPS?
• RQ2 Can a 3D feature-space based distance metric and a violation index effectively quantify social distancing violations in surveillance footage?
• RQ3 How well do the proposed visualization and tracking methods handle occlusions and crowd scenarios to indicate rule compliance?
• RQ4 What are the trade-offs between speed and accuracy for the detected and tracked people in live video streams?

Key findings

• YOLO v3 with Deepsort achieved the best balance of mean average precision (mAP) and frames per second (FPS) for real-time social distancing monitoring.
• YOLO v3 was compared against Faster R-CNN and SSD in terms of mAP, FPS, and loss, with YOLO v3+Deepsort showing balanced performance.
• A three-dimensional feature space based on bounding box centroids and depth was used to compute pairwise L2 distances to identify proximity-based groups.
• A violation index was proposed to quantify non-adherence to distancing protocols, and color-coded bounding boxes visualize associated groups.
• The framework enables real-time visualization and streaming plots indicating the number of social groups and the overall violation index mustering a scalable monitoring approach.