[Paper Review] Vision Meets Drones: A Challenge

Pengfei Zhu, Longyin Wen+3 more|arXiv (Cornell University)|Apr 20, 2018

Advanced Image and Video Retrieval TechniquesComputer Science3 references331 citations

TL;DR

VisDrone2018 presents a large-scale drone-based visual object detection and tracking benchmark with 2.5 million annotated instances over 179,264 frames from 14 Chinese cities, spanning four tasks (image/video detection, single and multi-object tracking).

ABSTRACT

In this paper we present a large-scale visual object detection and tracking benchmark, named VisDrone2018, aiming at advancing visual understanding tasks on the drone platform. The images and video sequences in the benchmark were captured over various urban/suburban areas of 14 different cities across China from north to south. Specifically, VisDrone2018 consists of 263 video clips and 10,209 images (no overlap with video clips) with rich annotations, including object bounding boxes, object categories, occlusion, truncation ratios, etc. With intensive amount of effort, our benchmark has more than 2.5 million annotated instances in 179,264 images/video frames. Being the largest such dataset ever published, the benchmark enables extensive evaluation and investigation of visual analysis algorithms on the drone platform. In particular, we design four popular tasks with the benchmark, including object detection in images, object detection in videos, single object tracking, and multi-object tracking. All these tasks are extremely challenging in the proposed dataset due to factors such as occlusion, large scale and pose variation, and fast motion. We hope the benchmark largely boost the research and development in visual analysis on drone platforms.

Motivation & Objective

• Motivate and facilitate visual understanding tasks on drone platforms via a large-scale benchmark.
• Provide rich, diverse annotations for four core tasks to stress-test detection and tracking algorithms.
• Present dataset statistics to enable robust evaluation across urban/drone scenarios.
• Encourage development of algorithms robust to occlusion, scale variation, and rapid motion in aerial imagery.

Proposed method

• Assemble 263 video clips (179,264 frames) and 10,209 static images from drone-captured scenes.
• Annotate over 2.5 million object instances across 10 categories and provide attributes like occlusion and truncation ratios.
• Define four tasks: image-based object detection, video-based object detection, single object tracking, and multi-object tracking.
• Release ground truth for training/validation and withheld testing labels to prevent overfitting, with optional use of external data.
• Offer a public evaluation website for submission and benchmarking across tasks.

Experimental results

Research questions

• RQ1 How well do state-of-the-art detection and tracking algorithms perform on drone-captured imagery with diverse viewpoints, scales, and occlusions?
• RQ2 What are the challenges and limitations of existing methods when applied to aerial drone data, and how can benchmarks guide improvements?
• RQ3 Can a unified drone-focused benchmark drive advances across both detection and tracking tasks in aerial environments?
• RQ4 How do dataset attributes (occlusion, truncation, view changes) impact performance across the four defined tasks.

Key findings

• VisDrone2018 is the largest drone-centered benchmark at the time, containing 263 video clips, 179,264 frames, and 10,209 images.
• The dataset includes over 2.5 million annotated object instances across 10 categories relevant to drone applications.
• Four tasks are established: object detection in images, object detection in videos, single object tracking, and multi-object tracking.
• Ground truth is provided for training/validation while testing ground truths are withheld to avoid overfitting, with an evaluation website for benchmarks.
• The benchmark emphasizes challenging conditions such as occlusion, large scale variation, pose variation, and fast motion in drone footage.