[Paper Review] Anti-UAV: A Large Multi-Modal Benchmark for UAV Tracking

Nan Jiang, Kuiran Wang+8 more|arXiv (Cornell University)|Jan 21, 2021

Video Surveillance and Tracking Methods94 references68 citations

TL;DR

This paper introduces Anti-UAV, a large RGB-T UAV tracking benchmark with 318 video pairs and 580k+ bounding boxes, plus a dual-flow semantic consistency (DFSC) training strategy to improve UAV tracking across multi-modal data.

ABSTRACT

Unmanned Aerial Vehicle (UAV) offers lots of applications in both commerce and recreation. With this, monitoring the operation status of UAVs is crucially important. In this work, we consider the task of tracking UAVs, providing rich information such as location and trajectory. To facilitate research on this topic, we propose a dataset, Anti-UAV, with more than 300 video pairs containing over 580k manually annotated bounding boxes. The releasing of such a large-scale dataset could be a useful initial step in research of tracking UAVs. Furthermore, the advancement of addressing research challenges in Anti-UAV can help the design of anti-UAV systems, leading to better surveillance of UAVs. Besides, a novel approach named dual-flow semantic consistency (DFSC) is proposed for UAV tracking. Modulated by the semantic flow across video sequences, the tracker learns more robust class-level semantic information and obtains more discriminative instance-level features. Experimental results demonstrate that Anti-UAV is very challenging, and the proposed method can effectively improve the tracker's performance. The Anti-UAV benchmark and the code of the proposed approach will be publicly available at https://github.com/ucas-vg/Anti-UAV.

Motivation & Objective

• Introduce a large, multi-modal UAV tracking dataset (RGB and infrared) with paired, non-aligned sequences to study long-distance UAV tracking.
• Provide comprehensive baselines and an evaluation protocol to assess UAV tracking under multi-modal and cross-sequence scenarios.
• Propose DFSC (dual-flow semantic consistency) to learn robust class-level and instance-level features for UAV tracking.
• Offer insights into dataset attributes, challenging scenarios, and evaluation metrics tailored for UAV status perception.

Proposed method

• Construct a 318 RGB-T video pair UAV tracking benchmark with 580k+ manually annotated bounding boxes and diverse scenes.
• Propose dual-flow semantic consistency (DFSC) training: class-level semantic modulation (CSM) across sequences and instance-level semantic modulation (ISM) within a sequence.
• Modulate search region proposals with cross-sequence UAV features in CSM and with current-sequence ROI features in ISM.
• Use a query-guided RPN and a subsequent RCNN stage to integrate modulated features for classification and regression.
• Provide protocol-specific training settings and baselines, including adaptations of GlobalTrack and Faster RCNN for RGB and IR data.

Experimental results

Research questions

• RQ1 How well do current trackers perform on a large-scale, multi-modal UAV tracking benchmark (RGB-T) with non-aligned sequences?
• RQ2 Can cross-sequence semantic modulation improve class-level discrimination for UAVs when only one object category is present?
• RQ3 Does instance-level semantic modulation enhance discrimination among similar UAV instances and distractors within a sequence?
• RQ4 What are the impacts of multi-modal (RGB and IR) data without alignment on UAV tracking performance and evaluation?
• RQ5 What training strategies and evaluation protocols best reveal UAV tracking capabilities on Anti-UAV?

Key findings

• Anti-UAV provides 318 RGB-T video pairs with over 580k bounding boxes and multiple attributes for analysis.
• DFSC significantly improves tracking performance over baselines on Anti-UAV, leveraging both cross-sequence and intra-sequence semantic modulation.
• Experiments show substantial performance variation across trackers and attributes, highlighting the dataset’s challenging nature for UAV tracking.
• Evaluation protocols I–III enable assessment with visible/IR data, training data usage, and multi-modal integration, including non-aligned RGB-T pairs.
• RF-based and deep trackers show varying gains, with DFSC achieving notable improvements compared to several state-of-the-art trackers on the benchmark.