[Paper Review] Virtual Worlds as Proxy for Multi-Object Tracking Analysis

Adrien Gaidon, Qiao Wang+2 more|arXiv (Cornell University)|May 20, 2016

Video Surveillance and Tracking Methods32 references297 citations

TL;DR

This paper introduces Virtual KITTI, a photo-realistic synthetic dataset cloned from real KITTI sequences to study real-to-virtual transferability for multi-object tracking and to examine virtual-data benefits for training and evaluation under varied conditions.

ABSTRACT

Modern computer vision algorithms typically require expensive data acquisition and accurate manual labeling. In this work, we instead leverage the recent progress in computer graphics to generate fully labeled, dynamic, and photo-realistic proxy virtual worlds. We propose an efficient real-to-virtual world cloning method, and validate our approach by building and publicly releasing a new video dataset, called Virtual KITTI (see http://www.xrce.xerox.com/Research-Development/Computer-Vision/Proxy-Virtual-Worlds), automatically labeled with accurate ground truth for object detection, tracking, scene and instance segmentation, depth, and optical flow. We provide quantitative experimental evidence suggesting that (i) modern deep learning algorithms pre-trained on real data behave similarly in real and virtual worlds, and (ii) pre-training on virtual data improves performance. As the gap between real and virtual worlds is small, virtual worlds enable measuring the impact of various weather and imaging conditions on recognition performance, all other things being equal. We show these factors may affect drastically otherwise high-performing deep models for tracking.

Motivation & Objective

• Motivate the use of photorealistic synthetic data to enable large-scale, varied, and automatically labeled video datasets for MOT and related tasks.
• Propose a cloning-based pipeline to create virtual worlds from a small seed set of real KITTI sequences.
• Quantify how well observations transfer from real to virtual worlds and demonstrate the value of virtual pre-training for MOT.
• Enable controlled studies of weather, lighting, and viewpoint effects on recognition performance in MOT.
• Provide publicly available Virtual KITTI dataset with automatic ground truth for detection, tracking, depth, segmentation, and optical flow.

Proposed method

• Clone seed real-world KITTI sequences into photo-realistic virtual worlds using a Unity-based pipeline.
• Automatically generate dense ground-truth annotations (2D/3D boxes, depth, segmentation, optical flow) via GPU shaders and rendering passes.
• Create synthetic videos with varied weather and imaging conditions by script-driven modifications (lighting, fog, rain, camera pose).
• Assess transferability by comparing real and cloned virtual videos using pre-trained detectors and optimized tracking hyper-parameters (Bayesian optimization).
• Evaluate virtual pre-training by training on Virtual KITTI clones and fine-tuning on real KITTI to measure performance gains.

Experimental results

Research questions

• RQ1 What is the degree of transferability of recognition performance from real KITTI data to their virtual clones?
• RQ2 Can virtual data pre-training improve real-world MOT performance compared to training solely on real data?
• RQ3 How do weather, lighting, and camera-view variations in virtual worlds affect MOT performance when models are trained on sunny real-world data?
• RQ4 Do virtual worlds provide a scalable, controllable means to study robustness of MOT systems under diverse conditions?

Key findings

• Real-to-virtual transfer is near lossless for MOT metrics on average (MOTA gap < 0.5% for both trackers).
• Virtual pre-training (virtual data followed by real fine-tuning) improves MOT performance, notably for the DP-MCF tracker.
• Weather and imaging variations (fog, rain, night-like conditions) significantly degrade MOT performance when models are trained on ideal sunny real data, with fog causing the strongest drop.
• Ground-truth in Virtual KITTI is consistently generated, reducing annotator subjectivity and enabling dense pixel-level labels across tasks.
• Virtual KITTI enables systematic, ceteris paribus analysis of factors like camera angle and lighting, which would be costly in real data.