[Paper Review] Lucid Data Dreaming for Multiple Object Tracking

Anna Khoreva, Rodrigo Benenson+3 more|arXiv (Cornell University)|Mar 28, 2017

Video Surveillance and Tracking Methods42 references31 citations

TL;DR

This paper proposes Lucid Data Dreaming, a training strategy that generates in-domain synthetic video frames from a single annotated first frame to enable high-performance multiple object tracking with 20x–100x less annotated data than prior methods. By synthesizing plausible future frames, the approach trains appearance- and motion-based models effectively even without ImageNet pre-training, achieving state-of-the-art results on three benchmarks with minimal data.

ABSTRACT

Convolutional networks reach top quality in pixel-level object tracking but require a large amount of training data (1k ~ 10k) to deliver such results. We propose a new training strategy which achieves state-of-the-art results across three evaluation datasets while using 20x ~ 100x less annotated data than competing methods. Instead of using large training sets hoping to generalize across domains, we generate in-domain training data using the provided annotation on the first frame of each video to synthesize ("lucid dream") plausible future video frames. In-domain per-video training data allows us to train high quality appearance- and motion-based models, as well as tune the post-processing stage. This approach allows to reach competitive results even when training from only a single annotated frame, without ImageNet pre-training. Our results indicate that using a larger training set is not automatically better, and that for the tracking task a smaller training set that is closer to the target domain is more effective. This changes the mindset regarding how many training samples and general "objectness" knowledge are required for the object tracking task.

Motivation & Objective

• To reduce the dependency on large-scale annotated datasets for multiple object tracking.
• To address the limitation of current methods that require thousands of annotated frames for high performance.
• To improve tracking performance using minimal, in-domain training data instead of relying on general pre-training or large-scale data.
• To investigate whether domain-specific data augmentation can outperform general data diversity in tracking tasks.

Proposed method

• Synthesize plausible future video frames for each video using the first-frame annotation as a conditional prior.
• Generate in-domain training data by leveraging temporal consistency and object motion priors from the initial annotation.
• Train appearance and motion models end-to-end on the synthetic data, enabling effective feature learning without ImageNet pre-training.
• Fine-tune post-processing modules using the same synthetic data to improve tracking robustness.
• Use a single annotated frame per video as the sole supervision signal, avoiding the need for full video annotations.
• Apply data augmentation techniques that preserve object identity and motion plausibility during frame generation.

Experimental results

Research questions

• RQ1 Can high-quality multiple object tracking be achieved with only a single annotated frame per video?
• RQ2 Does in-domain data generation outperform general-domain data in terms of tracking performance?
• RQ3 Is large-scale annotated data necessary for state-of-the-art tracking performance?
• RQ4 Can synthetic data generation reduce reliance on ImageNet pre-training in tracking models?
• RQ5 Does domain-specific data augmentation lead to better generalization than diverse but out-of-domain data?

Key findings

• The proposed method achieves state-of-the-art performance on three standard multiple object tracking benchmarks using only a single annotated frame per video.
• Performance improves significantly when training data is domain-specific, even with minimal data size, compared to large-scale but out-of-domain datasets.
• The method achieves competitive results without ImageNet pre-training, demonstrating the effectiveness of in-domain data for feature learning.
• Using 20x–100x fewer annotated frames than competing methods still yields superior or comparable tracking accuracy.
• The results challenge the assumption that more training data is always better, showing that data relevance to the target domain is more critical than sheer volume.
• Synthetic data generation from a single frame enables effective tuning of post-processing components, further enhancing tracking robustness.