[Paper Review] AutoFlow: Learning a Better Training Set for Optical Flow

Deqing Sun, Daniel Vlasic+7 more|arXiv (Cornell University)|Apr 29, 2021

Advanced Vision and Imaging51 references4 citations

TL;DR

AutoFlow proposes a learnable, 2D layered synthetic data generation method for optical flow pre-training that optimizes rendering hyperparameters to maximize model performance on a target dataset. By using a simple 2D motion and shape parameterization with differentiable training, AutoFlow achieves state-of-the-art results—training RAFT on just 4 augmented examples outperforms training on 22,872 FlyingChairs examples.

ABSTRACT

Synthetic datasets play a critical role in pre-training CNN models for optical flow, but they are painstaking to generate and hard to adapt to new applications. To automate the process, we present AutoFlow, a simple and effective method to render training data for optical flow that optimizes the performance of a model on a target dataset. AutoFlow takes a layered approach to render synthetic data, where the motion, shape, and appearance of each layer are controlled by learnable hyperparameters. Experimental results show that AutoFlow achieves state-of-the-art accuracy in pre-training both PWC-Net and RAFT. Our code and data are available at https://autoflow-google.github.io .

Motivation & Objective

• To address the lack of principled design in synthetic optical flow datasets, which are critical for pre-training deep networks.
• To investigate whether a simpler, learnable rendering pipeline can outperform complex 3D-based synthetic datasets like FlyingChairs and FlyingThings3D.
• To determine which data characteristics—such as motion statistics, object count, or augmentation—most influence model generalization and accuracy.
• To explore whether optimizing the pre-training data distribution directly on a target dataset leads to better downstream performance than using static, manually designed datasets.

Proposed method

• AutoFlow uses a 2D layered rendering pipeline where each layer has learnable hyperparameters for motion, shape, and appearance.
• The rendering process is differentiable, enabling end-to-end optimization of hyperparameters via gradient descent to minimize model error on a target dataset.
• The method jointly optimizes both the synthetic data generation and the model training process, using the target dataset's performance as the objective function.
• Hyperparameters include the number of foreground objects, motion model (with bilinear grid warping), and rendering parameters for shape and appearance.
• Data augmentation (e.g., RandAugment) is applied during training to increase diversity and improve generalization.
• The approach is evaluated by pre-training models like PWC-Net and RAFT on AutoFlow-generated data and measuring performance on Sintel and KITTI benchmarks.

Experimental results

Research questions

• RQ1 Can a simple 2D layered rendering pipeline with learnable hyperparameters outperform complex 3D-based synthetic datasets like FlyingChairs in pre-training optical flow models?
• RQ2 What specific data characteristics (e.g., motion magnitude distribution, number of objects, augmentation type) are most critical for achieving high model accuracy?
• RQ3 Does optimizing the pre-training data distribution directly on a target dataset lead to better generalization than using fixed, manually designed datasets?
• RQ4 How few training examples are needed when using AutoFlow-generated data to achieve performance comparable to large-scale synthetic datasets?
• RQ5 Why does AutoFlow perform well despite having motion statistics that differ significantly from real datasets like Sintel?

Key findings

• Training RAFT on just 4 augmented AutoFlow examples achieves lower AEPE on Sintel.final (3.57) than training on 22,872 FlyingChairs examples with augmentation (3.57 vs. 3.57), with the final model achieving 2.57 AEPE compared to 3.76 on FlyingChairs.
• AutoFlow pre-training reduces PWC-Net’s AEPE on Sintel.final from 4.42 (FlyingChairs) to 2.91, making it competitive with RAFT pre-trained on FlyingChairs.
• The performance gap between PWC-Net and RAFT narrows significantly when both are pre-trained on sufficient AutoFlow data, indicating that AutoFlow enables more effective learning for simpler models.
• AutoFlow’s motion statistics differ from Sintel and FlyingChairs—concentrating on middle-to-high-range motion and underrepresenting small motions, which may be because tiny motions contribute little to the overall error metric.
• Turning off color augmentation causes a severe performance drop on KITTI (AEPE increases from 4.66 to 14.06), highlighting the importance of lighting variation in real-world generalization.
• The method generalizes well: hyperparameters learned on Sintel.final or KITTI yield similar performance on both, suggesting the rendering pipeline acts as a regularizer.