[Paper Review] Kubric: A scalable dataset generator

Klaus Greff, Francois Belletti+33 more|arXiv (Cornell University)|Mar 7, 2022

Remote Sensing and LiDAR Applications4 citations

TL;DR

Kubric is an open-source, scalable Python framework that generates photo-realistic synthetic datasets with rich annotations by integrating PyBullet and Blender, enabling distributed training across thousands of machines and generating terabytes of data. It supports diverse vision tasks—from 3D NeRF to optical flow—through fine-grained control over scene complexity and automatic ground-truth labeling.

ABSTRACT

Data is the driving force of machine learning, with the amount and quality of training data often being more important for the performance of a system than architecture and training details. But collecting, processing and annotating real data at scale is difficult, expensive, and frequently raises additional privacy, fairness and legal concerns. Synthetic data is a powerful tool with the potential to address these shortcomings: 1) it is cheap 2) supports rich ground-truth annotations 3) offers full control over data and 4) can circumvent or mitigate problems regarding bias, privacy and licensing. Unfortunately, software tools for effective data generation are less mature than those for architecture design and training, which leads to fragmented generation efforts. To address these problems we introduce Kubric, an open-source Python framework that interfaces with PyBullet and Blender to generate photo-realistic scenes, with rich annotations, and seamlessly scales to large jobs distributed over thousands of machines, and generating TBs of data. We demonstrate the effectiveness of Kubric by presenting a series of 13 different generated datasets for tasks ranging from studying 3D NeRF models to optical flow estimation. We release Kubric, the used assets, all of the generation code, as well as the rendered datasets for reuse and modification.

Motivation & Objective

• To address the growing challenge of collecting, curating, and annotating real-world data at scale due to cost, privacy, fairness, and licensing concerns.
• To develop a general-purpose, reusable framework for synthetic data generation that supports rich annotations and fine-grained control over scene parameters.
• To enable large-scale, distributed data generation across thousands of machines, producing terabytes of high-quality data efficiently.
• To demonstrate the versatility of the framework by creating 13 diverse datasets for benchmarking vision models across multiple tasks.
• To promote reproducibility, shared assets, and ethical dataset design by providing open-source code, assets, and standardized data formats.

Proposed method

• Kubric integrates with PyBullet for physics simulation and Blender for high-fidelity rendering to generate photo-realistic scenes.
• It uses a simple Python API to define scenes, including objects, cameras, lighting, and scene parameters, enabling programmatic control.
• The framework automatically generates rich annotations such as instance segmentation masks, depth maps, optical flow, and camera poses.
• Kubric supports distributed data generation via scalable job orchestration, enabling TB-scale dataset creation across thousands of machines.
• It employs a standardized data export format to ensure compatibility with existing training pipelines and downstream vision tasks.
• Assets such as 3D objects, materials, and textures are sourced from multiple repositories and integrated into a unified, reusable library.

Figure 1 : Example scene created and rendered with Kubric along with some of the automatically generated annotations.

Experimental results

Research questions

• RQ1 Can a general-purpose synthetic data generation framework be built that supports scalable, distributed, and reproducible dataset creation at scale?
• RQ2 To what extent can synthetic data with rich annotations close the sim-to-real generalization gap in vision tasks like optical flow and 3D scene reconstruction?
• RQ3 How effectively can a unified framework generate diverse datasets across multiple vision tasks while maintaining photo-realism and annotation fidelity?
• RQ4 What are the environmental and ethical trade-offs of generating large-scale synthetic datasets, and how can they be mitigated?
• RQ5 Can synthetic data generation tools promote earlier detection of model biases and failure modes in safety-critical applications like autonomous driving?

Key findings

• Kubric successfully generated 13 diverse, high-quality datasets for tasks ranging from 3D NeRF modeling to optical flow estimation, demonstrating broad applicability.
• The framework enabled the generation of terabytes of data through distributed jobs running on thousands of machines, achieving high scalability and efficiency.
• All generated datasets include rich, automatic annotations such as instance segmentation masks, depth maps, optical flow, and camera poses, supporting multi-task learning.
• The use of synthetic data in Kubric allows for full control over data complexity, lighting, and viewpoint, reducing biases common in real-world datasets.
• The framework's open-source release includes all code, assets, and datasets, enabling reuse, replication, and modification by the research community.
• Despite high computational costs—e.g., ~3 CPU-years for the optical flow dataset—the framework promotes energy efficiency by enabling smaller, controlled experiments before scaling.

Figure 2 : Overview – a typical Kubric worker randomly populates a scene with assets loaded from an external source, possibly runs a physics simulation, renders the resulting frames, and finally exports the images, annotation layers, and other metadata.