[Paper Review] SceneNet RGB-D: 5M Photorealistic Images of Synthetic Indoor Trajectories with Ground Truth

John McCormac, Ankur Handa+2 more|ArXiv.org|Dec 15, 2016

Advanced Vision and Imaging18 references91 citations

TL;DR

SceneNet RGB-D introduces a large-scale synthetic dataset of 5 million photorealistic RGB-D images from 15,000 synthetic indoor trajectories, featuring pixel-perfect ground truth for semantic segmentation, instance segmentation, optical flow, depth estimation, camera pose, and 3D reconstruction. The dataset is generated using physically simulated object placements, random lighting, textured layouts, and motion-blurred camera trajectories via a ray-traced rendering pipeline, enabling high-fidelity pre-training for data-hungry vision models.

ABSTRACT

We introduce SceneNet RGB-D, expanding the previous work of SceneNet to enable large scale photorealistic rendering of indoor scene trajectories. It provides pixel-perfect ground truth for scene understanding problems such as semantic segmentation, instance segmentation, and object detection, and also for geometric computer vision problems such as optical flow, depth estimation, camera pose estimation, and 3D reconstruction. Random sampling permits virtually unlimited scene configurations, and here we provide a set of 5M rendered RGB-D images from over 15K trajectories in synthetic layouts with random but physically simulated object poses. Each layout also has random lighting, camera trajectories, and textures. The scale of this dataset is well suited for pre-training data-driven computer vision techniques from scratch with RGB-D inputs, which previously has been limited by relatively small labelled datasets in NYUv2 and SUN RGB-D. It also provides a basis for investigating 3D scene labelling tasks by providing perfect camera poses and depth data as proxy for a SLAM system. We host the dataset at http://robotvault.bitbucket.io/scenenet-rgbd.html

Motivation & Objective

• To address the scarcity of large-scale, high-quality, fully supervised RGB-D datasets for training data-hungry deep learning models.
• To reduce reliance on expensive manual annotation by generating photorealistic synthetic data with complete ground truth.
• To enable pre-training of computer vision models from scratch using realistic, diverse, and scalable synthetic data.
• To support temporal and geometric vision tasks by providing consistent camera trajectories and metric depth.
• To facilitate domain adaptation and 3D scene understanding through physically plausible, randomized indoor scene configurations.

Proposed method

• Photorealistic RGB-D images are generated using a ray-tracing renderer with full lighting effects, motion blur, and realistic camera response functions.
• Synthetic scenes are procedurally generated by randomly placing 3D objects from ShapeNet into physically simulated layouts using the Chrono Engine to ensure stable, realistic configurations.
• Camera trajectories are automatically generated to simulate natural human-like motion, with smooth, continuous paths and interpolated poses for motion blur.
• Random lighting is applied using 1–5 light sources (point and area lights) with randomized hue, power, and position, biased toward the upper half of the scene.
• Textures are randomly sampled from large collections of real-world materials (e.g., walls, floors, curtains), ensuring visual diversity across scenes.
• A non-linear Camera Response Function (CRF) is applied to simulate real camera behavior, and motion blur is implemented via temporal integration of camera poses during rendering.

Experimental results

Research questions

• RQ1 Can a fully synthetic, photorealistic RGB-D dataset with complete per-pixel ground truth be generated at scale to support data-hungry vision models?
• RQ2 How effective is synthetic data with realistic lighting, motion blur, and physically plausible object placements for pre-training models in tasks like optical flow and depth estimation?
• RQ3 To what extent can random scene and trajectory generation produce diverse, realistic, and scalable training data without manual design or annotation?
• RQ4 Can such a dataset enable better generalization and domain adaptation in real-world robotics and augmented reality applications?
• RQ5 How does the inclusion of accurate camera trajectory and depth data improve 3D scene labelling and SLAM-like system training?

Key findings

• The dataset comprises 5 million high-quality RGB-D images from over 15,000 unique camera trajectories in diverse, randomly generated indoor layouts.
• Each image is accompanied by pixel-perfect annotations for semantic segmentation, instance segmentation, optical flow, depth, and camera pose, enabling multi-task learning.
• The use of ray-tracing and realistic CRF ensures photorealistic image quality, closely mimicking real-world camera behavior.
• Motion blur is effectively simulated through temporal integration of interpolated camera poses, enhancing realism without affecting ground truth labels.
• Randomized lighting and texture application result in visually distinct renderings of the same geometric layout, increasing data diversity.
• Despite limitations such as static scenes and lack of dynamic physics, the dataset enables robust pre-training and shows strong potential for domain adaptation in robotics and AR.