[Paper Review] Learning to Generate Synthetic 3D Training Data through Hybrid Gradient

Dawei Yang, Jia Deng|arXiv (Cornell University)|Jun 29, 2019

3D Shape Modeling and Analysis62 references4 citations

TL;DR

This paper proposes a hybrid gradient method to automatically optimize synthetic 3D training data generation for improved generalization in single-image 3D perception tasks. By combining approximate gradients from finite differences and analytical gradients via backpropagation through unrolled training, the method efficiently optimizes design parameters (e.g., 3D shapes, textures) represented as a continuous vector, achieving state-of-the-art performance with significantly reduced computational cost on depth estimation and intrinsic image decomposition benchmarks.

ABSTRACT

Synthetic images rendered by graphics engines are a promising source for training deep networks. However, it is challenging to ensure that they can help train a network to perform well on real images, because a graphics-based generation pipeline requires numerous design decisions such as the selection of 3D shapes and the placement of the camera. In this work, we propose a new method that optimizes the generation of 3D training data based on what we call "hybrid gradient". We parametrize the design decisions as a real vector, and combine the approximate gradient and the analytical gradient to obtain the hybrid gradient of the network performance with respect to this vector. We evaluate our approach on the task of estimating surface normal, depth or intrinsic decomposition from a single image. Experiments on standard benchmarks show that our approach can outperform the prior state of the art on optimizing the generation of 3D training data, particularly in terms of computational efficiency.

Motivation & Objective

• To address the challenge of generating synthetic 3D training data that generalizes well to real images, particularly when manual design of graphics pipeline parameters is suboptimal.
• To overcome the computational inefficiency of black-box optimization methods in tuning 3D data generation pipelines.
• To enable end-to-end optimization of synthetic data generation by combining analytical and approximate gradients for differentiable and non-differentiable components of the pipeline.
• To improve network generalization on real-world benchmarks for tasks like depth estimation, surface normal prediction, and intrinsic image decomposition.
• To demonstrate that hybrid gradient optimization outperforms prior black-box methods in both accuracy and training efficiency.

Proposed method

• Parametrize design decisions—such as 3D shape composition, texture generation, and lighting—as a continuous real-valued vector β.
• Use finite difference approximations averaged over random directions to estimate the gradient from β to generated training images X, where analytical gradients are unavailable due to photorealistic rendering.
• Compute the analytical gradient from training images X to network performance L via backpropagation through unrolled SGD training steps, enabling differentiability.
• Combine the approximate gradient (β → X) and analytical gradient (X → L) to form a hybrid gradient ∇βL for optimizing β.
• Represent complex 3D shapes using a Probabilistic Context-Free Grammar (PCFG) to enable differentiable sampling and parameterization of shape generation.
• Apply the hybrid gradient to optimize β for specific downstream tasks, such as depth prediction and intrinsic image decomposition, using standard deep networks like Stacked Hourglass.

Experimental results

Research questions

• RQ1 Can hybrid gradient optimization outperform black-box methods in generating synthetic 3D training data that generalizes well to real images?
• RQ2 How does combining approximate and analytical gradients improve optimization efficiency in data generation pipelines?
• RQ3 To what extent can differentiable parametrization of 3D shapes and textures enhance the performance of downstream 3D perception networks?
• RQ4 Does the hybrid gradient approach achieve better generalization on real-world benchmarks compared to random or basic random search baselines?
• RQ5 Can the method scale to complex data generation tasks such as intrinsic image decomposition with multi-channel supervision?

Key findings

• On the Basel Face Model benchmark, the hybrid gradient method achieved a relative difference of 0.02256 (vs. 0.03718 for random β), representing a 39.5% improvement over random initialization.
• For depth estimation, the hybrid gradient reduced RMSE (linear) to 0.0293, outperforming basic random search (0.0299) and random β (0.1014), with a 70.8% reduction in error compared to random β.
• In intrinsic image decomposition on ShapeNet, the hybrid gradient reduced albedo RMSE from 0.198 (random β) to 0.189 and reconstruction error from 0.169 to 0.150, showing consistent improvement.
• The method achieved state-of-the-art performance on all evaluated tasks while requiring significantly fewer evaluations than black-box optimization, demonstrating superior computational efficiency.
• The hybrid gradient approach enabled end-to-end optimization of complex procedural pipelines for 3D shape and texture generation, with measurable gains in downstream network generalization.
• The use of PCFG-based shape parametrization allowed for differentiable, continuous optimization of complex 3D geometry, enabling effective gradient-based tuning.