[Paper Review] 3D-Aware Scene Manipulation via Inverse Graphics

Shunyu Yao, Tzu Ming Harry Hsu+5 more|arXiv (Cornell University)|Aug 28, 2018

Computer Graphics and Visualization Techniques10 references44 citations

TL;DR

The paper introduces 3D-SDN, an encoder-decoder framework that learns disentangled semantics, geometry, and appearance for scenes, enabling 3D-aware editing through a differentiable shape renderer and texture generator. It demonstrates superior 3D-aware editing over 2D baselines on Virtual KITTI and Cityscapes, with an image editing benchmark and ablations.

ABSTRACT

We aim to obtain an interpretable, expressive, and disentangled scene representation that contains comprehensive structural and textural information for each object. Previous scene representations learned by neural networks are often uninterpretable, limited to a single object, or lacking 3D knowledge. In this work, we propose 3D scene de-rendering networks (3D-SDN) to address the above issues by integrating disentangled representations for semantics, geometry, and appearance into a deep generative model. Our scene encoder performs inverse graphics, translating a scene into a structured object-wise representation. Our decoder has two components: a differentiable shape renderer and a neural texture generator. The disentanglement of semantics, geometry, and appearance supports 3D-aware scene manipulation, e.g., rotating and moving objects freely while keeping the consistent shape and texture, and changing the object appearance without affecting its shape. Experiments demonstrate that our editing scheme based on 3D-SDN is superior to its 2D counterpart.

Motivation & Objective

• Aim to obtain an interpretable, expressive, and disentangled scene representation that encodes scene semantics, object geometry/pose, and appearance.
• Enable 3D-aware scene manipulation such as moving and rotating objects while preserving shape and texture.
• Leverage a differentiable renderer to fuse geometry and texture for faithful scene reconstruction.
• Provide an encoder-decoder framework that supports object-wise editing and evaluation against 2D baselines.

Proposed method

• Use a three-branch encoder (semantic, geometry, texture) to de-render an image into structured object-wise representations.
• Apply a differentiable shape renderer to infer 3D mesh, pose, and Free-Form Deformation (FFD) for each object.
• Employ a textural de-renderer with instance-wise texture codes and a conditional GAN-based texture renderer to reconstruct the image.
• Decouple geometry and texture by feeding 3D pose/normal maps into the texture branch to discourage pose-ambiguous texture encodings.
• Train with a combination of 3D attribute prediction loss, reprojection loss, GAN-based texture loss, feature matching, and perceptual losses.

Experimental results

Research questions

• RQ1 Can an encoder-decoder framework learn disentangled semantic, geometric, and textural representations for scenes?
• RQ2 Does incorporating a differentiable 3D renderer enable accurate 3D-aware editing such as changing object pose/position while preserving texture?
• RQ3 What is the contribution of explicit 3D supervision and reprojection consistency to 3D attribute estimation and rendering fidelity?
• RQ4 How does 3D-aware editing compare to purely 2D editing approaches in terms of perceptual quality and realism?

Key findings

• The 3D-SDN framework enables 3D-aware editing by decoupling semantics, geometry, and appearance, allowing pose/position changes without altering textures and vice versa.
• On Virtual KITTI and Cityscapes, 3D-aware editing outperforms 2D baselines in perceptual metrics and human judgments.
• An ablation study shows that incorporating multiple CAD meshes with Free-Form Deformation (FFD), reprojection loss, and proper quaternion/spacing constraints improves 3D attribute accuracy and silhouette reprojection.
• Quantitative results indicate lower LPIPS scores and higher human-preference for 3D-SDN editing versus 2D baselines on the provided benchmarks.
• The method yields consistent 3D representations (instance, pose, normal maps) that support robust 3D manipulations and occlusion reasoning.