[Paper Review] ControlMat: A Controlled Generative Approach to Material Capture

Giuseppe Vecchio, R Víctor San Martín+5 more|arXiv (Cornell University)|Sep 4, 2023

Generative Adversarial Networks and Image Synthesis61 references4 citations

TL;DR

ControlMat introduces a diffusion-based generative framework that infers high-resolution, tileable, physically-based material properties (basecolor, roughness, metallic, normal, height, opacity) from a single photograph under uncontrolled lighting. By adapting latent diffusion models with ControlNet conditioning, noise rolling, patched diffusion, and multi-scale inference, it enables high-fidelity, tileable material synthesis while outperforming prior optimization and GAN-based methods.

ABSTRACT

Material reconstruction from a photograph is a key component of 3D content creation democratization. We propose to formulate this ill-posed problem as a controlled synthesis one, leveraging the recent progress in generative deep networks. We present ControlMat, a method which, given a single photograph with uncontrolled illumination as input, conditions a diffusion model to generate plausible, tileable, high-resolution physically-based digital materials. We carefully analyze the behavior of diffusion models for multi-channel outputs, adapt the sampling process to fuse multi-scale information and introduce rolled diffusion to enable both tileability and patched diffusion for high-resolution outputs. Our generative approach further permits exploration of a variety of materials which could correspond to the input image, mitigating the unknown lighting conditions. We show that our approach outperforms recent inference and latent-space-optimization methods, and carefully validate our diffusion process design choices. Supplemental materials and additional details are available at: https://gvecchio.com/controlmat/.

Motivation & Objective

• To address the ill-posed problem of single-image material capture under uncontrolled lighting by formulating it as a controlled generative synthesis task.
• To enable high-resolution, tileable, and physically plausible material generation from a single input image without requiring controlled lighting or multiple captures.
• To overcome limitations of GAN-based and latent-optimization methods, such as mode collapse and instability, by leveraging the stability and controllability of diffusion models.
• To design a diffusion inference process that supports arbitrary resolution generation while preserving tileability and minimizing artifacts.
• To validate the effectiveness of architectural and inference-time modifications through ablation and quantitative comparison with state-of-the-art methods.

Proposed method

• ControlMat uses a latent diffusion model (MatGen) trained on 10 million PBR material renderings from the Substance 3D Materials database to generate 9-channel SVBRDF outputs.
• A ControlNet is trained to condition the diffusion process on a single input photograph, enabling image-guided generation of material properties.
• Noise rolling is introduced during denoising to enforce periodic boundary conditions, enabling tileable output generation.
• Patched diffusion and decoding are employed to reduce peak memory consumption, allowing generation up to 4K resolution with constant memory usage.
• Multi-scale diffusion is used to improve feature fidelity, with blending of overlapping patches and mean matching to eliminate seams.
• Border inpainting is applied to correct misalignment and preserve structure during tileability enforcement.

Experimental results

Research questions

• RQ1 Can a diffusion model be effectively conditioned on a single, uncontrolled photograph to generate high-resolution, physically-based, and tileable materials?
• RQ2 How can the diffusion sampling process be modified to ensure tileability without compromising resolution or quality?
• RQ3 What role do multi-scale, patched, and rolled noise strategies play in enabling high-resolution generation with limited memory?
• RQ4 How does the generative approach compare to optimization-based or GAN-based methods in terms of fidelity, diversity, and robustness to lighting variations?
• RQ5 To what extent does the model generalize to structured or stochastic materials, and where do failures occur?

Key findings

• ControlMat achieves superior qualitative and quantitative results compared to recent inference and latent-optimization methods, including those based on GANs and CLIP-based image translation.
• The ablation study confirms that each component—noise rolling, patched decoding, mean matching, and multi-scale diffusion—significantly improves tileability and reduces artifacts.
• Peak memory consumption is reduced from 20GB to 14GB at 2K resolution by using patched decoding, enabling 4K generation where previously impossible.
• The method successfully generates 4K tileable materials with minimal seams and preserved signal, as validated in supplemental materials.
• Despite its generative nature, ControlMat produces results with high perceptual fidelity, though minor color shifts or incorrect metallic attribution can occur in challenging cases like strong vignetting.
• The approach generalizes well to flash-lit inputs, suggesting potential for future improvements in acquisition precision.