[Paper Review] Lasagna: Layered Score Distillation for Disentangled Object Relighting

Dina Bashkirova, Arijit Ray+5 more|arXiv (Cornell University)|Nov 30, 2023

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes Lasagna, a text-guided, layered score distillation method for disentangled object relighting that preserves image content while enabling intuitive lighting control. By distilling a diffusion model's prior on a synthetic relighting dataset (R-eLiT), Lasagna achieves state-of-the-art performance on real and artistic images, outperforming baselines in 91% of human preference evaluations.

ABSTRACT

Professional artists, photographers, and other visual content creators use object relighting to establish their photo's desired effect. Unfortunately, manual tools that allow relighting have a steep learning curve and are difficult to master. Although generative editing methods now enable some forms of image editing, relighting is still beyond today's capabilities; existing methods struggle to keep other aspects of the image -- colors, shapes, and textures -- consistent after the edit. We propose Lasagna, a method that enables intuitive text-guided relighting control. Lasagna learns a lighting prior by using score distillation sampling to distill the prior of a diffusion model, which has been finetuned on synthetic relighting data. To train Lasagna, we curate a new synthetic dataset ReLiT, which contains 3D object assets re-lit from multiple light source locations. Despite training on synthetic images, quantitative results show that Lasagna relights real-world images while preserving other aspects of the input image, outperforming state-of-the-art text-guided image editing methods. Lasagna enables realistic and controlled results on natural images and digital art pieces and is preferred by humans over other methods in over 91% of cases. Finally, we demonstrate the versatility of our learning objective by extending it to allow colorization, another form of image editing.

Motivation & Objective

• To address the lack of geometry-aware, controllable relighting in text-guided image editing methods.
• To reduce the sim-to-real domain gap in relighting by disentangling lighting edits from other image factors.
• To enable intuitive, language-driven relighting without requiring manual prompt tuning or extensive hyperparameter adjustment.
• To demonstrate generalization of a synthetic relighting prior to real-world and abstract digital art images.
• To extend the method’s utility beyond relighting to other editing tasks, such as sketch colorization.

Proposed method

• Leverages score distillation sampling (SDS) to distill a geometry-aware lighting prior from a diffusion model fine-tuned on synthetic relighting data.
• Represents relighting as a luminosity adjustment in pixel space, enabling disentangled edits that preserve colors, textures, and shapes.
• Introduces a layered editing framework where lighting is applied as a separate edit layer composed with the original image via weighted blending.
• Trains on a new synthetic dataset, R-eLiT, containing over 14,000 3D-rendered objects with multiple light source configurations and directions.
• Uses a single prompt template for all relighting instructions, minimizing the need for prompt tuning.
• Extends the framework to colorization by introducing a colorization layer and transparency layer, composed via element-wise blending with a CLIP-based structure regularizer.

Experimental results

Research questions

• RQ1 Can a diffusion model fine-tuned on synthetic relighting data generalize to real-world images without altering non-lighting attributes?
• RQ2 Can score distillation sampling effectively extract a geometry-aware lighting prior from a diffusion model for controlled editing?
• RQ3 Does a layered editing approach with luminosity adjustment preserve image content better than end-to-end diffusion-based editing?
• RQ4 Can a single, unified prior trained on a synthetic dataset support high-quality relighting across diverse domains, including natural images and digital art?
• RQ5 Can the same learning objective be extended to other editing tasks, such as sketch colorization, with minimal architectural changes?

Key findings

• Lasagna outperforms state-of-the-art text-guided editing methods, including InstructPix2Pix and P2P, in over 91% of human preference evaluations across multiple datasets.
• On the DreamBooth dataset, Lasagna is preferred over ControlNet in 91% of cases, demonstrating the effectiveness of prior distillation in mitigating sim-to-real gaps.
• Lasagna achieves 92% preference over Text2Live and DDS on the R-eLiT test set, confirming the importance of its layered and distillation-based design.
• The method generalizes well to digital art and natural images, maintaining consistent colors, textures, and shapes while enabling accurate lighting direction control.
• Failure cases include overexposure artifacts in backgrounds and challenges with highly abstract inputs, suggesting potential for improvement via foreground masking.
• The method successfully extends to sketch colorization, achieving plausible results using a modified layer composition and variational score distillation.