[Paper Review] EverLight: Indoor-Outdoor Editable HDR Lighting Estimation

Mohammad Reza Karimi Dastjerdi, Jonathan Eisenmann+2 more|arXiv (Cornell University)|Apr 26, 2023

Advanced Vision and Imaging4 citations

TL;DR

EverLight proposes a unified, editable HDR lighting estimation method that generates high-fidelity 360° panoramas from single images, combining parametric spherical Gaussian light sources with GAN-based panorama generation. It enables seamless indoor and outdoor scene editing by introducing lighting co-modulation, achieving state-of-the-art results in both realism and editability.

ABSTRACT

Because of the diversity in lighting environments, existing illumination estimation techniques have been designed explicitly on indoor or outdoor environments. Methods have focused specifically on capturing accurate energy (e.g., through parametric lighting models), which emphasizes shading and strong cast shadows; or producing plausible texture (e.g., with GANs), which prioritizes plausible reflections. Approaches which provide editable lighting capabilities have been proposed, but these tend to be with simplified lighting models, offering limited realism. In this work, we propose to bridge the gap between these recent trends in the literature, and propose a method which combines a parametric light model with 360° panoramas, ready to use as HDRI in rendering engines. We leverage recent advances in GAN-based LDR panorama extrapolation from a regular image, which we extend to HDR using parametric spherical gaussians. To achieve this, we introduce a novel lighting co-modulation method that injects lighting-related features throughout the generator, tightly coupling the original or edited scene illumination within the panorama generation process. In our representation, users can easily edit light direction, intensity, number, etc. to impact shading while providing rich, complex reflections while seamlessly blending with the edits. Furthermore, our method encompasses indoor and outdoor environments, demonstrating state-of-the-art results even when compared to domain-specific methods.

Motivation & Objective

• Address the gap between domain-specific lighting estimation methods (indoor vs. outdoor) by creating a unified framework.
• Enable user-editable HDR lighting parameters (intensity, direction, color, number of lights) while preserving realistic reflections and shading.
• Combine the realism of GAN-based panorama generation with the interpretability of parametric lighting models.
• Achieve competitive performance across both indoor and outdoor environments without domain-specific adaptation.
• Introduce a novel lighting co-modulation mechanism to tightly couple lighting edits with generator features during image synthesis.

Proposed method

• Leverage a GAN-based generator trained on LDR panorama extrapolation, extended to HDR using spherical Gaussian light representations.
• Introduce a lighting co-modulation module that injects lighting features throughout the generator’s residual blocks, ensuring consistent blending of edited lights with scene geometry.
• Use a learned lighting encoder to predict HDR spherical Gaussian parameters (position, intensity, color) from a single RGB image.
• Train the model end-to-end using a combination of perceptual, adversarial, and L1 loss terms to preserve texture and lighting fidelity.
• Apply dynamic range extension to input panoramas using a proxy network to recover saturated pixels, improving HDR estimation accuracy.
• Use a feed-forward generator with style and lighting co-modulation to ensure disentangled, controllable editing of lighting attributes.

Experimental results

Research questions

• RQ1 Can a single deep learning framework achieve state-of-the-art performance in HDR lighting estimation for both indoor and outdoor scenes?
• RQ2 How effective is lighting co-modulation in enabling realistic blending of editable light sources with complex scene textures?
• RQ3 To what extent can a GAN-based generator produce high-quality, editable HDR panoramas without relying on specialized hardware or domain-specific data?
• RQ4 Does combining parametric light modeling with GAN-based synthesis outperform purely parametric or purely generative approaches in realism and editability?
• RQ5 Can the method generalize across diverse lighting conditions while maintaining accurate shading and reflective properties?

Key findings

• EverLight achieves a Fréchet Inception Distance (FID) of 38.44 on outdoor test sets and 78.90 on indoor sets, outperforming non-editable and domain-specific baselines.
• Ablation studies show that removing lighting co-modulation increases FID to 50.29 on outdoor scenes, indicating its critical role in realistic light blending.
• Qualitative results demonstrate that edited lights produce plausible reflections on surfaces like water and polished floors, even in complex scenes.
• The method successfully generates realistic HDR panoramas that support intuitive user editing of light parameters without compromising texture quality.
• Despite using proxy ground-truth lighting, the model generalizes well across in-the-wild scenes, showing robustness to lighting diversity.
• The framework maintains competitive performance against domain-specific methods, proving its effectiveness as a unified solution for indoor and outdoor lighting estimation.