[Paper Review] Learning an Inverse Tone Mapping Network with a Generative Adversarial Regularizer

Shiyu Ning, Hongteng Xu+3 more|arXiv (Cornell University)|Apr 20, 2018

Image Enhancement Techniques12 references3 citations

TL;DR

This paper proposes a deep learning-based inverse tone mapping network (iTMN) that leverages a generative adversarial network (GAN) with a U-Net generator and a CNN discriminator to convert low-dynamic-range (LDR) images to high-dynamic-range (HDR) images. By jointly optimizing content loss and adversarial regularization, the method achieves state-of-the-art performance in HDR reconstruction quality, outperforming existing methods in HDR-VDP-2, mPSNR, and SSIM metrics.

ABSTRACT

Transferring a low-dynamic-range (LDR) image to a high-dynamic-range (HDR) image, which is the so-called inverse tone mapping (iTM), is an important imaging technique to improve visual effects of imaging devices. In this paper, we propose a novel deep learning-based iTM method, which learns an inverse tone mapping network with a generative adversarial regularizer. In the framework of alternating optimization, we learn a U-Net-based HDR image generator to transfer input LDR images to HDR ones, and a simple CNN-based discriminator to classify the real HDR images and the generated ones. Specifically, when learning the generator we consider the content-related loss and the generative adversarial regularizer jointly to improve the stability and the robustness of the generated HDR images. Using the learned generator as the proposed inverse tone mapping network, we achieve superior iTM results to the state-of-the-art methods consistently.

Motivation & Objective

• To address the limitations of traditional iTM methods that rely on heuristic, linear, or segmented mappings with poor color correlation and over-exposure issues.
• To develop an end-to-end deep learning framework capable of learning complex, nonlinear inverse tone mapping from LDR to HDR with high fidelity.
• To improve the robustness and visual quality of generated HDR images by incorporating a generative adversarial regularizer that aligns the distribution of generated HDR images with real HDR data.
• To demonstrate that adversarial training enhances convergence and stability compared to standard MSE-based training.

Proposed method

• A U-Net-based generator is trained to map input LDR images to HDR outputs, with skip connections preserving spatial details.
• A simple CNN-based discriminator is trained to distinguish real HDR images from generated ones, providing adversarial feedback.
• The generator is optimized using a combined loss: a content-based MSE loss and an adversarial regularizer derived from the discriminator's output.
• The training follows an alternating optimization scheme: update discriminator on real and fake HDR images, then update generator to minimize content and adversarial losses.
• The objective function is formulated as min_G max_D λℒ_G(L,H) + ℛ_{G,D}(L,H), where ℒ_G is the content loss and ℛ_{G,D} is the adversarial regularizer.
• The method incorporates a gradient-based dMSE loss to improve structural consistency in high-frequency regions.

Experimental results

Research questions

• RQ1 Can a GAN-based framework improve the visual quality and structural fidelity of inverse tone mapping compared to traditional methods?
• RQ2 How does the inclusion of a generative adversarial regularizer affect the stability and convergence of the HDR generation process?
• RQ3 To what extent does the proposed method outperform state-of-the-art iTM techniques in terms of perceptual and objective metrics?
• RQ4 What hyperparameter settings (e.g., λ, α) yield optimal performance in the combined loss function?

Key findings

• The proposed iTMN achieves the highest HDR-VDP-2 score of 73.77, significantly outperforming the next best method (72.47) and demonstrating superior perceptual quality.
• The method achieves an mPSNR of 21.61, which is higher than the next best method (20.62), indicating lower pixel-level distortion.
• The SSIM score of 0.8098 confirms that the generated HDR images preserve structural integrity better than competing methods, including DrTM (0.7968) and KO (0.6459).
• Ablation studies show that removing the dMSE loss (NoDMSE) or the adversarial regularizer (NoAdvReg) leads to degraded performance, proving the effectiveness of both components.
• The loss function converges faster and more stably with the adversarial regularizer, as shown in convergence curves (Fig. 2h), with consistent descent over iterations.
• Optimal hyperparameters are found to be λ = 10^4 and α = 10^5, yielding the best performance across all metrics in Table 2.