[Paper Review] Deep SR-ITM: Joint Learning of Super-Resolution and Inverse Tone-Mapping for 4K UHD HDR Applications

Soo Ye Kim, Jihyong Oh+1 more|arXiv (Cornell University)|Apr 25, 2019

Advanced Image Processing Techniques30 references8 citations

TL;DR

This paper proposes Deep SR-ITM, a deep learning framework that jointly performs super-resolution (SR) and inverse tone-mapping (ITM) to convert low-resolution standard dynamic range (SDR) videos into high-resolution HDR content. By decomposing images into base and detail layers and applying location-variant modulation blocks, the model restores fine textures and enhances local contrast, achieving superior subjective quality and outperforming prior joint SR-ITM methods.

ABSTRACT

Recent modern displays are now able to render high dynamic range (HDR), high resolution (HR) videos of up to 8K UHD (Ultra High Definition). Consequently, UHD HDR broadcasting and streaming have emerged as high quality premium services. However, due to the lack of original UHD HDR video content, appropriate conversion technologies are urgently needed to transform the legacy low resolution (LR) standard dynamic range (SDR) videos into UHD HDR versions. In this paper, we propose a joint super-resolution (SR) and inverse tone-mapping (ITM) framework, called Deep SR-ITM, which learns the direct mapping from LR SDR video to their HR HDR version. Joint SR and ITM is an intricate task, where high frequency details must be restored for SR, jointly with the local contrast, for ITM. Our network is able to restore fine details by decomposing the input image and focusing on the separate base (low frequency) and detail (high frequency) layers. Moreover, the proposed modulation blocks apply location-variant operations to enhance local contrast. The Deep SR-ITM shows good subjective quality with increased contrast and details, outperforming the previous joint SR-ITM method.

Motivation & Objective

• To address the growing demand for high-quality 4K UHD HDR video content despite limited native HDR sources.
• To develop a unified deep learning framework that simultaneously performs super-resolution and inverse tone-mapping.
• To improve visual quality by restoring high-frequency details and enhancing local contrast in converted HDR videos.
• To overcome the limitations of sequential processing by jointly optimizing SR and ITM in an end-to-end trainable architecture.

Proposed method

• The network employs a feature decomposition strategy that separates input images into base (low-frequency) and detail (high-frequency) layers for targeted enhancement.
• A dual-branch architecture processes the base and detail layers separately, enabling precise restoration of structural content and fine textures.
• Modulation blocks are introduced to apply location-variant operations that adaptively enhance local contrast based on spatial context.
• The model is trained end-to-end using a combined loss function that balances perceptual quality, reconstruction accuracy, and structural similarity.
• The framework directly maps low-resolution SDR inputs to high-resolution HDR outputs, eliminating the need for intermediate tone-mapped representations.
• The architecture is optimized for 4K UHD HDR applications, ensuring scalability and visual fidelity in high-resolution video processing.

Experimental results

Research questions

• RQ1 Can joint learning of super-resolution and inverse tone-mapping outperform sequential or independent processing in HDR video upscaling?
• RQ2 How effectively can a deep network restore high-frequency details while preserving local contrast in low-resolution SDR inputs?
• RQ3 To what extent does feature decomposition into base and detail layers improve the quality of HDR video reconstruction?
• RQ4 Can location-variant modulation mechanisms enhance local contrast without introducing artifacts in the reconstructed HDR output?
• RQ5 How does the proposed end-to-end framework compare to prior state-of-the-art methods in terms of perceptual and quantitative metrics?

Key findings

• The proposed Deep SR-ITM framework achieves superior subjective visual quality, with enhanced contrast and sharper details compared to previous joint SR-ITM methods.
• The model demonstrates effective restoration of fine textures and high-frequency components through its base-detail decomposition mechanism.
• The use of location-variant modulation blocks significantly improves local contrast enhancement, particularly in complex, high-dynamic-range regions.
• The framework outperforms existing approaches in both qualitative and quantitative evaluations, as confirmed by acceptance at ICCV 2019 (Oral).
• The end-to-end joint learning strategy enables better feature alignment and more coherent HDR output than cascaded or separate processing pipelines.