[Paper Review] Transform Domain Pyramidal Dilated Convolution Networks For Restoration of Under Display Camera Images

P. S. Hrishikesh, Densen Puthussery+2 more|arXiv (Cornell University)|Sep 20, 2020

Advanced Image Processing Techniques18 references4 citations

TL;DR

This paper proposes two novel deep learning architectures using pyramidal dilated convolutions within wavelet and DCT-based dual-domain networks to restore under-display camera (UDC) images. The methods achieve state-of-the-art performance on the ECCV 2020 UDC image restoration challenge, with the P-OLED method winning Track 2 and the T-OLED method securing 4th place based on PSNR and SSIM.

ABSTRACT

Under-display camera (UDC) is a novel technology that can make digital imaging experience in handheld devices seamless by providing large screen-to-body ratio. UDC images are severely degraded owing to their positioning under a display screen. This work addresses the restoration of images degraded as a result of UDC imaging. Two different networks are proposed for the restoration of images taken with two types of UDC technologies. The first method uses a pyramidal dilated convolution within a wavelet decomposed convolutional neural network for pentile-organic LED (P-OLED) based display system. The second method employs pyramidal dilated convolution within a discrete cosine transform based dual domain network to restore images taken using a transparent-organic LED (T-OLED) based UDC system. The first method produced very good quality restored images and was the winning entry in European Conference on Computer Vision (ECCV) 2020 challenge on image restoration for Under-display Camera - Track 2 - P-OLED evaluated based on PSNR and SSIM. The second method scored fourth position in Track-1 (T-OLED) of the challenge evaluated based on the same metrics.

Motivation & Objective

• Address the severe image degradation caused by under-display camera (UDC) systems in modern handheld devices.
• Restore UDC images captured using different display technologies—P-OLED and T-OLED—each presenting unique optical distortions.
• Develop specialized deep learning architectures tailored to the distinct degradation patterns of P-OLED and T-OLED UDC systems.
• Improve image quality in terms of PSNR and SSIM, the primary metrics used in the ECCV 2020 UDC restoration challenge.

Proposed method

• Integrate pyramidal dilated convolutions into a wavelet decomposition-based convolutional neural network for P-OLED UDC image restoration.
• Apply a dual-domain framework combining discrete cosine transform (DCT) and convolutional neural networks for T-OLED UDC image restoration.
• Leverage multi-scale感受 via pyramidal dilated convolutions to capture long-range dependencies and preserve fine details.
• Use wavelet decomposition to separate image features across scales, enabling targeted restoration in sub-bands.
• Design the DCT-based network to operate in both spatial and frequency domains, enhancing texture and edge recovery.
• Train both networks end-to-end using paired real UDC and clean reference images to optimize for PSNR and SSIM.

Experimental results

Research questions

• RQ1 Can a wavelet-based network with pyramidal dilated convolutions effectively restore P-OLED UDC images with high PSNR and SSIM?
• RQ2 Can a DCT-based dual-domain network achieve robust restoration of T-OLED UDC images under complex optical distortions?
• RQ3 How do pyramidal dilated convolutions improve feature representation and restoration quality in UDC image processing?
• RQ4 What is the comparative performance of the two proposed methods on distinct UDC display technologies in a standardized benchmark?

Key findings

• The wavelet-based pyramidal dilated network achieved the highest PSNR and SSIM scores in Track 2 of the ECCV 2020 UDC image restoration challenge, securing the winning position.
• The DCT-based dual-domain network with pyramidal dilated convolutions ranked fourth in Track 1 of the ECCV 2020 challenge for T-OLED UDC image restoration.
• The use of pyramidal dilated convolutions significantly enhanced multi-scale feature learning, improving texture and edge preservation.
• The wavelet decomposition approach effectively isolated degradation artifacts, enabling more precise restoration in sub-bands.
• The DCT-based framework demonstrated strong performance in frequency-domain denoising and artifact suppression for T-OLED images.
• Both methods outperformed baseline approaches on their respective UDC display types, validating the effectiveness of domain-specific network design.