[Paper Review] Towards Fast and Light-Weight Restoration of Dark Images

Mohit Lamba, Atul Balaji+1 more|arXiv (Cornell University)|Nov 28, 2020

Image Enhancement Techniques24 references15 citations

TL;DR

This paper proposes LLPackNet, a fast and lightweight deep learning architecture for real-time enhancement of extremely dark, high-resolution images. By performing most computations in a low-resolution space using novel Pack and UnPack operations, the method reduces model parameters by 2–7×, memory usage by 2–3×, and inference time by 5–20× compared to state-of-the-art methods, achieving near real-time performance (3 seconds on CPU) for 2848×4256 images while maintaining competitive reconstruction quality.

ABSTRACT

The ability to capture good quality images in the dark and near-zero lux conditions has been a long-standing pursuit of the computer vision community. The seminal work by Chen et al. [5] has especially caused renewed interest in this area, resulting in methods that build on top of their work in a bid to improve the reconstruction. However, for practical utility and deployment of low-light enhancement algorithms on edge devices such as embedded systems, surveillance cameras, autonomous robots and smartphones, the solution must respect additional constraints such as limited GPU memory and processing power. With this in mind, we propose a deep neural network architecture that aims to strike a balance between the network latency, memory utilization, model parameters, and reconstruction quality. The key idea is to forbid computations in the High-Resolution (HR) space and limit them to a Low-Resolution (LR) space. However, doing the bulk of computations in the LR space causes artifacts in the restored image. We thus propose Pack and UnPack operations, which allow us to effectively transit between the HR and LR spaces without incurring much artifacts in the restored image. We show that we can enhance a full resolution, 2848 x 4256, extremely dark single-image in the ballpark of 3 seconds even on a CPU. We achieve this with 2 - 7x fewer model parameters, 2 - 3x lower memory utilization, 5 - 20x speed up and yet maintain a competitive image reconstruction quality compared to the state-of-the-art algorithms.

Motivation & Objective

• Address the challenge of real-time, low-resource enhancement of extremely dark, high-resolution images on edge devices.
• Overcome the limitations of existing methods that require high GPU memory and processing power.
• Design a network that maintains high reconstruction quality while minimizing model size, latency, and memory footprint.
• Develop a self-contained mechanism for automatic amplification factor estimation without relying on ground truth during inference.
• Enable practical deployment of low-light enhancement on embedded systems, smartphones, and surveillance cameras with limited computational resources.

Proposed method

• Propose a novel Pack operation that downsamples feature maps by a factor of α by rearranging spatial pixels into channel dimensions, preserving information and enabling large-scale downsampling.
• Introduce the UnPack operation as a learnable, fast upsampling alternative to transposed convolution and interpolation, enabling efficient reconstruction from low-resolution features.
• Restrict all major network computations to the low-resolution space to drastically reduce computational cost and memory usage.
• Integrate a lightweight, input-adaptive amplification mechanism that estimates the optimal image gain factor directly from the input, eliminating reliance on ground truth during inference.
• Use a multi-scale feature fusion strategy that leverages the large receptive field enabled by Pack operations to enhance detail recovery.
• Replace standard upsampling layers (e.g., transposed convolution) with UnPack to reduce parameter count and accelerate inference.

Experimental results

Research questions

• RQ1 Can a deep neural network achieve high-quality dark image restoration while operating efficiently on CPU and edge devices with limited memory and processing power?
• RQ2 How can large-scale downsampling be performed without significant information loss, enabling efficient computation in low-resolution feature spaces?
• RQ3 What is the most effective and efficient way to upsample features back to high resolution while minimizing artifacts and computational cost?
• RQ4 Can the amplification factor for low-light images be estimated automatically from the input image without requiring ground truth during inference?
• RQ5 How does the proposed Pack/UnPack mechanism compare to conventional downsampling and upsampling operations in terms of speed, memory, and reconstruction quality?

Key findings

• LLPackNet enhances a 2848×4256 extremely dark image in approximately 3 seconds on a CPU, significantly faster than existing methods.
• The method reduces model parameters by 2–7× and memory utilization by 2–3× compared to state-of-the-art approaches.
• Inference speed is improved by 5–20× compared to SOTA networks, with the UnPack operation being 3–4× faster than transposed convolution.
• The proposed Pack/UnPack operations achieve better feature preservation and reconstruction quality than max-pooling, strided convolution, and interpolation for large downsampling and upsampling.
• The automatic amplification mechanism enables robust performance on unknown scenes without requiring ground truth for gain estimation.
• Qualitative results show that LLPackNet-4× achieves better color fidelity and reduced blurriness than LLPackNet-8×, demonstrating the benefit of adaptive downsampling factors.