[Paper Review] Dancing under the stars: video denoising in starlight

Kristina Monakhova, Stephan R. Richter+2 more|arXiv (Cornell University)|Apr 8, 2022

Image and Signal Denoising Methods4 citations

TL;DR

This paper presents the first photorealistic video denoising in starlight (below 0.001 lux) using a GAN-tuned physics-based noise model to simulate realistic low-light noise. By combining synthetic noisy video from the noise generator with real high-gain still images, the authors train a video denoiser that achieves superior temporal consistency and detail preservation in submillilux conditions, enabling clear video of moving subjects under the Milky Way.

ABSTRACT

Imaging in low light is extremely challenging due to low photon counts. Using sensitive CMOS cameras, it is currently possible to take videos at night under moonlight (0.05-0.3 lux illumination). In this paper, we demonstrate photorealistic video under starlight (no moon present, $

Motivation & Objective

• To enable photorealistic video denoising in extreme low-light conditions, specifically starlight levels below 0.001 lux.
• To address the challenge of non-Gaussian, sensor-specific noise in high-gain low-light imaging that undermines traditional denoising methods.
• To develop a data-efficient training pipeline that avoids the need for motion-aligned clean/noisy video pairs by leveraging still image bursts and synthetic video.
• To demonstrate the feasibility of capturing and denoising 5–10 fps video of dynamic scenes under moonless night conditions.

Proposed method

• A physics-inspired noise generator is trained using a limited set of long- and short-exposure still image pairs to synthesize realistic noise at the highest camera gain setting.
• The noise generator is trained via a GAN framework with a discriminator to distinguish real from synthetic noise, ensuring high-fidelity noise modeling.
• A video denoiser is trained using a hybrid dataset combining synthetic clean/noisy video clips (generated via the noise model) and real noisy still image bursts from the same camera.
• The denoiser is based on a modified FastDVDnet architecture, fine-tuned to handle the specific non-Gaussian noise characteristics of the target sensor.
• The training process decouples noise modeling from denoising, allowing the noise generator to be fixed while the denoiser is optimized on a combination of synthetic and real data.
• Temporal consistency is improved by leveraging video-specific architecture components and loss functions that minimize flicker across frames.

Figure 1: Method overview. (a) First we train our noise generator along with a discriminator, which aims to distinguish between real and synthetic noise. We use a limited dataset of long exposure/low gain and short exposure/high gain non-moving image pairs during this training process. After trainin

Experimental results

Research questions

• RQ1 Can photorealistic video denoising be achieved in starlight conditions (below 0.001 lux) where photon counts are extremely low?
• RQ2 How can a realistic noise model be learned for high-gain, low-light imaging when traditional noise assumptions (e.g., Gaussian) fail?
• RQ3 To what extent can a video denoiser trained on synthetic data and real still images generalize to real-world dynamic video sequences under submillilux illumination?
• RQ4 How does the proposed method compare in performance and temporal consistency to state-of-the-art video denoisers like V-BM4D and FastDVDnet on real low-light video?

Key findings

• The authors successfully captured 5–10 fps video at 0.6–0.7 millilux (0.0006–0.0007 lux) under moonless starlight, demonstrating the first video of moving subjects in such conditions.
• The proposed method achieved superior image quality and reduced flickering compared to V-BM4D and pretrained FastDVDnet, which exhibited significant temporal artifacts and streaking.
• In a blind A/B test with 10 workers and 300 comparisons, the proposed method was rated as having better image quality than alternatives in 95% of cases.
• The denoiser outperformed both classic (V-BM4D) and deep learning-based (FastDVDnet, L2SID) baselines, particularly in preserving fine details like stars and dynamic motion.
• Noise2Self failed due to structured noise patterns, highlighting the inadequacy of self-supervised methods when noise is highly correlated and non-i.i.d.
• The method maintains robustness to fast-moving objects, such as meteors, with no loss of detail or severe artifacts in the denoised output.

Figure 2: Physics-inspired noise generator. Our noise generator takes in a clean image and produces a synthetic noisy image. During training, our physics-inspired statistical noise parameters are optimized along with a U-Net to produce a synthetic noisy image that is indistinguishable from a real no