[Paper Review] LLVIP: A Visible-infrared Paired Dataset for Low-light Vision

Xinyu Jia, Chuang Zhu+4 more|arXiv (Cornell University)|Aug 24, 2021

Advanced Image Fusion Techniques20 references21 citations

TL;DR

This paper introduces LLVIP, the first large-scale, time- and space-aligned visible-infrared paired dataset specifically for low-light vision tasks. It enables advanced research in image fusion, pedestrian detection, and image-to-image translation under extreme low-light conditions, with experiments revealing significant performance gaps in existing algorithms across all three tasks, highlighting the dataset's challenge and utility for future development.

ABSTRACT

It is very challenging for various visual tasks such as image fusion, pedestrian detection and image-to-image translation in low light conditions due to the loss of effective target areas. In this case, infrared and visible images can be used together to provide both rich detail information and effective target areas. In this paper, we present LLVIP, a visible-infrared paired dataset for low-light vision. This dataset contains 30976 images, or 15488 pairs, most of which were taken at very dark scenes, and all of the images are strictly aligned in time and space. Pedestrians in the dataset are labeled. We compare the dataset with other visible-infrared datasets and evaluate the performance of some popular visual algorithms including image fusion, pedestrian detection and image-to-image translation on the dataset. The experimental results demonstrate the complementary effect of fusion on image information, and find the deficiency of existing algorithms of the three visual tasks in very low-light conditions. We believe the LLVIP dataset will contribute to the community of computer vision by promoting image fusion, pedestrian detection and image-to-image translation in very low-light applications. The dataset is being released in https://bupt-ai-cz.github.io/LLVIP. Raw data is also provided for further research such as image registration.

Motivation & Objective

• To address the lack of large-scale, aligned visible-infrared datasets tailored for low-light vision tasks such as image fusion, pedestrian detection, and image-to-image translation.
• To develop a reliable method for labeling pedestrians in extremely low-light visible images by leveraging aligned infrared images for reverse mapping.
• To evaluate the performance of state-of-the-art algorithms in image fusion, pedestrian detection, and image-to-image translation on a dataset with challenging low-light conditions.
• To provide a benchmark dataset that exposes the limitations of current models and promotes research in low-light computer vision.

Proposed method

• Collecting 30,976 images (15,488 aligned visible-infrared pairs) using a binocular camera system with synchronized visible and infrared sensors.
• Performing precise spatial and temporal registration to ensure strict alignment between visible and infrared images across all pairs.
• Proposing a reverse mapping labeling method that uses aligned infrared images to accurately annotate pedestrians in low-light visible images.
• Training and evaluating multiple deep learning models, including YOLOv3 and YOLOv5 for pedestrian detection, and pix2pixGAN for image-to-image translation.
• Conducting comprehensive quantitative and qualitative evaluations using metrics such as AP, AP50, AP75, SSIM, and PSNR across all tasks.
• Releasing the full dataset, including raw images and annotations, to support future research in image fusion, registration, and domain adaptation.

Experimental results

Research questions

• RQ1 How effective are current image fusion methods on low-light visible-infrared image pairs, and what are their limitations in preserving detail and target information?
• RQ2 To what extent do existing pedestrian detection models perform on low-light visible images when trained on aligned infrared annotations?
• RQ3 Why do image-to-image translation models like pix2pixGAN fail on low-light data, and how does performance degrade compared to standard datasets?
• RQ4 Can aligned infrared images be effectively used to label pedestrians in extremely low-light visible images where human annotation is infeasible?
• RQ5 How does the performance of state-of-the-art models vary across visible, infrared, and fused representations in low-light conditions?

Key findings

• Image fusion algorithms struggle to preserve fine details in low-light visible images, indicating a significant gap in current fusion methods under extreme illumination conditions.
• Pedestrian detection on low-light visible images achieves only 52.7% average precision (AP) with YOLOv5, while infrared images yield 67.0% AP, highlighting the need for better low-light detection models.
• Image-to-image translation using pix2pixGAN on LLVIP yields a PSNR of only 10.77 and SSIM of 0.176, far below the performance on KAIST (PSNR 28.99, SSIM 0.69), demonstrating poor generalization to low-light scenarios.
• The miss rate-FPPI curve shows that YOLOv5 and YOLOv3 have high miss rates on visible images, especially at low false positive per image (FPPI), indicating poor detection reliability in darkness.
• The dataset reveals that existing algorithms for all three tasks—fusion, detection, and translation—perform suboptimally in very low-light conditions, underscoring the need for new methods.
• The reverse mapping labeling method successfully enables accurate pedestrian annotation on low-light visible images by leveraging infrared image consistency, enabling reliable evaluation.