[Paper Review] Hyper-Skin: A Hyperspectral Dataset for Reconstructing Facial Skin-Spectra from RGB Images

Pai Chet Ng, Zhixiang Chi+3 more|arXiv (Cornell University)|Oct 27, 2023

Optical Imaging and Spectroscopy Techniques4 citations

TL;DR

Hyper-Skin introduces a novel hyperspectral dataset covering visible (400–700 nm) and near-infrared (700–1000 nm) wavelengths, with 330 hyperspectral cubes from 51 subjects, enabling reconstruction of facial skin spectra from RGB images. The dataset, paired with synthetic RGB images from 28 real camera responses, supports state-of-the-art spectral reconstruction models, achieving significant performance gains in both VIS and NIR bands, especially when background is removed.

ABSTRACT

We introduce Hyper-Skin, a hyperspectral dataset covering wide range of wavelengths from visible (VIS) spectrum (400nm - 700nm) to near-infrared (NIR) spectrum (700nm - 1000nm), uniquely designed to facilitate research on facial skin-spectra reconstruction. By reconstructing skin spectra from RGB images, our dataset enables the study of hyperspectral skin analysis, such as melanin and hemoglobin concentrations, directly on the consumer device. Overcoming limitations of existing datasets, Hyper-Skin consists of diverse facial skin data collected with a pushbroom hyperspectral camera. With 330 hyperspectral cubes from 51 subjects, the dataset covers the facial skin from different angles and facial poses. Each hyperspectral cube has dimensions of 1024$ imes$1024$ imes$448, resulting in millions of spectra vectors per image. The dataset, carefully curated in adherence to ethical guidelines, includes paired hyperspectral images and synthetic RGB images generated using real camera responses. We demonstrate the efficacy of our dataset by showcasing skin spectra reconstruction using state-of-the-art models on 31 bands of hyperspectral data resampled in the VIS and NIR spectrum. This Hyper-Skin dataset would be a valuable resource to NeurIPS community, encouraging the development of novel algorithms for skin spectral reconstruction while fostering interdisciplinary collaboration in hyperspectral skin analysis related to cosmetology and skin's well-being. Instructions to request the data and the related benchmarking codes are publicly available at: \url{https://github.com/hyperspectral-skin/Hyper-Skin-2023}.

Motivation & Objective

• To address the lack of publicly available, diverse, and ethically collected hyperspectral facial skin datasets covering both visible and near-infrared spectra.
• To enable the development of deep learning models that reconstruct high-fidelity skin spectral reflectance from consumer-grade RGB images.
• To support low-cost, accessible hyperspectral skin analysis for applications in cosmetology, dermatology, and personal skin well-being.
• To provide a benchmark dataset with paired hyperspectral and synthetic RGB data for training and evaluating spectral reconstruction algorithms.

Proposed method

• Acquired facial hyperspectral data using a pushbroom hyperspectral camera (Specim FX10) with 448 spectral bands spanning 400–1000 nm.
• Generated synthetic RGB images using 28 real camera response functions to simulate consumer smartphone imaging conditions.
• Collected data from 51 subjects across diverse facial poses and angles, ensuring spatial and spectral diversity in the 1024×1024×448 hyperspectral cubes.
• Applied rigorous ethical protocols, including informed consent, anonymization via subject IDs, and institutional ethics board approval.
• Trained state-of-the-art deep learning models on the dataset to reconstruct hyperspectral data from RGB inputs, using spectral angle mapper (SAM) as a key evaluation metric.
• Performed ablation studies to assess the impact of background presence on reconstruction performance, demonstrating improved results when background is removed.

Figure 1: A glimpse of our Hyper-Skin dataset, covering the skin spectra in the visible spectrum (400nm - 700nm) and near-infrared spectrum (700nm - 1000nm).

Experimental results

Research questions

• RQ1 Can a hyperspectral dataset covering both visible and near-infrared spectra enable accurate reconstruction of facial skin reflectance from consumer-grade RGB images?
• RQ2 How does the inclusion of near-infrared wavelengths improve the fidelity and physiological relevance of reconstructed skin spectra?
• RQ3 To what extent does background interference in RGB images degrade the performance of spectral reconstruction models?
• RQ4 How do different camera response functions affect the realism and generalization of synthetic RGB data for training reconstruction models?
• RQ5 Can models trained on Hyper-Skin generalize to real smartphone RGB images, enabling practical deployment on consumer devices?

Key findings

• Reconstruction performance significantly improved after retraining models on the Hyper-Skin dataset, with lower spectral angle mapper (SAM) values indicating better spectral fidelity.
• Models achieved better performance when background was removed from input RGB images, highlighting the negative impact of background interference on reconstruction accuracy.
• The inclusion of near-infrared (NIR) spectrum (700–1000 nm) enabled more accurate estimation of physiological parameters such as melanin and hemoglobin concentrations.
• The synthetic RGB images, generated using real camera response functions, enabled effective training of models that generalize to real smartphone captures.
• Demonstrated successful reconstruction of skin spectral reflectance from a real smartphone selfie, validating the practical applicability of the trained models.
• The dataset enables state-of-the-art performance in reconstructing 31 bands of hyperspectral data across VIS and NIR, with notable improvements in spectral consistency and spatial detail.

Figure 2: The image on the right illustrates our experimental setup for data collection, while the accompanying schematic representation provides a visual depiction of the setup.