[Paper Review] Skin cancer diagnosis using NIR spectroscopy data of skin lesions in vivo using machine learning algorithms

Flavio P. Loss, Pedro Henrique da Cunha+13 more|arXiv (Cornell University)|Jan 2, 2024

Spectroscopy Techniques in Biomedical and Chemical ResearchBiochemistry, Genetics and Molecular Biology3 citations

TL;DR

This study introduces the NIR-SC-UFES dataset, a novel in vivo near-infrared spectroscopy dataset of skin lesions collected from patients, and evaluates machine learning models—particularly LightGBM with SNV pre-processing and GAN-based data augmentation—for classifying malignant (melanoma, BCC, SCC) and benign (nevus, ACK, SEK) lesions. The best model achieved 83.9% balanced accuracy, 85.1% recall, 85.2% precision, and 85.0% F1-score, marking a foundational step toward automated, spectroscopy-based skin cancer triage.

ABSTRACT

Skin lesions are classified in benign or malignant. Among the malignant, melanoma is a very aggressive cancer and the major cause of deaths. So, early diagnosis of skin cancer is very desired. In the last few years, there is a growing interest in computer aided diagnostic (CAD) using most image and clinical data of the lesion. These sources of information present limitations due to their inability to provide information of the molecular structure of the lesion. NIR spectroscopy may provide an alternative source of information to automated CAD of skin lesions. The most commonly used techniques and classification algorithms used in spectroscopy are Principal Component Analysis (PCA), Partial Least Squares - Discriminant Analysis (PLS-DA), and Support Vector Machines (SVM). Nonetheless, there is a growing interest in applying the modern techniques of machine and deep learning (MDL) to spectroscopy. One of the main limitations to apply MDL to spectroscopy is the lack of public datasets. Since there is no public dataset of NIR spectral data to skin lesions, as far as we know, an effort has been made and a new dataset named NIR-SC-UFES, has been collected, annotated and analyzed generating the gold-standard for classification of NIR spectral data to skin cancer. Next, the machine learning algorithms XGBoost, CatBoost, LightGBM, 1D-convolutional neural network (1D-CNN) were investigated to classify cancer and non-cancer skin lesions. Experimental results indicate the best performance obtained by LightGBM with pre-processing using standard normal variate (SNV), feature extraction providing values of 0.839 for balanced accuracy, 0.851 for recall, 0.852 for precision, and 0.850 for F-score. The obtained results indicate the first steps in CAD of skin lesions aiming the automated triage of patients with skin lesions in vivo using NIR spectral data.

Motivation & Objective

• To address the lack of publicly available in vivo NIR spectroscopy datasets for skin lesions.
• To evaluate modern machine learning and deep learning algorithms on spectral data for automated skin cancer diagnosis.
• To investigate the impact of pre-processing, feature extraction, and data augmentation (SMOTE and GAN) on classification performance.
• To identify key spectral regions critical for distinguishing cancerous from non-cancerous lesions using SHAP explainability.
• To establish a foundation for future clinical triage tools using non-invasive, molecular-level spectroscopic data.

Proposed method

• Collected in vivo NIR spectral data (900–1700 nm) from 168 patients using a portable Micronir spectrometer.
• Constructed the NIR-SC-UFES dataset with 6 lesion types: melanoma (MEL), basal cell carcinoma (BCC), squamous cell carcinoma (SCC), nevus (NEV), actinic keratosis (ACK), and seborrheic keratosis (SEK).
• Applied standard normal variate (SNV) pre-processing to reduce scatter effects in spectral data.
• Performed feature extraction using statistical descriptors and applied SMOTE and GAN-based oversampling to address class imbalance.
• Trained and compared XGBoost, CatBoost, LightGBM, 1D-CNN, SVM, and PLS-DA on pre-processed and augmented data.
• Used SHAP analysis to interpret model predictions and identify the most discriminative spectral regions (939.072–994.821 nm).

Figure 2: Acquisition of the NIR spectral data of a patient lesion using the Micronir portable spectrometer.

Experimental results

Research questions

• RQ1 Can machine learning models effectively classify skin lesions using in vivo NIR spectroscopy data despite limited sample sizes?
• RQ2 How do different pre-processing and data augmentation techniques (SNV, GAN, SMOTE) affect classification performance on spectral data?
• RQ3 Which machine learning algorithm—traditional (SVM, PLS-DA) or modern (GBM, 1D-CNN)—performs best on this novel spectral dataset?
• RQ4 Which spectral regions are most informative for distinguishing malignant from benign skin lesions?
• RQ5 To what extent can explainability methods like SHAP identify biologically relevant spectral features in the classification process?

Key findings

• LightGBM with SNV pre-processing and GAN-based data augmentation achieved the highest performance, with 83.9% balanced accuracy, 85.1% recall, 85.2% precision, and 85.0% F1-score.
• The most discriminative spectral region for lesion classification lies between 939.072 nm and 994.821 nm, as identified by SHAP analysis.
• Among gradient boosting models, LightGBM slightly outperformed CatBoost, while XGBoost showed comparable but less consistent results.
• Data augmentation using GANs significantly improved model generalization, especially for underrepresented lesion types like melanoma.
• 1D-CNN showed strong performance in reducing false positives and negatives when combined with SNV pre-processing, though it required more data to fully converge.
• The NIR-SC-UFES dataset is the first publicly available in vivo NIR spectral dataset for skin lesions, providing a critical benchmark for future research.

Figure 3: Spectral data sample of the six kind of skin lesions contained in the NIR-SC-UFES dataset.