[Paper Review] Inflation of test accuracy due to data leakage in deep learning-based classification of OCT images

Iulian Emil Tampu, Anders Eklund+1 more|arXiv (Cornell University)|Feb 21, 2022

Retinal Imaging and AnalysisMedicine42 references66 citations

TL;DR

This study demonstrates that improper data splitting in deep learning for OCT image classification—specifically, splitting at the 2D slice level instead of the 3D volume or subject level—leads to significant data leakage, inflating test accuracy by up to 30% (MCC increase of 0.07–0.43). The authors show that models trained on overlapping training and test sets overestimate generalization performance, highlighting the critical need for volume- or subject-level splitting to ensure reliable evaluation in OCT-based deep learning research.

ABSTRACT

In the application of deep learning on optical coherence tomography (OCT) data, it is common to train classification networks using 2D images originating from volumetric data. Given the micrometer resolution of OCT systems, consecutive images are often very similar in both visible structures and noise. Thus, an inappropriate data split can result in overlap between the training and testing sets, with a large portion of the literature overlooking this aspect. In this study, the effect of improper dataset splitting on model evaluation is demonstrated for three classification tasks using three OCT open-access datasets extensively used, Kermany's and Srinivasan's ophthalmology datasets, and AIIMS breast tissue dataset. Results show that the classification performance is inflated by 0.07 up to 0.43 in terms of Matthews Correlation Coefficient (accuracy: 5% to 30%) for models tested on datasets with improper splitting, highlighting the considerable effect of dataset handling on model evaluation. This study intends to raise awareness on the importance of dataset splitting given the increased research interest in implementing deep learning on OCT data.

Motivation & Objective

• To investigate the impact of improper data splitting strategies on model evaluation in deep learning-based OCT image classification.
• To demonstrate that per-image splitting—common in the literature—leads to data leakage due to high similarity between consecutive 2D OCT slices.
• To quantify the inflation of test accuracy and MCC caused by overlapping training and testing sets in OCT datasets.
• To advocate for volume- or subject-level data splitting as a necessary standard to ensure unbiased model evaluation in OCT research.

Proposed method

• Implemented a custom data splitting function to create training and testing sets using both per-image and per-volume/subject strategies.
• Used the LightOCT model, a shallow two-convolutional-layer CNN with ReLU and max-pooling, followed by a fully connected layer with softmax output.
• Trained models from scratch using stochastic gradient descent with momentum (m=0.9), a constant learning rate (0.0001), batch size 64, and 250 epochs without early stopping.
• Applied ten-times repeated five-fold cross-validation to ensure result reliability across splits.
• Calculated evaluation metrics including Matthews Correlation Coefficient (MCC), accuracy, precision, recall, F1-score, and AUC for each class.
• Conducted a random label experiment with 10,000 simulations to detect bias: compared MCC from models trained on random labels to a null distribution built without overlap, using one-sample Wilcoxon tests to compute p-values.

Experimental results

Research questions

• RQ1 To what extent does per-image data splitting inflate test accuracy in deep learning models for OCT image classification?
• RQ2 How does data leakage due to overlapping training and testing sets affect the Matthews Correlation Coefficient (MCC) in OCT-based classification tasks?
• RQ3 What is the magnitude of performance inflation when using overlapping data splits compared to proper volume-level splits?
• RQ4 Can random label experiments detect data leakage in model evaluation, and how do p-values from such tests reflect the presence of overlap?

Key findings

• Data leakage due to per-image splitting inflated test accuracy by 5% to 30% across three OCT datasets, with MCC increasing by 0.07 to 0.43.
• In the Kermany’s OCT2017 version 2 dataset, 92% of test images belonged to subject-IDs also present in the training set, indicating substantial overlap.
• The original_v3 split of Kermany’s dataset showed no subject-ID overlap, but the original_v2 split had significant leakage, explaining inflated performance in prior studies.
• Random label experiments detected statistically significant bias (p < 0.05) in models trained on overlapping splits, confirming data leakage effects.
• The study found that models using the original_v2 split reported 99.6% accuracy, which is likely inflated due to data leakage, while proper per-volume splitting would yield more reliable estimates.
• The authors conclude that improper splitting undermines model generalization assessment and that volume- or subject-level splitting is essential for valid evaluation in OCT deep learning research.