[Paper Review] The effect of data augmentation and 3D-CNN depth on Alzheimer's Disease detection

Rosanna Turrisi, Alessandro Verri+1 more|arXiv (Cornell University)|Sep 13, 2023

Medical Imaging and AnalysisEngineering3 citations

TL;DR

This study investigates the impact of data augmentation and 3D-CNN depth on Alzheimer’s Disease (AD) detection using MRI scans from the ADNI dataset. By employing k-fold cross-validation, multiple training trials, and rigorous experimental design, the authors demonstrate that affine transformations applied separately (Strategy B) and a model with 8 convolutional layers yield the highest accuracy (81.95% on test set), with performance varying by up to 10% based on augmentation and depth choices.

ABSTRACT

Machine Learning (ML) has emerged as a promising approach in healthcare, outperforming traditional statistical techniques. However, to establish ML as a reliable tool in clinical practice, adherence to best practices regarding data handling, experimental design, and model evaluation is crucial. This work summarizes and strictly observes such practices to ensure reproducible and reliable ML. Specifically, we focus on Alzheimer's Disease (AD) detection, which serves as a paradigmatic example of challenging problem in healthcare. We investigate the impact of different data augmentation techniques and model complexity on the overall performance. We consider MRI data from ADNI dataset to address a classification problem employing 3D Convolutional Neural Network (CNN). The experiments are designed to compensate for data scarcity and initial random parameters by utilizing cross-validation and multiple training trials. Within this framework, we train 15 predictive models, considering three different data augmentation strategies and five distinct 3D CNN architectures, each varying in the number of convolutional layers. Specifically, the augmentation strategies are based on affine transformations, such as zoom, shift, and rotation, applied concurrently or separately. The combined effect of data augmentation and model complexity leads to a variation in prediction performance up to 10% of accuracy. When affine transformation are applied separately, the model is more accurate, independently from the adopted architecture. For all strategies, the model accuracy followed a concave behavior at increasing number of convolutional layers, peaking at an intermediate value of layers. The best model (8 CL, (B)) is the most stable across cross-validation folds and training trials, reaching excellent performance both on the testing set and on an external test set.

Motivation & Objective

• To evaluate the impact of different data augmentation strategies on 3D-CNN performance in Alzheimer’s Disease detection.
• To investigate how model depth (number of convolutional layers) affects classification accuracy and generalization.
• To ensure reproducibility and robustness by applying rigorous experimental practices, including cross-validation and multiple training trials.
• To test model generalization on external data, particularly under domain shift (e.g., 3T vs. 1.5T MRI scans).
• To provide a publicly available, reproducible pipeline for benchmarking in AD detection using deep learning.

Proposed method

• Utilized 1.5T T1-weighted MRI scans from the ADNI dataset for binary classification of Alzheimer’s Disease (AD) vs. Cognitively Normal (CN) subjects.
• Employed 3D-Convolutional Neural Networks (3D-CNNs) with varying depths (4 to 12 layers) to assess the effect of model complexity.
• Applied three data augmentation strategies: (A) no augmentation, (B) affine transformations (zoom, shift, rotation) applied separately, and (C) all transformations applied simultaneously.
• Conducted 15 distinct models by combining five network depths with three augmentation strategies, using k-fold cross-validation and multiple random weight initializations.
• Evaluated model performance using accuracy, standard deviation across folds and trials, and tested generalization on an external 3T MRI test set.
• Ensured reproducibility by versioning code, disclosing data splits, and publicly releasing the list of selected samples.

Figure 1: MRI collection in ADNI dataset. Schema representing ADNI phases (ADNI1, ADNI GO, ADNI2, ADNI3). Different phases include a variable proportion of subjects: circles represent CN subjects, triangles represent Mild Cognitive Impaired (MCI), early MCI (EMCI) or late MCI (LMCI) subjects and squ

Experimental results

Research questions

• RQ1 How does the choice of data augmentation strategy (separate vs. combined affine transformations) affect 3D-CNN performance in AD detection?
• RQ2 What is the optimal depth (number of convolutional layers) for 3D-CNNs in classifying AD vs. CN using limited MRI data?
• RQ3 How does model depth influence performance, and does a concave relationship exist between depth and accuracy?
• RQ4 Can the best-performing model generalize to external data with different acquisition parameters (e.g., 3T vs. 1.5T MRI)?
• RQ5 To what extent do rigorous experimental practices (cross-validation, multiple trials) improve model stability and reliability in clinical ML applications?

Key findings

• The combination of 8 convolutional layers and separate affine transformations (Strategy B) achieved the highest test accuracy of 81.95% ± 1.26%, outperforming other configurations.
• Performance varied by up to 10% in accuracy depending on the combination of data augmentation and model depth, highlighting the critical role of these design choices.
• Model accuracy followed a concave trend with increasing depth, peaking at 8 layers, indicating that deeper networks do not always generalize better with limited data.
• Strategy B (separate transformations) consistently outperformed Strategy C (simultaneous transformations), suggesting that applying transformations individually may be more effective.
• The 8 CL, (B) model showed the highest stability across cross-validation folds and training trials, with a validation accuracy of 87.21% ± 0.88%.
• On an external 3T MRI test set, the best model achieved 71% accuracy, demonstrating its ability to generalize under domain shift despite resolution differences.

Figure 2: 2D visualization of 3D MRI scans. Axial, coronal and sagittal planes of two brain images from ADNI dataset.