[Paper Review] Video Pretraining Advances 3D Deep Learning on Chest CT Tasks

Alexander Ke, Shih-Cheng Huang+4 more|arXiv (Cornell University)|Apr 2, 2023

Radiomics and Machine Learning in Medical ImagingMedicine3 citations

TL;DR

This paper demonstrates that video pretraining on large-scale out-of-domain datasets like Kinetics significantly improves 3D deep learning performance on chest CT tasks, outperforming both 2D models and in-domain pretraining. Across seven 3D models and two clinical tasks (pulmonary embolism and nodule detection), video pretraining consistently boosted performance, especially on small datasets, establishing large-scale video pretraining as a superior strategy for data-scarce 3D medical imaging.

ABSTRACT

Pretraining on large natural image classification datasets such as ImageNet has aided model development on data-scarce 2D medical tasks. 3D medical tasks often have much less data than 2D medical tasks, prompting practitioners to rely on pretrained 2D models to featurize slices. However, these 2D models have been surpassed by 3D models on 3D computer vision benchmarks since they do not natively leverage cross-sectional or temporal information. In this study, we explore whether natural video pretraining for 3D models can enable higher performance on smaller datasets for 3D medical tasks. We demonstrate video pretraining improves the average performance of seven 3D models on two chest CT datasets, regardless of finetuning dataset size, and that video pretraining allows 3D models to outperform 2D baselines. Lastly, we observe that pretraining on the large-scale out-of-domain Kinetics dataset improves performance more than pretraining on a typically-sized in-domain CT dataset. Our results show consistent benefits of video pretraining across a wide array of architectures, tasks, and training dataset sizes, supporting a shift from small-scale in-domain pretraining to large-scale out-of-domain pretraining for 3D medical tasks. Our code is available at: https://github.com/rajpurkarlab/chest-ct-pretraining

Motivation & Objective

• To evaluate whether large-scale video pretraining enhances 3D deep learning models for chest CT tasks, especially in low-data regimes.
• To compare the effectiveness of out-of-domain video pretraining (e.g., Kinetics) against in-domain CT pretraining and sequential pretraining.
• To assess whether video pretraining generalizes across diverse 3D architectures and clinical tasks in medical imaging.
• To understand how pretraining performance scales with downstream dataset size, particularly in small-data settings common in medicine.
• To provide a comprehensive benchmark of pretraining strategies across models, tasks, and data regimes to guide future 3D medical AI development.

Proposed method

• Pretrained seven 3D models (e.g., MViT, Swin-T, SlowFast) and three 2D models on ImageNet and Kinetics datasets for video pretraining.
• Conducted sequential pretraining by first pretraining on the in-domain Stanford PE detection dataset, then on Kinetics, to isolate video pretraining effects.
• Fine-tuned all models on two public chest CT datasets: RSNA for pulmonary embolism detection and LIDC-IDRI for lung nodule detection.
• Evaluated performance across three training dataset sizes (1%, 10%, 100%) to assess data efficiency and scalability of pretraining strategies.
• Used standard metrics (AUC) with 95% confidence intervals to compare model performance across pretraining regimes and architectures.
• Applied hyperparameter search across models and pretraining protocols to ensure fair and generalizable comparisons.

Figure 1: Visual summary of our methods. We examined seven 3D (three 2D) models that were pre-trained either on Kinetics (ImageNet), or on the Stanford dataset for PE detection, or on both sequentially. We then transferred these weights and finetuned them on RSNA for PE detection and LIDC for lung n

Experimental results

Research questions

• RQ1 Does video pretraining on large-scale out-of-domain videos (e.g., Kinetics) improve 3D medical image classification performance compared to ImageNet pretraining or no pretraining?
• RQ2 How does video pretraining compare to in-domain CT pretraining in terms of performance gain, especially on small downstream datasets?
• RQ3 Does the benefit of video pretraining scale with model architecture diversity and task variety in 3D chest CT analysis?
• RQ4 How does the performance of video-pretrained models compare to 2D baselines across different downstream dataset sizes?
• RQ5 What is the interaction between pretraining strategy and downstream dataset size in low-data medical imaging settings?

Key findings

• Video pretraining on Kinetics improved the average performance of seven 3D models across both RSNA and LIDC datasets, regardless of downstream dataset size.
• On the RSNA dataset with 1% of training data (50 studies), video pretraining achieved an AUC of 0.599, significantly outperforming the 2D baseline (AUC 0.517) and in-domain pretraining (AUC 0.510).
• For the LIDC dataset at 10% training size (71 studies), video pretraining achieved an AUC of 0.831, surpassing the 2D baseline (AUC 0.530) and in-domain pretraining (AUC 0.500).
• Video pretraining enabled 3D models to outperform 2D models across all dataset sizes, with the largest relative gains observed in low-data regimes (1% and 10% of training data).
• Pretraining on Kinetics yielded higher performance than pretraining on a typically-sized in-domain CT dataset, indicating that out-of-domain video pretraining is more effective for 3D medical tasks.
• Sequential pretraining (in-domain then Kinetics) did not outperform Kinetics-only pretraining, suggesting that the benefits of video pretraining are not additive and that Kinetics pretraining alone is sufficient and superior.

Figure 2: Video pretraining consistently improves 3D models’ AUC on 10% and 100% of RSNA and LIDC, except MViT on 10% of LIDC ( $p=0.254$ )