[Paper Review] Models Genesis: Generic Autodidactic Models for 3D Medical Image Analysis

Zongwei Zhou, Vatsal Sodha+5 more|arXiv (Cornell University)|Aug 19, 2019

AI in cancer detection15 references44 citations

TL;DR

The paper introduces Generic Autodidactic Models (Models Genesis) trained self-supervised on unlabeled 3D medical images, yielding strong transfer performance across multiple 3D and 2D tasks and outperforming learning-from-scratch and 2D ImageNet baselines. The models are open-sourced for community use.

ABSTRACT

Transfer learning from natural image to medical image has established as one of the most practical paradigms in deep learning for medical image analysis. However, to fit this paradigm, 3D imaging tasks in the most prominent imaging modalities (e.g., CT and MRI) have to be reformulated and solved in 2D, losing rich 3D anatomical information and inevitably compromising the performance. To overcome this limitation, we have built a set of models, called Generic Autodidactic Models, nicknamed Models Genesis, because they are created ex nihilo (with no manual labeling), self-taught (learned by self-supervision), and generic (served as source models for generating application-specific target models). Our extensive experiments demonstrate that our Models Genesis significantly outperform learning from scratch in all five target 3D applications covering both segmentation and classification. More importantly, learning a model from scratch simply in 3D may not necessarily yield performance better than transfer learning from ImageNet in 2D, but our Models Genesis consistently top any 2D approaches including fine-tuning the models pre-trained from ImageNet as well as fine-tuning the 2D versions of our Models Genesis, confirming the importance of 3D anatomical information and significance of our Models Genesis for 3D medical imaging. This performance is attributed to our unified self-supervised learning framework, built on a simple yet powerful observation: the sophisticated yet recurrent anatomy in medical images can serve as strong supervision signals for deep models to learn common anatomical representation automatically via self-supervision. As open science, all pre-trained Models Genesis are available at https://github.com/MrGiovanni/ModelsGenesis.

Motivation & Objective

• Demonstrate that source models built from unlabeled medical images can yield high-performance target models via transfer learning.
• Show that 3D Models Genesis leverage 3D anatomical information better than 2D transfers.
• Evaluate Genesis across multiple organs, diseases, and modalities to establish generality.
• Compare with learning from scratch and ImageNet-based transfer to quantify benefits of 3D self-supervision.

Proposed method

• Propose a unified self-supervised framework with four transformations (non-linear intensity, local pixel shuffling, out-painting, in-painting) integrated into a single image restoration task.
• Train an encoder–decoder network to restore original patches from transformed patches, enabling transferable feature learning.
• Pre-train Genesis Chest CT on unlabeled chest CT data (LIDC-IDRI) and Genesis Chest X-ray on ChestX-ray83; produce a 2D version (Genesis Chest CT 2D) for slice-based tasks.
• Fine-tune the pretrained encoder or encoder–decoder on downstream tasks to assess segmentation and classification performance.
• Evaluate across seven target tasks spanning 3D and 2D classification and segmentation (e.g., NCC, NCS, ECC, LCS, BMS, DXC, IUC).
• Compare with training from scratch, ImageNet finetuning (2D and 3D), and NiftyNet baselines; analyze same-domain vs cross-domain transfer.

Experimental results

Research questions

• RQ1 Can self-supervised pre-training on unlabeled medical images yield transferable representations for diverse 3D medical imaging tasks?
• RQ2 Does leveraging 3D anatomical information in Genesis provide an advantage over 2D transfer learning from ImageNet?
• RQ3 Is there a benefit to a unified multi-task self-supervised framework over individual proxy tasks for medical imaging representations?
• RQ4 How do same-domain and cross-domain transfers compare when using Genesis across diseases, organs, and modalities?

Key findings

• Genesis models consistently outperform training from scratch across seven target tasks.
• Genesis Chest CT (3D) generally outperforms 2D ImageNet-finetuned models on the same tasks, highlighting the value of 3D anatomy.
• Downgraded 2D versions (Genesis Chest CT 2D and Genesis X-ray) achieve equivalent performance to ImageNet finetuning within the same modality.
• Across tasks, same-domain transfer learning is preferred, with cross-domain transfer showing potential but generally weaker performance than same-domain transfer.
• The unified self-supervised framework yields robust improvements across multiple tasks and outperforms individual proxy schemes.
• All pre-trained Models Genesis are released publicly to support open science.