[Paper Review] Nervus: A Comprehensive Deep Learning Classification, Regression, and Prognostication Tool for both Medical Image and Clinical Data Analysis

Toshimasa Matsumoto, Shannon L. Walston+2 more|arXiv (Cornell University)|Dec 12, 2022

AI in cancer detection4 citations

TL;DR

Nervus is a PyTorch-based deep learning framework designed for medical imaging and clinical data analysis, integrating grayscale images and tabular data into a unified pipeline for classification, regression, and prognostication tasks. It employs a four-part architecture—dataloader, feature extractor, feature mixer, and classifier—enabling flexible, multi-input, multi-label learning with high usability for radiology and clinical research.

ABSTRACT

The goal of our research is to create a comprehensive and flexible library that is easy to use for medical imaging research, and capable of handling grayscale images, multiple inputs (both images and tabular data), and multi-label tasks. We have named it Nervus. Based on the PyTorch library, which is suitable for AI for research purposes, we created a four-part model to handle comprehensive inputs and outputs. Nervus consists of four parts. First is the dataloader, then the feature extractor, the feature mixer, and finally the classifier. The dataloader preprocesses the input data, the feature extractor extracts the features between the training data and ground truth labels, feature mixer mixes the features of the extractors, and the classifier classifies the input data from feature mixer based on the task. We have created Nervus, which is a comprehensive and flexible model library that is easy to use for medical imaging research which can handle grayscale images, multi-inputs and multi-label tasks. This will be helpful for researchers in the field of radiology.

Motivation & Objective

• To develop a flexible, user-friendly deep learning library tailored for medical imaging and clinical data research.
• To address the challenge of integrating heterogeneous data modalities—grayscale medical images and tabular clinical data—within a single deep learning framework.
• To support diverse machine learning tasks, including multi-label classification, regression, and prognostic modeling, in a unified pipeline.
• To improve accessibility and reproducibility in radiology and clinical AI research by providing a modular, extensible architecture.
• To enable end-to-end training and inference with minimal code complexity for researchers without deep deep learning expertise.

Proposed method

• The framework uses a four-stage architecture: dataloader for input preprocessing, feature extractor for learning representations from images and tabular data, feature mixer for fusing multi-modal features, and classifier for task-specific output.
• The dataloader handles data loading and preprocessing, including normalization and augmentation for medical images and standardization for tabular data.
• The feature extractor employs convolutional neural networks (CNNs) for image data and fully connected networks for tabular data, enabling joint feature learning.
• The feature mixer combines modality-specific features using concatenation or attention-based mechanisms to model cross-modality interactions.
• The classifier head is task-adaptive, using softmax for multi-class classification, sigmoid for multi-label tasks, and regression heads for continuous outcome prediction.
• The entire pipeline is built on PyTorch, ensuring compatibility with research workflows and enabling easy customization and deployment.

Experimental results

Research questions

• RQ1 Can a unified deep learning framework effectively handle both medical images and clinical tabular data in a single training pipeline?
• RQ2 How does the integration of multi-modal inputs (images and tabular data) improve performance in medical classification and prognostic tasks?
• RQ3 To what extent does the modular design of Nervus support diverse machine learning tasks, including multi-label and regression problems?
• RQ4 How does Nervus compare in usability and performance to existing frameworks in radiology and clinical AI research?
• RQ5 Can the feature mixer component effectively capture cross-modality dependencies between imaging and clinical features?

Key findings

• Nervus successfully integrates grayscale medical images and clinical tabular data into a single, end-to-end deep learning pipeline.
• The framework supports multiple learning tasks—classification, regression, and multi-label prediction—within the same architecture.
• The modular design enables researchers to easily swap components such as feature extractors or classifiers for specific use cases.
• The use of PyTorch ensures high compatibility with existing deep learning research tools and workflows.
• The framework demonstrates strong usability for medical imaging researchers, reducing the need for custom implementation of data pipelines and model architectures.
• Nervus enables joint learning from multi-modal data, potentially improving model performance in prognostic and diagnostic tasks over single-modality models.