[Paper Review] A Transformer-based representation-learning model with unified processing of multimodal input for clinical diagnostics

Hong-Yu Zhou, Yizhou Yu+8 more|arXiv (Cornell University)|Jun 1, 2023

COVID-19 diagnosis using AI11 citations

TL;DR

A Transformer-based model unifies multimodal clinical data (images, text, labs, demographics) into a single representation to aid diagnosis and predict COVID-19 outcomes, outperforming image-only and non-unified baselines.

ABSTRACT

During the diagnostic process, clinicians leverage multimodal information, such as chief complaints, medical images, and laboratory-test results. Deep-learning models for aiding diagnosis have yet to meet this requirement. Here we report a Transformer-based representation-learning model as a clinical diagnostic aid that processes multimodal input in a unified manner. Rather than learning modality-specific features, the model uses embedding layers to convert images and unstructured and structured text into visual tokens and text tokens, and bidirectional blocks with intramodal and intermodal attention to learn a holistic representation of radiographs, the unstructured chief complaint and clinical history, structured clinical information such as laboratory-test results and patient demographic information. The unified model outperformed an image-only model and non-unified multimodal diagnosis models in the identification of pulmonary diseases (by 12% and 9%, respectively) and in the prediction of adverse clinical outcomes in patients with COVID-19 (by 29% and 7%, respectively). Leveraging unified multimodal Transformer-based models may help streamline triage of patients and facilitate the clinical decision process.

Motivation & Objective

• Motivate the need for models that integrate multimodal clinical information during diagnostics.
• Develop a Transformer-based representation-learning model that processes images, unstructured text, and structured data in a unified way.
• Demonstrate improved diagnostic performance over image-only and non-unified multimodal approaches.
• Show potential benefits for triage and clinical decision-making.
• Highlight applicability to radiographs, chief complaints, clinical history, labs, and demographics.

Proposed method

• Use embedding layers to convert images and text (unstructured and structured) into visual and text tokens.
• Employ bidirectional Transformer blocks with intramodal and intermodal attention to learn a holistic representation.
• Process radiographs, chief complaint, clinical history, laboratory results, and demographic information in a unified architecture.
• Compare unified multimodal model to image-only and non-unified multimodal baselines.
• Evaluate on pulmonary disease identification and COVID-19 adverse outcome prediction.

Experimental results

Research questions

• RQ1 Can a unified multimodal Transformer model outperform image-only models in identifying pulmonary diseases?
• RQ2 Can the unified model improve prediction of adverse clinical outcomes in COVID-19 compared to non-unified multimodal approaches?
• RQ3 Does unifying multiple data modalities (images, text, structured data) enhance diagnostic decision-making?
• RQ4 What is the impact of intramodal and intermodal attention in learning holistic clinical representations?

Key findings

• The unified multimodal model outperformed an image-only model in pulmonary-disease identification by 12%.
• The unified model outperformed non-unified multimodal models in pulmonary-disease identification by 9%.
• For adverse COVID-19 outcomes, the unified model achieved a 29% improvement over the image-only baseline and a 7% improvement over the non-unified multimodal model.