[Paper Review] A Foundational Multimodal Vision Language AI Assistant for Human Pathology

Ming Y. Lu, Bowen Chen+9 more|arXiv (Cornell University)|Dec 13, 2023

Artificial Intelligence in Healthcare and Education15 citations

TL;DR

PathChat is a vision-language AI assistant for pathology built on a UNI-derived vision encoder connected to a 13B LLM, trained with 257k pathology instructions, and it outperforms baselines on multiple-choice and open-ended pathology questions, especially when clinical context is provided.

ABSTRACT

The field of computational pathology has witnessed remarkable progress in the development of both task-specific predictive models and task-agnostic self-supervised vision encoders. However, despite the explosive growth of generative artificial intelligence (AI), there has been limited study on building general purpose, multimodal AI assistants tailored to pathology. Here we present PathChat, a vision-language generalist AI assistant for human pathology using an in-house developed foundational vision encoder pretrained on 100 million histology images from over 100,000 patient cases and 1.18 million pathology image-caption pairs. The vision encoder is then combined with a pretrained large language model and the whole system is finetuned on over 250,000 diverse disease agnostic visual language instructions. We compare PathChat against several multimodal vision language AI assistants as well as GPT4V, which powers the commercially available multimodal general purpose AI assistant ChatGPT-4. When relevant clinical context is provided with the histology image, PathChat achieved a diagnostic accuracy of 87% on multiple-choice questions based on publicly available cases of diverse tissue origins and disease models. Additionally, using open-ended questions and human expert evaluation, we found that overall PathChat produced more accurate and pathologist-preferable responses to diverse queries related to pathology. As an interactive and general vision language AI assistant that can flexibly handle both visual and natural language inputs, PathChat can potentially find impactful applications in pathology education, research, and human-in-the-loop clinical decision making.

Motivation & Objective

• Motivate and enable a general-purpose multimodal AI assistant tailored to pathology.
• Develop PathChat by coupling a pathology-foundational vision encoder with a large language model.
• Curate and utilize a large pathology-focused instruction dataset for fine-tuning.
• Evaluate PathChat against open-source and commercial multimodal models across diagnostic and interactive tasks.

Proposed method

• Use UNI as the starting vision encoder pretrained on over 100 million histology images.
• Fine-tune a vision-language aligned encoder (CONCH-Large) on 1.18 million pathology image-caption pairs.
• Connect the vision encoder to a 13B parameter pretrained LLM via a multimodal projector module.
• Fine-tune the combined MLLM on a dataset of 257k pathology instructions (PathChatInstruct).
• Evaluate PathChat on multiple-choice diagnostic questions and open-ended questions against LLaVA, LLaVA-Med, and GPT4V, including context-aware scenarios.

Experimental results

Research questions

• RQ1 Can PathChat diagnose histology images in a zero-shot or few-shot setting without task-specific fine-tuning beyond PathChatInstruct?
• RQ2 How does PathChat perform relative to general-purpose and biomedically specialized MLLMs on microscopy-based diagnosis and open-ended pathology questions?
• RQ3 Does providing clinical context improve diagnostic accuracy and utility of the PathChat assistant?
• RQ4 What are the comparative strengths and weaknesses of PathChat across microscopy, diagnosis, clinical knowledge, and ancillary testing categories?

Key findings

• PathChat achieves 70.8% accuracy in image-only multiple-choice questions and 81.2% with clinical context on a combined pathology benchmark.
• PathChat outperforms LLaVA 1.5 and LLaVA-Med in both image-only and image-with-context settings.
• On open-ended questions, PathChat attains 86.1% overall accuracy, outperforming GPT4V (59.1%), LLaVA 1.5 (42.6%), and LLaVA-Med (50.4%).
• PathChat shows especially strong performance in Microscopy and Diagnosis categories, with higher accuracy than GPT4V in those areas, while GPT4V excels in Clinical and Ancillary Testing questions.
• PathChat supports interactive, multi-turn conversations and human-in-the-loop differential diagnosis workflows.