[Paper Review] PathAsst: A Generative Foundation AI Assistant Towards Artificial General Intelligence of Pathology

Yuxuan Sun, Chenglu Zhu+7 more|arXiv (Cornell University)|May 24, 2023

Data Mining Algorithms and Applications4 citations

TL;DR

PathAsst is a generative foundation AI assistant tailored for pathology, combining a vision encoder with a large language model to interpret histopathology images and invoke specialized sub-models for complex tasks like cell classification and detection. It achieves superior diagnostic reasoning compared to baseline MLLMs by leveraging 142K curated image-text pairs and 180K instruction-following samples, demonstrating advanced multimodal reasoning in clinical contexts.

ABSTRACT

As advances in large language models (LLMs) and multimodal techniques continue to mature, the development of general-purpose multimodal large language models (MLLMs) has surged, offering significant applications in interpreting natural images. However, the field of pathology has largely remained untapped, particularly in gathering high-quality data and designing comprehensive model frameworks. To bridge the gap in pathology MLLMs, we present PathAsst, a multimodal generative foundation AI assistant to revolutionize diagnostic and predictive analytics in pathology. The development of PathAsst involves three pivotal steps: data acquisition, CLIP model adaptation, and the training of PathAsst's multimodal generative capabilities. Firstly, we collect over 207K high-quality pathology image-text pairs from authoritative sources. Leveraging the advanced power of ChatGPT, we generate over 180K instruction-following samples. Furthermore, we devise additional instruction-following data specifically tailored for invoking eight pathology-specific sub-models we prepared, allowing the PathAsst to effectively collaborate with these models, enhancing its diagnostic ability. Secondly, by leveraging the collected data, we construct PathCLIP, a pathology-dedicated CLIP, to enhance PathAsst's capabilities in interpreting pathology images. Finally, we integrate PathCLIP with the Vicuna-13b and utilize pathology-specific instruction-tuning data to enhance the multimodal generation capacity of PathAsst and bolster its synergistic interactions with sub-models. The experimental results of PathAsst show the potential of harnessing AI-powered generative foundation model to improve pathology diagnosis and treatment processes.

Motivation & Objective

• To address the lack of multimodal large language models (MLLMs) tailored for pathology by developing a foundation model that understands and reasons over histopathology images.
• To improve diagnostic and predictive analytics in pathology by enabling dynamic invocation of specialized sub-models based on user intent and image content.
• To create and release a high-quality, open-source dataset of 142K pathology image-text pairs and 180K instruction-following samples for training robust pathology-specific AI models.
• To support researchers in building and fine-tuning their own pathology foundation models through open-sourced data collection tools and preprocessing pipelines.

Proposed method

• Curated 142K high-quality pathology image-text pairs from PubMed, pathology textbooks, reputable websites, and expert-annotated private data after rigorous cleaning and optimization.
• Generated 180K instruction-following samples using GPT-4 to train the foundation model on diverse diagnostic and analytical tasks.
• Designed specialized instruction-following data for model invocation, enabling PathAsst to dynamically select and utilize eight task-specific sub-models (e.g., LBC cell generator, PD-L1 detector).
• Trained PathAsst on a Vicuna-13B language model with a CLIP vision encoder (clip-vit-base-patch32) to enable multimodal understanding and reasoning.
• Developed a pipeline for automated extraction of image-text pairs from PDF pathology books using custom scripts.
• Planned to retrain and release multiple CLIP variants (e.g., vit-large-patch14-336) fine-tuned on over 200K pathology samples for improved feature extraction.

Experimental results

Research questions

• RQ1 Can a generative foundation model effectively interpret complex histopathology images and generate clinically relevant diagnostic insights?
• RQ2 How does dynamic invocation of specialized sub-models improve diagnostic accuracy and task generalization in pathology compared to end-to-end multimodal generation?
• RQ3 To what extent does high-quality, curated data enhance the performance of foundation models in pathology-specific tasks?
• RQ4 Can instruction tuning with pathology-specific prompts significantly improve zero-shot and few-shot capabilities in diagnostic reasoning?

Key findings

• PathAsst outperforms baseline MLLMs like LLaVA and MiniGPT-4 in recognizing pathological features such as enlarged nuclei and irregular nuclear membranes in histopathology images.
• The model successfully generates liquid-based cytology (LBC) cells of the LSIL category with irregular cytoplasm by invoking a dedicated LBC cell generation model, demonstrating task-specific adaptability.
• For cell counting tasks, PathAsst correctly invokes a PD-L1 cell detection model, marks positive cells, and outputs structured results in markdown format with accurate statistics.
• The model’s ability to reason about image content and select appropriate sub-models leads to more accurate and contextually relevant diagnostic outputs than direct multimodal generation alone.
• The use of GPT-4 to generate 180K instruction-following samples significantly enhances the model’s ability to follow complex, multi-step diagnostic instructions.