[Paper Review] PathFinder: A Multi-Modal Multi-Agent System for Medical Diagnostic Decision-Making Applied to Histopathology

Fatemeh Ghezloo, Mehmet Saygın Seyfioğlu+7 more|ArXiv.org|Feb 13, 2025

Biomedical Text Mining and Ontologies5 citations

TL;DR

PathFinder is a four-agent framework that mimics pathologist decision-making to diagnose whole slide images in histopathology, achieving state-of-the-art accuracy on melanoma classification while providing natural language explanations.

ABSTRACT

Diagnosing diseases through histopathology whole slide images (WSIs) is fundamental in modern pathology but is challenged by the gigapixel scale and complexity of WSIs. Trained histopathologists overcome this challenge by navigating the WSI, looking for relevant patches, taking notes, and compiling them to produce a final holistic diagnostic. Traditional AI approaches, such as multiple instance learning and transformer-based models, fail short of such a holistic, iterative, multi-scale diagnostic procedure, limiting their adoption in the real-world. We introduce PathFinder, a multi-modal, multi-agent framework that emulates the decision-making process of expert pathologists. PathFinder integrates four AI agents, the Triage Agent, Navigation Agent, Description Agent, and Diagnosis Agent, that collaboratively navigate WSIs, gather evidence, and provide comprehensive diagnoses with natural language explanations. The Triage Agent classifies the WSI as benign or risky; if risky, the Navigation and Description Agents iteratively focus on significant regions, generating importance maps and descriptive insights of sampled patches. Finally, the Diagnosis Agent synthesizes the findings to determine the patient's diagnostic classification. Our Experiments show that PathFinder outperforms state-of-the-art methods in skin melanoma diagnosis by 8% while offering inherent explainability through natural language descriptions of diagnostically relevant patches. Qualitative analysis by pathologists shows that the Description Agent's outputs are of high quality and comparable to GPT-4o. PathFinder is also the first AI-based system to surpass the average performance of pathologists in this challenging melanoma classification task by 9%, setting a new record for efficient, accurate, and interpretable AI-assisted diagnostics in pathology. Data, code and models available at https://pathfinder-dx.github.io/

Motivation & Objective

• Motivate efficient, holistic histopathology diagnostics at gigapixel scales.
• Propose a four-agent system (Triage, Navigation, Description, Diagnosis) to emulate pathologist workflows.
• Enable interpretable diagnoses through natural language descriptions of patches and findings.
• Demonstrate improved diagnostic accuracy over state-of-the-art methods and human averages on the M-Path melanoma dataset.

Proposed method

• Divide WSIs into 512x512 patches; use Quilt-Net to extract patch features.
• Triage Agent: transformer-based classifier to separate benign from risky WSIs.
• Navigation Agent: iteratively generates a text-conditioned importance map to select patches for description.
• Description Agent: fine-tuned Quilt-LLaVA to produce concise patch descriptions.
• Diagnosis Agent: LLM with a classification head, trained with LoRA on trajectories of patch descriptions.
• Major evaluation against baselines includes ablations that vary navigation and description strategies.

Experimental results

Research questions

• RQ1 Can a multi-agent, multi-modal framework outperform single-model approaches on WSI melanoma classification?
• RQ2 Does iterative, text-conditioned navigation with patch descriptions improve diagnostic accuracy and interpretability?
• RQ3 How do description quality and agent coordination affect final diagnoses and explainability?
• RQ4 What is the contribution of the Triage step to overall system performance?

Key findings

• PathFinder achieves 74% accuracy on the M-Path melanoma diagnosis task, surpassing human experts (65%) and the best baseline (66%).
• PathFinder with certain configurations reaches 74% accuracy, and ablations show the importance of iterative, text-conditioned navigation over non-iterative or purely image-based methods.
• The Description Agent delivers human-parsable patch descriptions comparable in quality to GPT-4o while using a much smaller model (7B).
• Qualitative pathologist evaluations indicate high-quality descriptions from the Description Agent, supporting interpretability.
• PathFinder’s approach also surpasses the average performance of pathologists on this task by 9% and is the first AI-based system to do so in melanoma classification on the M-Path dataset.