[Paper Review] Exploring the Inquiry-Diagnosis Relationship with Advanced Patient Simulators

Zhaocheng Liu, Qiufen Tu+8 more|ArXiv.org|Jan 16, 2025

Simulation-Based Education in Healthcare3 citations

TL;DR

The paper builds a data-driven patient simulator guided by real doctor-patient dialogue strategies to study how inquiry quality affects diagnosis in online medical consultations, demonstrating Liebig’s law in this context.

ABSTRACT

Recently, large language models have shown great potential to transform online medical consultation. Despite this, most research targets improving diagnostic accuracy with ample information, often overlooking the inquiry phase. Some studies try to evaluate or refine doctor models by using prompt-engineered patient agents. However, prompt engineering alone falls short in accurately simulating real patients. We need to explore new paradigms for patient simulation. Furthermore, the relationship between inquiry and diagnosis remains unexplored. This paper extracts dialogue strategies from real doctor-patient conversations to guide the training of a patient simulator. Our simulator shows higher anthropomorphism and lower hallucination rates, using dynamic dialogue strategies. This innovation offers a more accurate evaluation of diagnostic models and generates realistic synthetic data. We conduct extensive experiments on the relationship between inquiry and diagnosis, showing they adhere to Liebig's law: poor inquiry limits diagnosis effectiveness, regardless of diagnostic skill, and vice versa. The experiments also reveal substantial differences in inquiry performance among models. To delve into this phenomenon, the inquiry process is categorized into four distinct types. Analyzing the distribution of inquiries across these types helps explain the performance differences. The weights of our patient simulator are available https://github.com/PatientSimulator/PatientSimulator.

Motivation & Objective

• Extract real-world patient dialogue strategies from authentic doctor-patient conversations.
• Train a patient simulator that closely mirrors real patient behavior using synthesized data.
• Investigate how inquiry quality and diagnostic capability interact to affect final diagnosis.
• Categorize inquiry types and analyze their distribution across models to explain performance differences.

Proposed method

• Annotate real doctor-patient conversations with a curated set of dialogue strategy tags.
• Synthesize doctor-patient dialogues using in-context learning with medical records and strategy flows.
• Fine-tune a patient simulator (LoRA on Qwen2.5-72B-Instruct) to output realistic patient responses.
• Use the simulator to generate fixed-round inquiry records and evaluate cross-model diagnostic accuracy.
• Model a workflow to extract and compare diagnostic results across different doctor models with a consistent evaluation pipeline.

Experimental results

Research questions

• RQ1 How does the quality of patient inquiries influence diagnostic accuracy under varying diagnostic capabilities?
• RQ2 Do different inquiry strategies (types) affect the final diagnosis, and how do model differences explain performance gaps?
• RQ3 Can a data-driven patient simulator reproduce realistic inquiry-diagnosis dynamics more accurately than prompt-engineered baselines?
• RQ4 What are the four types of patient inquiries and how do their distributions vary across models and rounds?

Key findings

• Inquiry and diagnosis follow Liebig’s law: poor inquiry quality limits diagnostic effectiveness regardless of diagnostic capability, and vice versa.
• Our patient simulator achieves lower hallucination (HR) and higher anthropomorphism (AS) than baselines, though with somewhat higher irrelevant responses (IRR) than GPT-4o-based AgentClinic.
• There are significant differences in inquiry quality across models, with Claude-3-5-sonnet showing comparatively poorer inquiry performance.
• More inquiry rounds generally improve diagnostic accuracy, and models vary in how they allocate inquiry types.
• Specifically, more emphasis on known-symptom specification correlates with lower overall diagnostic accuracy in some setups.
• Four inquiry types were identified: chief complaint, specification of known symptoms, accompanying symptoms, and family/medical history.