[Paper Review] Fast Prototyping a Dialogue Comprehension System for Nurse-Patient Conversations on Symptom Monitoring

Zhengyuan Liu, Hazel Lim+10 more|arXiv (Cornell University)|Mar 8, 2019

Topic Modeling27 references4 citations

TL;DR

This paper proposes a fast-prototyping framework for a dialogue comprehension system in nurse-patient symptom monitoring using minimal real-world conversations. By simulating human-human spoken dialogues with linguistic realism—such as topic drift, thinking aloud, and self-contradiction—it trains a bidirectional attention pointer network on synthetic data, achieving 80.18% F1 score on real-world nurse-patient conversations, demonstrating feasibility for clinical information extraction with limited data.

ABSTRACT

Data for human-human spoken dialogues for research and development are currently very limited in quantity, variety, and sources; such data are even scarcer in healthcare. In this work, we investigate fast prototyping of a dialogue comprehension system by leveraging on minimal nurse-to-patient conversations. We propose a framework inspired by nurse-initiated clinical symptom monitoring conversations to construct a simulated human-human dialogue dataset, embodying linguistic characteristics of spoken interactions like thinking aloud, self-contradiction, and topic drift. We then adopt an established bidirectional attention pointer network on this simulated dataset, achieving more than 80% F1 score on a held-out test set from real-world nurse-to-patient conversations. The ability to automatically comprehend conversations in the healthcare domain by exploiting only limited data has implications for improving clinical workflows through red flag symptom detection and triaging capabilities. We demonstrate the feasibility for efficient and effective extraction, retrieval and comprehension of symptom checking information discussed in multi-turn human-human spoken conversations.

Motivation & Objective

• To address the scarcity of annotated, large-scale spoken dialogue data in healthcare, especially for human-human interactions.
• To develop a feasible, data-efficient method for automatic comprehension of symptom monitoring conversations between nurses and patients.
• To simulate linguistically realistic spoken dialogues that capture key characteristics like topic drift, thinking aloud, and anaphora for training.
• To prototype a question-answering system capable of extracting clinically relevant symptom attributes from multi-turn spoken dialogues.
• To evaluate model performance on real-world nurse-patient conversations despite distributional shifts from simulated training data.

Proposed method

• Constructed a simulated human-human dialogue dataset using clinically validated templates inspired by nurse-initiated symptom monitoring conversations.
• Incorporated linguistic features of spoken dialogue, including topic drift, thinking aloud, self-contradiction, and zero anaphora, to enhance realism.
• Employed a bidirectional attention pointer network as the core QA model to extract answers from multi-turn dialogues based on symptom-related queries.
• Used pre-trained GloVe embeddings and a bi-directional attention mechanism to improve context modeling and answer span prediction.
• Trained the model on synthetic data of varying sizes (10k to 150k samples) and evaluated generalization on in-distribution, out-of-distribution, and real-world test sets.
• Conducted ablation studies by removing bi-attention and pre-trained embeddings to assess their contribution to performance.

Experimental results

Research questions

• RQ1 Can a dialogue comprehension system be effectively prototyped using only minimal real-world nurse-patient conversations?
• RQ2 How well does a model trained on simulated spoken dialogues generalize to real-world, naturally occurring nurse-patient conversations?
• RQ3 To what extent do linguistic phenomena like topic drift, thinking aloud, and anaphora affect model performance in dialogue comprehension?
• RQ4 What is the impact of key architectural components—bi-directional attention and pre-trained word embeddings—on model performance?
• RQ5 How does the model handle out-of-distribution symptoms and non-clinical utterances like chit-chat in real-world settings?

Key findings

• The best-performing model achieved an F1 score of 80.18% on a held-out real-world test set of nurse-patient conversations, demonstrating strong generalization despite data distribution shifts.
• Performance improved with training data size, peaking at 100k samples, with F1 scores of 94.17% on the base evaluation set and 85.69% on the augmented set.
• The model maintained strong performance on out-of-distribution symptoms, indicating robustness to unseen symptom expressions.
• Ablation studies showed that removing bi-directional attention reduced F1 by ~10%, and removing pre-trained embeddings caused an 18% performance drop on real-world data.
• Error analysis revealed that sparsity in expression pools, chit-chat in real conversations, and causal elaborations between symptoms were major sources of performance degradation.
• The model achieved 78.23% exact match and 80.18% F1 on real-world conversations, indicating practical viability for clinical workflow support.