[Paper Review] Coupling Symbolic Reasoning with Language Modeling for Efficient Longitudinal Understanding of Unstructured Electronic Medical Records

Shivani Shekhar, Simran Tiwari+3 more|arXiv (Cornell University)|Aug 7, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes a hybrid framework that integrates symbolic reasoning with large language models (LLMs) to improve longitudinal understanding of unstructured electronic medical records (EMRs). By combining LLMs' natural language understanding with symbolic logic for structured reasoning, the approach enhances extraction of medical variables and achieves performance comparable to commercial LLMs, even with open-source models, while demonstrating that symbolic steering significantly boosts LLM reliability.

ABSTRACT

The application of Artificial Intelligence (AI) in healthcare has been revolutionary, especially with the recent advancements in transformer-based Large Language Models (LLMs). However, the task of understanding unstructured electronic medical records remains a challenge given the nature of the records (e.g., disorganization, inconsistency, and redundancy) and the inability of LLMs to derive reasoning paradigms that allow for comprehensive understanding of medical variables. In this work, we examine the power of coupling symbolic reasoning with language modeling toward improved understanding of unstructured clinical texts. We show that such a combination improves the extraction of several medical variables from unstructured records. In addition, we show that the state-of-the-art commercially-free LLMs enjoy retrieval capabilities comparable to those provided by their commercial counterparts. Finally, we elaborate on the need for LLM steering through the application of symbolic reasoning as the exclusive use of LLMs results in the lowest performance.

Motivation & Objective

• To address the challenge of extracting meaningful medical variables from disorganized, inconsistent, and redundant unstructured EMRs.
• To investigate whether symbolic reasoning can enhance the reasoning capabilities of LLMs in clinical text understanding.
• To evaluate the performance of state-of-the-art open-source LLMs in EMR extraction tasks, especially in comparison to commercial models.
• To demonstrate that symbolic reasoning steering improves LLM reliability and reduces hallucination in clinical reasoning.

Proposed method

• The framework couples a pre-trained LLM with a symbolic reasoning engine to process longitudinal clinical notes.
• Symbolic rules are designed to encode clinical logic, such as temporal relationships, diagnostic criteria, and treatment sequences.
• The LLM extracts candidate entities and relations from unstructured text, which are then validated and refined using symbolic reasoning.
• A retrieval-augmented generation pipeline enhances context awareness, enabling better handling of long-term patient histories.
• The system uses rule-based filtering and consistency checks to correct LLM-generated outputs and improve factual accuracy.
• The approach is evaluated on real-world EMR datasets using standard medical variable extraction benchmarks.

Experimental results

Research questions

• RQ1 Can symbolic reasoning improve the accuracy and consistency of medical variable extraction from unstructured EMRs when combined with LLMs?
• RQ2 How does the performance of open-source LLMs compare to commercial LLMs in longitudinal clinical text understanding when enhanced with symbolic reasoning?
• RQ3 To what extent does symbolic steering reduce hallucination and improve reasoning reliability in LLM-based clinical systems?
• RQ4 Can a hybrid symbolic-LLM system achieve performance on par with commercial models in EMR understanding tasks?

Key findings

• The hybrid symbolic-LLM system significantly improves extraction accuracy of key medical variables compared to LLMs used in isolation.
• Open-source LLMs, when guided by symbolic reasoning, achieve retrieval and reasoning performance comparable to commercial LLMs.
• Symbolic reasoning reduces hallucination and improves consistency in clinical reasoning, especially in longitudinal contexts.
• The combination of LLMs with symbolic rules leads to more reliable and interpretable clinical decision support outputs.