[Paper Review] EHRSQL: A Practical Text-to-SQL Benchmark for Electronic Health Records

Gyubok Lee, Hyeonji Hwang+7 more|arXiv (Cornell University)|Jan 16, 2023

Electronic Health Records Systems12 citations

TL;DR

EHRSQL introduces a large-scale text-to-SQL benchmark for structured EHR data, linking questions to MIMIC-III and eICU, with time-aware queries and unanswerable cases to evaluate trustworthy semantic parsing.

ABSTRACT

We present a new text-to-SQL dataset for electronic health records (EHRs). The utterances were collected from 222 hospital staff members, including physicians, nurses, and insurance review and health records teams. To construct the QA dataset on structured EHR data, we conducted a poll at a university hospital and used the responses to create seed questions. We then manually linked these questions to two open-source EHR databases, MIMIC-III and eICU, and included various time expressions and held-out unanswerable questions in the dataset, which were also collected from the poll. Our dataset poses a unique set of challenges: the model needs to 1) generate SQL queries that reflect a wide range of needs in the hospital, including simple retrieval and complex operations such as calculating survival rate, 2) understand various time expressions to answer time-sensitive questions in healthcare, and 3) distinguish whether a given question is answerable or unanswerable. We believe our dataset, EHRSQL, can serve as a practical benchmark for developing and assessing QA models on structured EHR data and take a step further towards bridging the gap between text-to-SQL research and its real-life deployment in healthcare. EHRSQL is available at https://github.com/glee4810/EHRSQL.

Motivation & Objective

• Reflect real-world hospital information needs by collecting questions from 222 hospital staff across professions.
• Link questions to two open-source EHR databases (MIMIC-III and eICU) to enable practical QA over structured data.
• Incorporate time-sensitive questions with diverse time expressions to mirror healthcare workflows.
• Include unanswerable questions to evaluate model reliability and abstention in addition to answer generation.

Proposed method

• Collect real-world questions via a hospital poll and templatize them into answerable and unanswerable templates.
• Design three time-filter types (global, within, exact) with multiple expression types, units, and intervals for NL and SQL representations.
• Manually annotate SQL queries for MIMIC-III and eICU and favor nested queries over broad JOIN usage to reflect EHR schemas.
• Paraphrase templates to increase linguistic diversity using human paraphrasing and ML-based back-translation with quality filtering.
• Introduce a trustworthy semantic parsing task that jointly evaluates answerability recognition and executable SQL correctness.

Figure 1: Demographics of the 222 respondents by hospital departments and the years of experience.

Experimental results

Research questions

• RQ1 How well can a model translate hospital staff questions into SQL queries over MIMIC-III and eICU schemas?
• RQ2 Can the dataset support time-sensitive healthcare questions with diverse time expressions and intervals?
• RQ3 Can a model correctly identify unanswerable questions and refrain from querying the database?
• RQ4 What is the impact of schema information on model performance in healthcare text-to-SQL?
• RQ5 How do state-of-the-art semantic parsers transfer to healthcare domains and handle complex EHR queries?

Key findings

• EHRSQL combines 24K question-SQL pairs across two databases with substantial time-awareness and unanswerable instances.
• Baseline models (e.g., T5 variants) show varying performance, with percentile-based uncertainty thresholds yielding best validation/test F1 and execution metrics.
• Zero-shot cross-domain transfer from Spider-based models to EHRSQL is limited, underscoring domain-specific challenges in healthcare SQL generation.
• Inclusion of time expressions significantly increases query complexity, necessitating time-aware SQL patterns (strftime, datetime, NULL handling).
• EHRSQL outperforms comparable EHR QA datasets on several realism axes, including unanswerable questions and schema-unaware question origins.

Figure 2: EHR schema is structured in a hierarchy, and real-world questions often require multi-step reasoning over tables.