[Paper Review] The 2022 n2c2/UW Shared Task on Extracting Social Determinants of Health

Kevin Lybarger, Meliha Yetişgen+1 more|arXiv (Cornell University)|Jan 13, 2023

Food Security and Health in Diverse PopulationsHealth Professions3 citations

TL;DR

This paper presents the 2022 n2c2/UW Shared Task on extracting Social Determinants of Health (SDOH) from clinical notes using the Social History Annotated Corpus (SHAC), which features event-based annotations for SDOH like substance use, employment, and living status with attributes for status, extent, and temporality. The top-performing system used a sequence-to-sequence transformer model (T5) achieving 0.901 F1 on Subtask A, outperforming rule-based and traditional ML methods, with lower performance observed for high-risk SDOH factors such as homelessness and polysubstance use compared to protective factors.

ABSTRACT

Objective: The n2c2/UW SDOH Challenge explores the extraction of social determinant of health (SDOH) information from clinical notes. The objectives include the advancement of natural language processing (NLP) information extraction techniques for SDOH and clinical information more broadly. This paper presents the shared task, data, participating teams, performance results, and considerations for future work. Materials and Methods: The task used the Social History Annotated Corpus (SHAC), which consists of clinical text with detailed event-based annotations for SDOH events such as alcohol, drug, tobacco, employment, and living situation. Each SDOH event is characterized through attributes related to status, extent, and temporality. The task includes three subtasks related to information extraction (Subtask A), generalizability (Subtask B), and learning transfer (Subtask C). In addressing this task, participants utilized a range of techniques, including rules, knowledge bases, n-grams, word embeddings, and pretrained language models (LM). Results: A total of 15 teams participated, and the top teams utilized pretrained deep learning LM. The top team across all subtasks used a sequence-to-sequence approach achieving 0.901 F1 for Subtask A, 0.774 F1 Subtask B, and 0.889 F1 for Subtask C. Conclusions: Similar to many NLP tasks and domains, pretrained LM yielded the best performance, including generalizability and learning transfer. An error analysis indicates extraction performance varies by SDOH, with lower performance achieved for conditions, like substance use and homelessness, that increase health risks (risk factors) and higher performance achieved for conditions, like substance abstinence and living with family, that reduce health risks (protective factors).

Motivation & Objective

• To advance natural language processing (NLP) techniques for extracting social determinants of health (SDOH) from unstructured clinical notes.
• To evaluate the performance of diverse NLP methods—ranging from rule-based systems to deep learning models—on a newly annotated clinical corpus.
• To assess model generalizability across domains and transfer learning potential in SDOH extraction.
• To identify challenges in extracting nuanced, high-risk SDOH factors such as substance use and homelessness.
• To provide insights into bias in NLP models related to health risk factors, particularly for vulnerable populations.

Proposed method

• The shared task used the Social History Annotated Corpus (SHAC), a clinical text corpus with event-based annotations for 13 SDOH concepts including alcohol, drug, tobacco, employment, and living situation.
• Annotations included attributes for status (present/absent), extent (frequency, duration), and temporality (past/present/future), enabling fine-grained event extraction.
• Three subtasks were defined: Subtask A (SDOH extraction on MIMIC data), Subtask B (generalizability to UW data), and Subtask C (transfer learning using in-domain data).
• Participants applied a range of techniques, including rule-based systems, n-grams, word embeddings, knowledge bases, and especially pretrained language models (e.g., BERT, T5).
• The top-performing system used a sequence-to-sequence (seq2seq) model based on T5, which generated structured SDOH event outputs directly from raw text.
• Performance was evaluated using F1 scores, with error analysis focusing on differences by SDOH type (risk vs. protective factors) and event density.

Figure 1 : BRAT annotation example.

Experimental results

Research questions

• RQ1 How do different NLP architectures—rule-based, traditional ML, and deep learning—perform in extracting SDOH from clinical notes?
• RQ2 To what extent can models trained on one EHR source generalize to another domain, as measured in Subtask B?
• RQ3 How effective is transfer learning when fine-tuning on in-domain data, as evaluated in Subtask C?
• RQ4 Why do certain SDOH factors, particularly high-risk ones like homelessness and polysubstance use, show lower extraction performance?
• RQ5 How does the density of SDOH events in a note affect extraction accuracy?

Key findings

• The top-performing system used a sequence-to-sequence T5-based model, achieving 0.901 F1 on Subtask A, 0.774 F1 on Subtask B, and 0.889 F1 on Subtask C.
• Pretrained language models significantly outperformed rule-based, n-gram, and knowledge-base methods, with deep learning models dominating performance across all subtasks.
• Models showed lower extraction performance for high-risk SDOH factors such as current/past substance use, homelessness, and disability status compared to protective factors like abstinence and living with family.
• Performance decreased with increasing SDOH event density, suggesting that notes with multiple complex social needs are more challenging to extract accurately.
• Generalizability was higher for alcohol, drug, tobacco, and employment SDOH than for living status, indicating domain shift challenges for certain SDOH types.
• The error analysis revealed a potential bias in NLP models toward under-extracting high-risk SDOH, which may impact health equity efforts.

Figure 2 : Annotation examples describing event extraction as a slot filling task.