[Paper Review] A Study of Social and Behavioral Determinants of Health in Lung Cancer Patients Using Transformers-based Natural Language Processing Models

Zehao Yu, Xi Yang+9 more|PubMed|Aug 10, 2021

Food Security and Health in Diverse Populations38 references38 citations

TL;DR

The paper compares BERT and RoBERTa transformer-based NLP models to extract SBDoH concepts from clinical narratives and shows narratives add detail beyond structured EHRs for a lung cancer cohort.

ABSTRACT

Social and behavioral determinants of health (SBDoH) have important roles in shaping people's health. In clinical research studies, especially comparative effectiveness studies, failure to adjust for SBDoH factors will potentially cause confounding issues and misclassification errors in either statistical analyses and machine learning-based models. However, there are limited studies to examine SBDoH factors in clinical outcomes due to the lack of structured SBDoH information in current electronic health record (EHR) systems, while much of the SBDoH information is documented in clinical narratives. Natural language processing (NLP) is thus the key technology to extract such information from unstructured clinical text. However, there is not a mature clinical NLP system focusing on SBDoH. In this study, we examined two state-of-the-art transformer-based NLP models, including BERT and RoBERTa, to extract SBDoH concepts from clinical narratives, applied the best performing model to extract SBDoH concepts on a lung cancer screening patient cohort, and examined the difference of SBDoH information between NLP extracted results and structured EHRs (SBDoH information captured in standard vocabularies such as the International Classification of Diseases codes). The experimental results show that the BERT-based NLP model achieved the best strict/lenient F1-score of 0.8791 and 0.8999, respectively. The comparison between NLP extracted SBDoH information and structured EHRs in the lung cancer patient cohort of 864 patients with 161,933 various types of clinical notes showed that much more detailed information about smoking, education, and employment were only captured in clinical narratives and that it is necessary to use both clinical narratives and structured EHRs to construct a more complete picture of patients' SBDoH factors.

Motivation & Objective

• Motivate the importance of social and behavioral determinants of health (SBDoH) in clinical outcomes and reduce confounding/misclassification in analyses.
• Assess the capability of state-of-the-art transformer NLP models to extract SBDoH concepts from clinical narratives.
• Compare NLP-extracted SBDoH information with structured EHR data to evaluate completeness of SBDoH capture.
• Apply the best-performing model to a lung cancer screening cohort to characterize SBDoH factors.

Proposed method

• Evaluate two transformer NLI models, BERT and RoBERTa, for SBDoH concept extraction from clinical narratives.
• Measure model performance using strict and lenient F1-scores on SBDoH extraction.
• Compare NLP-derived SBDoH data with structured EHR SBDoH data across a cohort of 864 patients with 161,933 notes.
• Analyze differences in information captured for smoking, education, and employment between narrative vs structured records.

Experimental results

Research questions

• RQ1 Can transformer-based NLP models accurately extract SBDoH concepts from unstructured clinical narratives?
• RQ2 Which model (BERT or RoBERTa) provides higher accuracy for SBDoH extraction in clinical text?
• RQ3 How does NLP-extracted SBDoH information compare to structured EHR SBDoH data in terms of completeness?
• RQ4 What SBDoH factors are better captured in narratives (e.g., smoking, education, employment) in a lung cancer cohort?

Key findings

• BERT-based NLP achieved the best strict/lenient F1-scores of 0.8791 and 0.8999, respectively.
• NLP extracted SBDoH information detected much more detail on smoking, education, and employment than structured EHR vocabularies.
• In a cohort of 864 lung cancer patients with 161,933 notes, narratives complemented structured EHRs to form a more complete SBDoH picture.
• Both clinical narratives and structured EHR data are necessary to construct a comprehensive SBDoH profile for patients.