[Paper Review] Model-assisted cohort selection with bias analysis for generating large-scale cohorts from the EHR for oncology research

Benjamin Birnbaum, Nathan C. Nussbaum+8 more|arXiv (Cornell University)|Jan 13, 2020

Machine Learning in HealthcareComputer Science11 references200 citations

TL;DR

The paper introduces Model-Assisted Cohort Selection (MACS) with Bias Analysis to efficiently generate large EHR-based oncology cohorts, demonstrating high predictive performance and no detectable bias in subsequent analyses.

ABSTRACT

Objective Electronic health records (EHRs) are a promising source of data for health outcomes research in oncology. A challenge in using EHR data is that selecting cohorts of patients often requires information in unstructured parts of the record. Machine learning has been used to address this, but even high-performing algorithms may select patients in a non-random manner and bias the resulting cohort. To improve the efficiency of cohort selection while measuring potential bias, we introduce a technique called Model-Assisted Cohort Selection (MACS) with Bias Analysis and apply it to the selection of metastatic breast cancer (mBC) patients. Materials and Methods We trained a model on 17,263 patients using term-frequency inverse-document-frequency (TF-IDF) and logistic regression. We used a test set of 17,292 patients to measure algorithm performance and perform Bias Analysis. We compared the cohort generated by MACS to the cohort that would have been generated without MACS as reference standard, first by comparing distributions of an extensive set of clinical and demographic variables and then by comparing the results of two analyses addressing existing example research questions. Results Our algorithm had an area under the curve (AUC) of 0.976, a sensitivity of 96.0%, and an abstraction efficiency gain of 77.9%. During Bias Analysis, we found no large differences in baseline characteristics and no differences in the example analyses. Conclusion MACS with bias analysis can significantly improve the efficiency of cohort selection on EHR data while instilling confidence that outcomes research performed on the resulting cohort will not be biased.

Motivation & Objective

• Motivate using EHR data for oncology outcomes research and address non-random cohort selection due to unstructured data.
• Develop a scalable cohort selection method that incorporates bias assessment to trust downstream analyses.
• Apply the method to metastatic breast cancer (mBC) to demonstrate efficiency and bias containment.

Proposed method

• Train a TF-IDF + logistic regression model on 17,263 patients to identify the target cohort.
• Evaluate performance on a held-out test set of 17,292 patients.
• Perform Bias Analysis to compare MACS-generated cohorts with a reference standard across numerous clinical and demographic variables.
• Compare distributions of variables between MACS and non-MACS cohorts.
• Show that (i) MACS achieves high discrimination and (ii) biases do not materially alter example analyses.

Experimental results

Research questions

• RQ1 Can MACS improve the efficiency of cohort selection from EHR data for oncology research?
• RQ2 Does MACS introduce detectable bias in baseline characteristics compared to a reference standard?
• RQ3 Do analyses conducted on MACS-generated cohorts yield results consistent with bias-free reference analyses?

Key findings

• AUC of 0.976 indicates strong discrimination for the MACS selector.
• Sensitivity of 96.0% shows high true-positive capture of the target cohort.
• Abstraction efficiency gain of 77.9% demonstrates substantial workflow improvement.
• Bias Analysis revealed no large differences in baseline characteristics between MACS and reference cohorts.
• No differences were found in the example analyses between the MACS-derived cohort and the reference analyses.