[Paper Review] Advancing PICO Element Detection in Biomedical Text via Deep Neural Networks

Di Jin, Peter Szolovits|arXiv (Cornell University)|Oct 30, 2018

Topic Modeling28 references4 citations

TL;DR

This paper proposes a deep learning model that enhances PICO element detection in biomedical texts by stacking a second bi-LSTM layer on top of sentence representations to capture cross-sentence context, and improves generalization via adversarial training and unsupervised pre-training. The model achieves state-of-the-art performance, improving F1 scores by 5.5%, 7.9%, and 5.8% on PubMed-PICO for P, I, and O elements, respectively, and 2.4%, 13.6%, and 1.0% on NICTA-PIBOSO.

ABSTRACT

In evidence-based medicine (EBM), defining a clinical question in terms of the specific patient problem aids the physicians to efficiently identify appropriate resources and search for the best available evidence for medical treatment. In order to formulate a well-defined, focused clinical question, a framework called PICO is widely used, which identifies the sentences in a given medical text that belong to the four components typically reported in clinical trials: Participants/Problem (P), Intervention (I), Comparison (C) and Outcome (O). In this work, we propose a novel deep learning model for recognizing PICO elements in biomedical abstracts. Based on the previous state-of-the-art bidirectional long-short term memory (biLSTM) plus conditional random field (CRF) architecture, we add another layer of biLSTM upon the sentence representation vectors so that the contextual information from surrounding sentences can be gathered to help infer the interpretation of the current one. In addition, we propose two methods to further generalize and improve the model: adversarial training and unsupervised pre-training over large corpora. We tested our proposed approach over two benchmark datasets. One is the PubMed-PICO dataset, where our best results outperform the previous best by 5.5%, 7.9%, and 5.8% for P, I, and O elements in terms of F1 score, respectively. And for the other dataset named NICTA-PIBOSO, the improvements for P/I/O elements are 2.4%, 13.6%, and 1.0% in F1 score, respectively. Overall, our proposed deep learning model can obtain unprecedented PICO element detection accuracy while avoiding the need for any manual feature selection.

Motivation & Objective

• To improve the accuracy of PICO element detection in biomedical abstracts, which is critical for evidence-based medicine and clinical decision support.
• To address the limitations of prior models that rely on handcrafted features and fail to exploit contextual information from surrounding sentences.
• To reduce overfitting in low-resource settings by enhancing model generalization through adversarial training and unsupervised pre-training.
• To achieve state-of-the-art performance without manual feature engineering, leveraging deep neural networks for end-to-end learning.

Proposed method

• Stacks a second bi-LSTM layer on top of sentence-level representations from a bi-LSTM+CRF model to capture contextual information from neighboring sentences.
• Uses adversarial training and virtual adversarial training to regularize the model and improve robustness to input perturbations.
• Applies unsupervised pre-training on large biomedical corpora before fine-tuning on target PICO datasets to enhance feature learning.
• Employs a bi-LSTM+CRF architecture where the bi-LSTM encodes sentence representations and the CRF optimizes label sequence predictions.
• Utilizes both RNN and Transformer-based sentence encoders, with ablation studies comparing their effectiveness.
• Integrates contextual sentence representations into a CRF layer to ensure globally consistent label sequences.

Experimental results

Research questions

• RQ1 Can stacking an additional bi-LSTM layer on top of sentence representations improve PICO element detection by capturing inter-sentence context?
• RQ2 Does adversarial training significantly improve generalization and robustness in low-resource PICO labeling scenarios?
• RQ3 To what extent does unsupervised pre-training on large biomedical corpora enhance downstream PICO detection performance?
• RQ4 How does the proposed model compare to state-of-the-art models in terms of F1 score across multiple benchmark datasets?

Key findings

• The proposed model achieves a 5.5% absolute improvement in F1 score for the PICO element on the PubMed-PICO dataset compared to the previous state-of-the-art.
• The model improves F1 scores by 7.9% for the Intervention (I) element and 5.8% for the Outcome (O) element on the PubMed-PICO dataset.
• On the NICTA-PIBOSO dataset, the model achieves a 13.6% improvement in F1 score for the Intervention (I) element, the largest gain among all elements.
• The model reduces reliance on manual feature engineering by learning contextual representations end-to-end through deep neural networks.
• Adversarial training and unsupervised pre-training contribute significantly to performance gains, especially in low-data regimes.
• Confusion matrices reveal that the main error sources are ambiguous labels and CRF-induced label transitions, with 'Method' and 'Other' labels being frequent confusions.