[Paper Review] Claim Extraction in Biomedical Publications using Deep Discourse Model and Transfer Learning

Titipat Achakulvisut, Chandra Bhagavatula+1 more|arXiv (Cornell University)|Jul 1, 2019

Topic Modeling37 references19 citations

TL;DR

This paper introduces a new dataset of 1,500 expert-annotated biomedical abstracts for scientific claim extraction and proposes a deep discourse model enhanced with transfer learning using a fine-tuned Bi-LSTM-CRF architecture. The method achieves a 14.1 percentage point F1-score improvement over a non-transfer learning baseline, demonstrating the effectiveness of transfer learning in claim extraction from scientific literature.

ABSTRACT

Claims are a fundamental unit of scientific discourse. The exponential growth in the number of scientific publications makes automatic claim extraction an important problem for researchers who are overwhelmed by this information overload. Such an automated claim extraction system is useful for both manual and programmatic exploration of scientific knowledge. In this paper, we introduce a new dataset of 1,500 scientific abstracts from the biomedical domain with expert annotations for each sentence indicating whether the sentence presents a scientific claim. We introduce a new model for claim extraction and compare it to several baseline models including rule-based and deep learning techniques. Moreover, we show that using a transfer learning approach with a fine-tuning step allows us to improve performance from a large discourse-annotated dataset. Our final model increases F1-score by over 14 percent points compared to a baseline model without transfer learning. We release a publicly accessible tool for discourse and claims prediction along with an annotation tool. We discuss further applications beyond biomedical literature.

Motivation & Objective

• To address the growing challenge of information overload in biomedical literature by enabling automated extraction of scientific claims.
• To create a large-scale, expert-annotated dataset of 1,500 biomedical abstracts with sentence-level claim annotations.
• To develop a deep learning model for claim extraction that leverages discourse structure and transfer learning from pre-trained language representations.
• To improve claim extraction performance beyond rule-based and non-transfer learning deep learning baselines.

Proposed method

• The authors introduce a new dataset of 1,500 biomedical abstracts with expert-annotated claims, where each sentence is labeled as a claim or not.
• They employ a Bi-LSTM with Conditional Random Field (CRF) architecture for sequence labeling, pre-trained on a discourse-annotated PubMed dataset.
• Transfer learning is applied by fine-tuning the pre-trained Bi-LSTM-CRF model on the new claim-annotated dataset to adapt general discourse representations to claim detection.
• The model uses pre-trained word embeddings (GloVe and PubMed-specific) and is trained end-to-end with cross-entropy loss and CRF decoding.
• An annotation tool was developed to support expert labeling with inter-annotator agreement checks, ensuring high-quality data.
• A publicly available web service and codebase are released, including tools for discourse and claim prediction.

Experimental results

Research questions

• RQ1 Can transfer learning from a large discourse-annotated dataset improve claim extraction performance in biomedical literature?
• RQ2 How does a deep learning model with transfer learning compare to rule-based and non-transfer learning deep learning baselines in claim extraction?
• RQ3 What is the impact of pre-trained word embeddings (e.g., GloVe vs. PubMed-specific) on claim detection performance?
• RQ4 To what extent can a model trained on structured abstracts generalize to claim detection in full scientific abstracts?
• RQ5 Can the proposed framework be extended to other scientific domains beyond biomedicine?

Key findings

• The proposed transfer learning-based model achieves a 14.1 percentage point increase in F1-score compared to a baseline model without transfer learning.
• The model outperforms a rule-based method by 47% in F1-score, demonstrating the superiority of deep learning with transfer learning.
• Fine-tuning the Bi-LSTM-CRF model on the new claim-annotated dataset significantly improves performance, indicating effective knowledge transfer.
• The use of GloVe word embeddings (300-dimensional) yields better performance than PubMed-specific word vectors (200-dimensional), likely due to larger corpus size and better co-occurrence modeling.
• The model shows a slight bias toward labeling later sentences in abstracts as claims, which is common in biomedical papers but may not generalize to other domains.
• The authors release a public dataset, code, and a web-based tool for claim and discourse prediction, enabling community extension and reuse.