[Paper Review] Resolving the Scope of Speculation and Negation using Transformer-Based Architectures

Benita Kathleen Britto, Aditya Khandelwal|arXiv (Cornell University)|Jan 9, 2020

Topic Modeling21 references11 citations

TL;DR

This paper proposes using BERT, XLNet, and RoBERTa for speculation and negation scope resolution, achieving state-of-the-art results on BioScope and SFU datasets. Joint training across datasets improves performance, with XLNet outperforming BERT and RoBERTa consistently, especially on sequence labeling tasks like scope resolution.

ABSTRACT

Speculation is a naturally occurring phenomena in textual data, forming an integral component of many systems, especially in the biomedical information retrieval domain. Previous work addressing cue detection and scope resolution (the two subtasks of speculation detection) have ranged from rule-based systems to deep learning-based approaches. In this paper, we apply three popular transformer-based architectures, BERT, XLNet and RoBERTa to this task, on two publicly available datasets, BioScope Corpus and SFU Review Corpus, reporting substantial improvements over previously reported results (by at least 0.29 F1 points on cue detection and 4.27 F1 points on scope resolution). We also experiment with joint training of the model on multiple datasets, which outperforms the single dataset training approach by a good margin. We observe that XLNet consistently outperforms BERT and RoBERTa, contrary to results on other benchmark datasets. To confirm this observation, we apply XLNet and RoBERTa to negation detection and scope resolution, reporting state-of-the-art results on negation scope resolution for the BioScope Corpus (increase of 3.16 F1 points on the BioScope Full Papers, 0.06 F1 points on the BioScope Abstracts) and the SFU Review Corpus (increase of 0.3 F1 points).

Motivation & Objective

• To improve speculation and negation scope resolution in biomedical text using transformer-based architectures.
• To investigate whether joint training on multiple datasets enhances model generalization and performance.
• To compare the effectiveness of BERT, XLNet, and RoBERTa on sequence labeling tasks related to uncertainty and negation.
• To determine why XLNet outperforms RoBERTa in this context, contrary to trends on standard GLUE benchmarks.
• To establish state-of-the-art results for both speculation and negation scope resolution tasks on public datasets.

Proposed method

• Fine-tuned three transformer models—BERT, XLNet, and RoBERTa—on speculation and negation detection tasks using sequence labeling with token-level classification.
• Applied a standard 70-15-15 train-validation-test split per dataset, with 5 runs for single-dataset training and 3 runs for joint training.
• Used Hugging Face's PyTorch implementation for model training and inference on Google Colab.
• Performed joint training by merging training and validation sets across multiple datasets (BF, BA, SFU) while keeping test sets separate for evaluation.
• Employed standard tokenization and label encoding (1: Normal Cue, 2: Multiword Cue, 3: Not a Cue, 4: Pad) for sequence labeling.
• Evaluated models using F1-score for both cue detection and scope resolution, with results averaged across runs for statistical robustness.

Experimental results

Research questions

• RQ1 Can transformer-based models like BERT, XLNet, and RoBERTa achieve state-of-the-art performance on speculation and negation scope resolution tasks?
• RQ2 Does joint training on multiple datasets (e.g., BioScope Abstracts, BioScope Full Papers, SFU Review Corpus) improve model generalization and performance compared to single-dataset training?
• RQ3 Why does XLNet outperform RoBERTa on these sequence labeling tasks, despite RoBERTa's strong performance on standard NLP benchmarks?
• RQ4 To what extent do domain-specific characteristics of biomedical text influence model performance and transferability?
• RQ5 Can the same model architecture achieve SOTA results across both speculation and negation scope resolution tasks?

Key findings

• Joint training on multiple datasets improved performance across all models, with gains of up to 10.6 F1 points on speculation detection and 2.16 F1 points on scope resolution for the BioScope Full Papers dataset.
• XLNet achieved the highest F1 scores on all datasets for both speculation and negation tasks, outperforming BERT and RoBERTa consistently.
• On speculation scope resolution, XLNet achieved a 4.27 F1 point improvement over prior work on the BioScope Abstracts dataset and 8.06 F1 points on the BioScope Full Papers dataset.
• For negation scope resolution, XLNet achieved a 3.16 F1 point gain on the BioScope Full Papers dataset and 0.3 F1 point gain on the SFU Review Corpus, setting new SOTA results.
• The performance gap between XLNet and RoBERTa was more pronounced on sequence labeling tasks than on standard GLUE benchmarks, suggesting architectural differences matter more in sequence-level prediction.
• The SFU Review Corpus showed better cross-dataset generalization than the BioScope datasets, possibly due to differences in domain and sentence structure.