[Paper Review] SemEval-2023 Task 7: Multi-Evidence Natural Language Inference for Clinical Trial Data

Maël Jullien, Marco Valentino+4 more|arXiv (Cornell University)|May 4, 2023

Topic Modeling4 citations

TL;DR

This paper presents SemEval-2023 Task 7, a benchmark for multi-evidence natural language inference (NLI) and evidence selection on clinical trial data, requiring complex multi-hop and numerical reasoning. The highest-performing system achieved an F1 score of 0.856 on the entailment task and 0.853 on evidence selection, with model size having a stronger impact than biomedical pre-training.

ABSTRACT

This paper describes the results of SemEval 2023 task 7 -- Multi-Evidence Natural Language Inference for Clinical Trial Data (NLI4CT) -- consisting of 2 tasks, a Natural Language Inference (NLI) task, and an evidence selection task on clinical trial data. The proposed challenges require multi-hop biomedical and numerical reasoning, which are of significant importance to the development of systems capable of large-scale interpretation and retrieval of medical evidence, to provide personalized evidence-based care. Task 1, the entailment task, received 643 submissions from 40 participants, and Task 2, the evidence selection task, received 364 submissions from 23 participants. The tasks are challenging, with the majority of submitted systems failing to significantly outperform the majority class baseline on the entailment task, and we observe significantly better performance on the evidence selection task than on the entailment task. Increasing the number of model parameters leads to a direct increase in performance, far more significant than the effect of biomedical pre-training. Future works could explore the limitations of large models for generalization and numerical inference, and investigate methods to augment clinical datasets to allow for more rigorous testing and to facilitate fine-tuning. We envisage that the dataset, models, and results of this task will be useful to the biomedical NLI and evidence retrieval communities. The dataset, competition leaderboard, and website are publicly available.

Motivation & Objective

• To develop and evaluate systems for multi-evidence natural language inference on clinical trial reports, requiring multi-hop and numerical reasoning.
• To assess the performance of NLP models in retrieving supporting evidence from clinical trial documents to justify entailment predictions.
• To investigate the impact of model size and biomedical pre-training on reasoning in clinical NLP tasks.
• To provide a publicly available dataset and benchmark to advance evidence-based medicine and clinical decision support systems.

Proposed method

• The task uses a dataset of 2,000+ clinical trial reports (CTRs) annotated with natural language statements and their corresponding entailment labels (entailment, contradiction, neutral).
• Participants were required to perform two tasks: (1) predict the entailment relation between a statement and a CTR, and (2) extract supporting evidence from the CTR to justify the label.
• The NLI4CT dataset was constructed using a negative-rewriting strategy, generating one entailment and one contradictory statement per CTR premise.
• Systems were evaluated using F1 scores on both the entailment and evidence selection tasks, with results aggregated on a public leaderboard.
• The study analyzed system behavior, including the order of evidence extraction versus inference, and assessed for confirmation bias and statistical artifacts.
• Statistical robustness checks were performed to rule out superficial patterns or token distribution biases inflating performance.

Figure 1: We propose two tasks for reasoning on clinical trial data expressed in natural language. Firstly, to predict the entailment of a Statement and a CTR premise, and secondly, to extract evidence to support the label.

Experimental results

Research questions

• RQ1 How well do state-of-the-art NLP models perform on multi-evidence NLI and evidence selection in clinical trial data?
• RQ2 What is the relative impact of model size versus biomedical pre-training on performance in clinical NLI tasks?
• RQ3 Does the order of evidence selection versus entailment classification affect system performance?
• RQ4 To what extent do statistical artifacts or superficial patterns in statements influence model predictions?
• RQ5 Can systems generalize effectively to clinical NLI when relying on multi-hop and numerical reasoning?

Key findings

• The highest-performing system, @THiFLY, achieved an F1 score of 0.856 on the entailment task and 0.853 on the evidence selection task.
• Most submitted systems failed to significantly outperform the majority class baseline on the entailment task, indicating high difficulty.
• Performance on the evidence selection task was consistently higher than on the entailment task, suggesting it is less challenging.
• Increasing model size had a stronger positive effect on performance than biomedical pre-training, which had a comparatively minor impact.
• Systems that performed evidence selection before inference achieved significantly better F1 and precision scores, suggesting they avoided confirmation bias.
• The dataset is robust to statistical artifacts, as no significant token distribution or length biases were found between entailment classes in the test set.

Figure 2: Graph comparing system F1 scores across the entailment task and the evidence selection task