[Paper Review] SciFact-Open: Towards open-domain scientific claim verification

David Wadden, Kyle Lo+5 more|arXiv (Cornell University)|Oct 25, 2022

Topic Modeling4 citations

TL;DR

This paper introduces SciFact-Open, a new open-domain test collection for scientific claim verification comprising 279 claims verified against 500K research abstracts. By pooling predictions from four state-of-the-art models, it enables evaluation of generalization in real-world settings, revealing that existing models suffer at least a 15 F1-point performance drop, highlighting critical gaps in robustness and generalization.

ABSTRACT

While research on scientific claim verification has led to the development of powerful systems that appear to approach human performance, these approaches have yet to be tested in a realistic setting against large corpora of scientific literature. Moving to this open-domain evaluation setting, however, poses unique challenges; in particular, it is infeasible to exhaustively annotate all evidence documents. In this work, we present SciFact-Open, a new test collection designed to evaluate the performance of scientific claim verification systems on a corpus of 500K research abstracts. Drawing upon pooling techniques from information retrieval, we collect evidence for scientific claims by pooling and annotating the top predictions of four state-of-the-art scientific claim verification models. We find that systems developed on smaller corpora struggle to generalize to SciFact-Open, exhibiting performance drops of at least 15 F1. In addition, analysis of the evidence in SciFact-Open reveals interesting phenomena likely to appear when claim verification systems are deployed in practice, e.g., cases where the evidence supports only a special case of the claim. Our dataset is available at https://github.com/dwadden/scifact-open.

Motivation & Objective

• To evaluate scientific claim verification systems in a realistic, open-domain setting with large-scale scientific literature.
• To address the infeasibility of exhaustive evidence annotation in large corpora by adapting pooling techniques from information retrieval.
• To identify and characterize real-world phenomena—such as conflicting or mismatched evidence—that challenge current claim verification systems.
• To provide a benchmark that reveals the limitations of existing models when scaled beyond small, curated datasets.
• To support future research on model generalization, claim revision, and evidence summarization in scientific verification.

Proposed method

• Construct a test collection by pooling the top-k most confident predictions from four state-of-the-art scientific claim verification models for each claim.
• Use a single retrieval system to identify candidate abstracts from a 500K-sentence corpus of scientific research abstracts.
• Apply a TREC-style pooling strategy to select evidence candidates, reducing annotation burden while maintaining coverage.
• Annotate claim/abstract pairs (CAPs) for support, refutation, or not enough info (NEI) labels using human annotators.
• Evaluate model performance using F1 score on abstract-level labels, excluding rationale annotations for efficiency.
• Perform sensitivity analysis on pooling depth and number of systems to validate the robustness of the test collection.

Experimental results

Research questions

• RQ1 How do state-of-the-art claim verification models perform when evaluated on a large-scale, open-domain corpus of 500K scientific abstracts?
• RQ2 What real-world phenomena—such as conflicting or mismatched evidence—emerge when verifying claims in open-domain settings?
• RQ3 To what extent do models trained on small, curated datasets like SciFact generalize to broader scientific literature?
• RQ4 How sensitive is the test collection to the number of models and depth of pooling used in evidence collection?
• RQ5 Can the dataset reveal meaningful performance differences between models that are not apparent in smaller benchmarks?

Key findings

• State-of-the-art models trained on the original SciFact dataset suffer a performance drop of at least 15 F1 points when evaluated on SciFact-Open, indicating poor generalization to open-domain settings.
• The dataset reveals significant real-world phenomena, including conflicting evidence where multiple abstracts support or refute different aspects of a claim.
• Evidence in SciFact-Open often does not match the specificity of the claim, leading to cases where evidence supports only a special case of the claim.
• The pooling strategy effectively captures relevant evidence, with a single retrieval system recovering the majority of relevant abstracts, as confirmed by ablation studies.
• The dataset contains claims with highly imbalanced evidence counts, suggesting that some claims are far more studied than others, which may affect model evaluation.
• The authors release 91 claim revisions derived from evidence, enabling future research into fine-grained claim revision and improved user interpretability.