[Paper Review] TwoWingOS: A Two-Wing Optimization Strategy for Evidential Claim Verification

Wenpeng Yin, Dan Roth|arXiv (Cornell University)|Aug 10, 2018

Topic Modeling20 references4 citations

TL;DR

This paper proposes TwoWingOS, a joint optimization framework that simultaneously identifies supporting evidence and verifies claim truth value in a single end-to-end model, outperforming pipeline methods by achieving a 30% F1 improvement in evidence identification and a 23% accuracy gain in claim verification on the FEVER benchmark.

ABSTRACT

Determining whether a given claim is supported by evidence is a fundamental NLP problem that is best modeled as Textual Entailment. However, given a large collection of text, finding evidence that could support or refute a given claim is a challenge in itself, amplified by the fact that different evidence might be needed to support or refute a claim. Nevertheless, most prior work decouples evidence identification from determining the truth value of the claim given the evidence. We propose to consider these two aspects jointly. We develop TwoWingOS (two-wing optimization strategy), a system that, while identifying appropriate evidence for a claim, also determines whether or not the claim is supported by the evidence. Given the claim, TwoWingOS attempts to identify a subset of the evidence candidates; given the predicted evidence, it then attempts to determine the truth value of the corresponding claim. We treat this challenge as coupled optimization problems, training a joint model for it. TwoWingOS offers two advantages: (i) Unlike pipeline systems, it facilitates flexible-size evidence set, and (ii) Joint training improves both the claim entailment and the evidence identification. Experiments on a benchmark dataset show state-of-the-art performance. Code: https://github.com/yinwenpeng/FEVER

Motivation & Objective

• To address the limitations of pipeline systems that decouple evidence identification from claim verification, which often leads to suboptimal performance.
• To develop a joint learning framework that enables mutual enhancement between evidence retrieval and claim verification.
• To handle flexible-sized evidence sets rather than relying on single-sentence premises.
• To improve both precision and recall in evidence identification while maintaining high claim verification accuracy.
• To demonstrate the effectiveness of joint optimization through empirical evaluation on the FEVER benchmark.

Proposed method

• TwoWingOS formulates claim verification and evidence identification as a coupled optimization problem using a shared representation of the claim.
• The model uses a binary vector to select a subset of candidate evidence sentences from a corpus, and an n-valued vector to predict the entailment label (entailment, contradiction, or no label).
• A shared neural network encodes the claim and candidate evidence, enabling joint training of both modules.
• The framework treats evidence identification and claim verification as interdependent tasks, allowing knowledge transfer between them during training.
• The model is trained end-to-end on the FEVER dataset using a joint loss function that optimizes both objectives simultaneously.
• The system employs attention mechanisms and contextual representations to capture dependencies across multiple evidence sentences.

Experimental results

Research questions

• RQ1 Can joint optimization of evidence identification and claim verification improve performance compared to pipeline approaches?
• RQ2 Does the ability to select variable-sized evidence sets enhance verification accuracy?
• RQ3 How does shared representation learning between evidence retrieval and entailment prediction affect model generalization?
• RQ4 To what extent do entity mentions and multi-sentence evidence composition improve verification outcomes?
• RQ5 How does the joint training scheme compare to separate training in terms of precision, recall, and F1 on evidence identification?

Key findings

• TwoWingOS achieves a 30% relative improvement in F1 score for evidence identification compared to the pipeline baseline on the FEVER benchmark.
• The model improves claim verification accuracy by approximately 23% over the baseline pipeline system.
• The system maintains high precision (over 80%) across different evidence set sizes, even as recall decreases with larger gold evidence sets.
• The joint training scheme significantly outperforms pipeline architectures, especially in handling complex claims requiring multi-sentence evidence.
• Error analysis reveals that the model struggles with out-of-vocabulary terms and requires background knowledge for certain refutations, indicating room for improvement in semantic generalization.
• The model performs robustly on claims that can be verified by a single sentence, as 83.18% of claims in FEVER are correctly entailed by just one gold evidence sentence.