[Paper Review] SURFACE: Semantically Rich Fact Validation with Explanations

Ankur Padia, Francis Ferraro+1 more|arXiv (Cornell University)|Oct 31, 2018

Topic Modeling4 citations

TL;DR

SURFACE is a multi-task neural model that uses FrameNet-structured semantic frames to jointly retrieve relevant Wikipedia sentences and classify claims as supported, refuted, or unsure. It achieves a 90% relative improvement in evidence retrieval and a ~70% relative improvement in classification accuracy over state-of-the-art baselines on the FEVER dataset.

ABSTRACT

Judging the veracity of a sentence making one or more claims is an important and challenging problem with many dimensions. The recent FEVER task asked participants to classify input sentences as either SUPPORTED, REFUTED or NotEnoughInfo using Wikipedia as a source of true facts. SURFACE does this task and explains its decision through a selection of sentences from the trusted source. Our multi-task neural approach uses semantic lexical frames from FrameNet to jointly (i) find relevant evidential sentences in the trusted source and (ii) use them to classify the input sentence's veracity. An evaluation of our efficient three-parameter model on the FEVER dataset showed an improvement of 90% over the state-of-the-art baseline on retrieving relevant sentences and a 70% relative improvement in classification.

Motivation & Objective

• To address the challenge of automated fact validation in natural language claims using a trusted source like Wikipedia.
• To improve the interpretability of claim classification by providing semantically grounded, extractable evidence sentences from Wikipedia.
• To jointly optimize evidence retrieval and claim classification using a multi-task learning framework with semantic frames.
• To reduce reliance on heuristic retrieval methods like TF-IDF by leveraging rich semantic representations from FrameNet.
• To demonstrate that incorporating sentence utility prediction enhances both retrieval and classification performance in a parameter-efficient manner.

Proposed method

• Uses FrameNet lexical frames to extract semantic representations from both claims and Wikipedia sentences, enabling semantic matching for evidence retrieval.
• Employs a multi-task learning architecture where the model jointly predicts evidence relevance and sentence utility (importance) for each candidate sentence.
• Applies a Gumbel-Softmax reparameterization trick to differentially select high-utility evidence sentences during training, improving end-to-end optimization.
• Utilizes a three-parameter model family: Verifier, Verifier + MultiTask, and Verifier + MultiTask with Gumbel utility, progressively increasing complexity and performance.
• Performs frame-based sentence matching to identify candidate evidence sentences from Wikipedia, followed by utility-based filtering and classification.
• Employs a parameter-efficient design with shared sentence encoding and task-specific heads for retrieval and classification, enabling efficient training and inference.

Experimental results

Research questions

• RQ1 Can semantic frames from FrameNet improve the retrieval of relevant evidence sentences for claim verification compared to traditional methods like TF-IDF?
• RQ2 Does jointly learning evidence retrieval and claim classification via multi-task learning lead to better performance than separate optimization?
• RQ3 How does predicting sentence utility (importance) for evidence affect the final classification accuracy and retrieval F1 score?
• RQ4 To what extent does post-processing (e.g., removing low-utility sentences) impact retrieval recall and overall FEVER score?
• RQ5 Can a parameter-efficient model with minimal architectural complexity outperform state-of-the-art baselines on the FEVER benchmark?

Key findings

• SURFACE achieved a 90% relative improvement in evidence retrieval performance over the state-of-the-art baseline on the FEVER dataset.
• The model's best configuration (Verifier + MultiTask) achieved a FEVER score of 0.3452 on the development set, representing a ~70% relative improvement in classification over the baseline.
• The use of multi-task learning significantly boosted classification performance, with the Verifier + MultiTask model outperforming both single-task and Gumbel-regularized variants.
• Post-processing to remove low-utility evidence sentences (u=10) reduced recall and F1, leading to a drop in FEVER score from 0.3452 to 0.2554, indicating that such filtering harms overall performance.
• The Gumbel-Softmax utility prediction did not improve performance compared to raw utility scores, though increasing training epochs to 100 helped mitigate this gap.
• Increasing the number of frame-based sentences retrieved (K) improved performance up to a point, but performance degraded when too many sentences were included, suggesting an optimal retrieval window.