[Paper Review] TabFact: A Large-scale Dataset for Table-based Fact Verification

Wenhu Chen, Hongmin Wang+6 more|arXiv (Cornell University)|Sep 5, 2019

Advanced Text Analysis Techniques40 references181 citations

TL;DR

TabFact introduces a large-scale table-based fact verification dataset (118k statements, 16k Wikipedia tables) and presents two strong baseline models—Table-BERT and Latent Program Algorithm (LPA)—to handle linguistic and symbolic reasoning over semi-structured evidence.

ABSTRACT

The problem of verifying whether a textual hypothesis holds based on the given evidence, also known as fact verification, plays an important role in the study of natural language understanding and semantic representation. However, existing studies are mainly restricted to dealing with unstructured evidence (e.g., natural language sentences and documents, news, etc), while verification under structured evidence, such as tables, graphs, and databases, remains under-explored. This paper specifically aims to study the fact verification given semi-structured data as evidence. To this end, we construct a large-scale dataset called TabFact with 16k Wikipedia tables as the evidence for 118k human-annotated natural language statements, which are labeled as either ENTAILED or REFUTED. TabFact is challenging since it involves both soft linguistic reasoning and hard symbolic reasoning. To address these reasoning challenges, we design two different models: Table-BERT and Latent Program Algorithm (LPA). Table-BERT leverages the state-of-the-art pre-trained language model to encode the linearized tables and statements into continuous vectors for verification. LPA parses statements into programs and executes them against the tables to obtain the returned binary value for verification. Both methods achieve similar accuracy but still lag far behind human performance. We also perform a comprehensive analysis to demonstrate great future opportunities. The data and code of the dataset are provided in \url{https://github.com/wenhuchen/Table-Fact-Checking}.

Motivation & Objective

• Study fact verification using semi-structured evidence (tables) rather than unstructured text.
• Create a large, high-quality dataset of table-backed statements labeled ENTAILED or REFUTED.
• Develop and compare models that can perform linguistic reasoning and symbolic table reasoning.

Proposed method

• Construct TabFact from WikiTables with 16k tables and 118k human-annotated statements labeled ENTAILED or REFUTED.
• Annotate using a two-channel collection and a negative rewriting strategy to mitigate artifacts.
• Propose Table-BERT by linearizing tables and using a pre-trained language model for NLI-style verification.
• Propose Latent Program Algorithm (LPA) that performs latent program search and a discriminator to rank program hypotheses.
• Evaluate both approaches against simple and complex test splits and human performance.

Experimental results

Research questions

• RQ1 Can fact verification be effectively performed over semi-structured table evidence?
• RQ2 How do linguistic and symbolic reasoning interact in table-based verification tasks?
• RQ3 What are the strengths and limitations of neural and program-synthesis based approaches for TabFact?
• RQ4 How close can Table-BERT and LPA approach human-level accuracy on TabFact?
• RQ5 What insights emerge from error analysis and human evaluation on linking, search, and reasoning steps?

Key findings

• TabFact contains 118,275 annotated statements across 16,573 tables, with strong inter-annotator agreement (Fleiss κ = 0.75).
• Two baseline models achieve similar accuracy but lag behind human performance on both simple and complex splits.
• Table-BERT benefits from natural-language table templates and horizontal/vertical linearization, with the best variant achieving substantial gains over naive baselines.
• LPA achieves competitive results by converting statements into executable programs over tables, using a discriminator to pick consistent traces.
• Human evaluation reveals linking and program search limitations (approx. 58% correct linking, ~51% recall for true programs), highlighting spurious reasoning as a major challenge.
• Overall, both approaches demonstrate the feasibility of table-based fact verification while highlighting substantial room for improvement.