[Paper Review] FOLIO: Natural Language Reasoning with First-Order Logic

Simeng Han, Hailey Schoelkopf+33 more|arXiv (Cornell University)|Sep 2, 2022

Topic Modeling31 citations

TL;DR

FOLIO provides a human-annotated, open-domain dataset for natural-language reasoning with explicit first-order logic annotations, and benchmarks LMs on truly deductive reasoning and NL-FOL translation.

ABSTRACT

Large language models (LLMs) have achieved remarkable performance on a variety of natural language understanding tasks. However, existing benchmarks are inadequate in measuring the complex logical reasoning capabilities of a model. We present FOLIO, a human-annotated, logically complex and diverse dataset for reasoning in natural language (NL), equipped with first-order logic (FOL) annotations. FOLIO consists of 1,430 examples (unique conclusions), each paired with one of 487 sets of premises used to deductively reason for the validity of each conclusion. The logical correctness of the premises and conclusions is ensured by their FOL annotations, which are automatically verified by an FOL inference engine. In addition to the main NL reasoning task, NL-FOL pairs in FOLIO constitute a new NL-FOL translation dataset. Our experiments on FOLIO systematically evaluate the FOL reasoning ability of supervised fine-tuning on medium-sized language models. For both NL reasoning and NL-FOL translation, we benchmark multiple state-of-the-art language models. Our results show that a subset of FOLIO presents a challenge for one of the most capable {Large Language Model (LLM)} publicly available, GPT-4.

Motivation & Objective

• Create a large, expert-written NL reasoning dataset annotated with parallel first-order logic (FOL) formulas.
• Ensure logical validity of premises and conclusions via an FOL inference engine.
• Provide NL-FOL translation task and evaluation metrics for translating NL stories to FOL.
• Benchmark medium-sized models and large LLMs on NL reasoning and NL-FOL translation.
• Offer dataset and code to enable future evaluation of first-order logic reasoning by LMs.

Proposed method

• Collect and annotate 1,435 conclusions across 487 premise sets with real-world knowledge and expert authorship.
• Annotate parallel FOL formulas for premises and conclusions and verify them with a custom FOL inference engine.
• Align NL stories to FOL via NL-FOL translation task and define evaluation metrics for syntax, ASTs, predicates, and execution.
• Evaluate neural models with supervised fine-tuning (BERT/RoBERTa) and few-shot prompting (GPT-3, Codex, OPT, GPT-NeoX) on NL reasoning.
• Evaluate NL-FOL translation using syntactic validity, syntactic exact match, AST match, predicate fuzzy match, and execution accuracy.

Experimental results

Research questions

• RQ1 How difficult is first-order logic reasoning when grounded in natural language for current LMs?
• RQ2 To what extent can standard fine-tuning and few-shot prompting infer conclusions from NL premises under FOL reasoning?
• RQ3 How well can large language models translate NL reasoning stories into correct FOL representations?
• RQ4 What are the limitations of LLMs in handling false or unknown conclusions in FOL reasoning?
• RQ5 What insights does NL-FOL translation provide about aligning NL with formal logical representations?

Key findings

• RoBERTa-large achieves 62.11% accuracy under full supervision on NL reasoning, higher than BERT variants.
• Few-shot prompting with GPT-3 or Codex yields limited gains, with GPT-3 davinci reaching 51.10% and Codex davinci 56.04% in NL reasoning cases.
• GPT-3 davinci attains 43.44% on the best published GPT-3 variant for NL reasoning, while Codex davinci reaches 56.04% in few-shot prompts.
• NL-FOL translation metrics show syntactic validity around 90% but low exact-match and AST alignment, indicating translation quality lags behind raw NL reasoning performance.
• Models perform notably better on True conclusions than False or Unknown, with False/Unknown achieving around 54.41% (RoBERTa) or 36.91% (8-shot prompts) respectively.
• Longer reasoning chains (5–8 premises) pose a greater challenge for state-of-the-art LMs compared to shorter ones.