[Paper Review] Logic2Text: High-Fidelity Natural Language Generation from Logical Forms

Zhiyu Chen, Wenhu Chen+5 more|arXiv (Cornell University)|Apr 30, 2020

Topic Modeling33 references4 citations

TL;DR

This paper proposes Logic2Text, a large-scale dataset of 10,753 (logical form, natural language description) pairs for high-fidelity natural language generation from multi-row tables. By providing logical forms as intermediate meaning representations, the approach enables controllable, factually correct generation, with GPT-2 achieving 82.4% factual correctness in human evaluation—demonstrating the critical role of logical forms in improving fidelity.

ABSTRACT

Previous works on Natural Language Generation (NLG) from structured data have primarily focused on surface-level descriptions of record sequences. However, for complex structured data, e.g., multi-row tables, it is often desirable for an NLG system to describe interesting facts from logical inferences across records. If only provided with the table, it is hard for existing models to produce controllable and high-fidelity logical generations. In this work, we formulate logical level NLG as generation from logical forms in order to obtain controllable, high-fidelity, and faithful generations. We present a new large-scale dataset, extsc{Logic2Text}, with 10,753 descriptions involving common logic types paired with the underlying logical forms. The logical forms show diversified graph structure of free schema, which poses great challenges on the model's ability to understand the semantics. We experiment on (1) Fully-supervised training with the full datasets, and (2) Few-shot setting, provided with hundreds of paired examples; We compare several popular generation models and analyze their performances. We hope our dataset can encourage research towards building an advanced NLG system capable of natural, faithful, and human-like generation. The dataset and code are available at https://github.com/czyssrs/Logic2Text.

Motivation & Objective

• To address the limitations of existing NLG systems that generate only surface-level descriptions from structured data, especially for complex, multi-row tables.
• To improve generation fidelity and controllability by introducing logical forms as intermediate representations for higher-level reasoning across table records.
• To provide a high-quality, large-scale dataset with diverse, free-schema logical forms and human-annotated descriptions for training and evaluation.
• To evaluate the impact of logical forms on generation quality through fully-supervised and few-shot learning settings.
• To encourage research in advanced NLG systems capable of faithful, human-like, and logically correct text generation.

Proposed method

• The paper introduces a new dataset, Logic2Text, consisting of 5,600 open-domain tables and 10,800 (logical form, description) pairs annotated by human experts.
• Logical forms are represented as Python-like program structures with diverse graph-like semantics, encoding operations such as count, superlative, comparative, aggregation, majority, unique, and ordinal.
• The task is formulated as text generation conditioned on both the table and its corresponding logical form, decoupling reasoning from language realization.
• Baseline models include pointer-generator, graph2seq, transformer, and GPT-2, evaluated under fully-supervised and few-shot learning settings.
• Human evaluation is conducted by experts to assess semantic correctness and fluency, with 200 examples per method examined for fidelity.
• Ablation studies are performed to analyze the contribution of table caption, header, and content in the input to generation quality.

Experimental results

Research questions

• RQ1 How does providing logical forms as input affect the factual correctness and fluency of NLG outputs compared to generating from tables alone?
• RQ2 To what extent can pre-trained language models like GPT-2 achieve high-fidelity generation when guided by logical forms in both fully-supervised and few-shot settings?
• RQ3 What is the contribution of different input components—table caption, header, and content—to the quality of generated descriptions?
• RQ4 How do different neural generation models compare in terms of semantic correctness and fluency when generating from logical forms?
• RQ5 Can the proposed dataset support reverse tasks such as semantic parsing (text-to-logical-form generation), and how does it improve evaluation reliability?

Key findings

• The GPT-2 model achieves 82.4% factual correctness in human evaluation when generating from logical forms, significantly outperforming prior work that achieved only 20.2% factual correctness when generating from tables alone.
• In the fully-supervised setting, GPT-2 achieves a BLEU-4 score of 31.44 and a ROUGE-L score of 53.99, demonstrating strong performance on automatic metrics.
• Even with only 1,000 training examples, the GPT-2 model achieves a ROUGE-L score of 47.67, showing strong few-shot generalization capability.
• Ablation studies show that the table caption and header provide strong contextual signals, with the caption contributing most to performance.
• Human evaluation confirms that semantic correctness is significantly improved when logical forms are used, with GPT-2 maintaining high fidelity despite occasional factual and logical errors.
• The dataset enables precise evaluation of semantic correctness due to the unique mapping between each logical form and its reference description, avoiding ambiguity in reference-based metrics.