[Paper Review] Language Models as Inductive Reasoners

Zonglin Yang, Li Dong+6 more|arXiv (Cornell University)|Dec 21, 2022

Natural Language Processing Techniques7 citations

TL;DR

This paper proposes a new paradigm for inductive reasoning that uses pretrained language models (PLMs) to infer natural language rules from natural language facts, introducing the DEER dataset of 1.2k rule-fact pairs and novel evaluation metrics. The framework, inspired by philosophical reasoning, outperforms baselines in both automatic and human evaluations, demonstrating PLMs' capability to perform general inductive reasoning in natural language without formal logic representations.

ABSTRACT

Inductive reasoning is a core component of human intelligence. In the past research of inductive reasoning within computer science, formal language is used as representations of knowledge (facts and rules, more specifically). However, formal language can cause systematic problems for inductive reasoning such as disability of handling raw input such as natural language, sensitiveness to mislabeled data, and incapacity to handle ambiguous input. To this end, we propose a new paradigm (task) for inductive reasoning, which is to induce natural language rules from natural language facts, and create a dataset termed DEER containing 1.2k rule-fact pairs for the task, where rules and facts are written in natural language. New automatic metrics are also proposed and analysed for the evaluation of this task. With DEER, we investigate a modern approach for inductive reasoning where we use natural language as representation for knowledge instead of formal language and use pretrained language models as ''reasoners''. Moreover, we provide the first and comprehensive analysis of how well pretrained language models can induce natural language rules from natural language facts. We also propose a new framework drawing insights from philosophy literature for this task, which we show in the experiment section that surpasses baselines in both automatic and human evaluations. We discuss about our future perspectives for inductive reasoning in Section 7. Dataset and code are available at https://github.com/ZonglinY/Inductive_Reasoning.

Motivation & Objective

• To address systematic limitations of formal language and symbolic reasoning in inductive reasoning, such as sensitivity to mislabeled data and inability to process raw natural language.
• To propose a new paradigm where both facts and rules are represented in natural language, enabling more human-like, flexible inductive reasoning.
• To create a new dataset, DEER, with 1.2k high-quality rule-fact pairs in natural language to support this new paradigm.
• To develop new automatic evaluation metrics tailored for natural language inductive reasoning tasks.
• To comprehensively evaluate the ability of pretrained language models to induce general, natural language rules from natural language observations.

Proposed method

• Propose a new inductive reasoning task: inducing general natural language rules from natural language facts, avoiding formal logic representations.
• Construct the DEER dataset containing 1.2k rule-fact pairs drawn from commonsense and scientific knowledge, with human-annotated rules generalizing over observed facts.
• Design new automatic evaluation metrics—such as rule generalization and factual consistency—specifically for natural language inductive reasoning.
• Develop a novel reasoning framework inspired by philosophical literature on inductive reasoning, incorporating principles like hypothesis generalization and evidence-based inference.
• Fine-tune and evaluate state-of-the-art pretrained language models (PLMs) as 'reasoners' to generate rules from input facts, using few-shot prompting and chain-of-thought techniques.
• Conduct both automatic and human evaluations to compare the proposed framework against strong baselines, including zero-shot and few-shot prompting methods.

Experimental results

Research questions

• RQ1 Can pretrained language models effectively induce general, natural language rules from natural language facts without relying on formal logic representations?
• RQ2 How do automatic metrics designed for natural language inductive reasoning correlate with human judgments of rule quality and generalization?
• RQ3 To what extent does a philosophy-inspired reasoning framework improve the quality and generalization of induced rules compared to standard prompting methods?
• RQ4 How does the proposed paradigm mitigate systematic issues of symbolic ILP systems, such as sensitivity to label noise and inability to handle ambiguous or raw natural language inputs?
• RQ5 What are the key challenges in scaling this approach, particularly in generating novel, non-trivial scientific hypotheses that are not already known in the literature?

Key findings

• The proposed framework, informed by philosophical principles of inductive reasoning, significantly outperforms strong baselines in both automatic and human evaluations, demonstrating superior rule generalization and factual consistency.
• The DEER dataset successfully captures diverse, high-quality rule-fact relationships across commonsense and scientific domains, with rules generalizing over multiple natural language facts.
• New automatic evaluation metrics show strong correlation with human judgments, validating their utility in assessing rule quality and inductive strength.
• Pretrained language models exhibit strong capability in generating plausible, general natural language rules from raw text facts, suggesting potential for use as 'copilots' in scientific hypothesis generation.
• The study reveals that rule quality is highly sensitive to fact diversity and semantic coherence, with poorly structured or redundant facts leading to weaker induced rules.
• The framework shows promise for scientific hypothesis discovery, with subsequent work (Yang et al., 2023b) demonstrating LLMs can generate novel, valid hypotheses not found in existing literature.