[Paper Review] Inductive or Deductive? Rethinking the Fundamental Reasoning Abilities of LLMs

Kewei Cheng, Jingfeng Yang+10 more|arXiv (Cornell University)|Jul 31, 2024

Law, AI, and Intellectual Property4 citations

TL;DR

This paper introduces SolverLearner, a novel framework that isolates and evaluates the pure inductive reasoning abilities of Large Language Models (LLMs) by training them to infer input-output mapping functions from in-context examples. Despite near-perfect inductive performance (ACC ≈ 1), LLMs exhibit significant weaknesses in deductive reasoning, especially in counterfactual reasoning tasks, revealing a critical asymmetry in their reasoning capabilities.

ABSTRACT

Reasoning encompasses two typical types: deductive reasoning and inductive reasoning. Despite extensive research into the reasoning capabilities of Large Language Models (LLMs), most studies have failed to rigorously differentiate between inductive and deductive reasoning, leading to a blending of the two. This raises an essential question: In LLM reasoning, which poses a greater challenge - deductive or inductive reasoning? While the deductive reasoning capabilities of LLMs, (i.e. their capacity to follow instructions in reasoning tasks), have received considerable attention, their abilities in true inductive reasoning remain largely unexplored. To investigate into the true inductive reasoning capabilities of LLMs, we propose a novel framework, SolverLearner. This framework enables LLMs to learn the underlying function (i.e., $y = f_w(x)$), that maps input data points $(x)$ to their corresponding output values $(y)$, using only in-context examples. By focusing on inductive reasoning and separating it from LLM-based deductive reasoning, we can isolate and investigate inductive reasoning of LLMs in its pure form via SolverLearner. Our observations reveal that LLMs demonstrate remarkable inductive reasoning capabilities through SolverLearner, achieving near-perfect performance with ACC of 1 in most cases. Surprisingly, despite their strong inductive reasoning abilities, LLMs tend to relatively lack deductive reasoning capabilities, particularly in tasks involving ``counterfactual'' reasoning.

Motivation & Objective

• To address the lack of clear differentiation between inductive and deductive reasoning in LLM research.
• To investigate which reasoning type—inductive or deductive—poses a greater challenge for LLMs.
• To develop a framework that isolates pure inductive reasoning from LLM-based deductive reasoning.
• To evaluate the true inductive reasoning capacity of LLMs using a consistent task across varying reasoning foci.
• To reveal the asymmetry in LLM reasoning abilities, particularly in counterfactual reasoning scenarios.

Proposed method

• Propose SolverLearner, a framework that enables LLMs to learn the underlying function $ y = f_w(x) $ mapping inputs to outputs using only in-context examples.
• Design comparative experiments using a consistent task (e.g., arithmetic) across four methods, transitioning from pure deductive to pure inductive reasoning.
• Use Input-Output (IO) prompting as a baseline for inductive reasoning, but identify its limitation in disentangling inductive and deductive reasoning.
• Isolate inductive reasoning by removing explicit function definitions and requiring the model to infer the mapping from few-shot examples.
• Evaluate performance using accuracy (ACC) across diverse reasoning tasks, including base-9 and base-10 arithmetic.
• Compare results across LLMs (e.g., GPT-3.5 vs. GPT-4) to assess the impact of model scale on inductive reasoning.

Experimental results

Research questions

• RQ1 Does LLMs’ strong performance in few-shot learning reflect true inductive reasoning or a blend with deductive reasoning?
• RQ2 How does the performance of LLMs in pure inductive reasoning compare to their performance in deductive reasoning?
• RQ3 What is the role of model scale in determining inductive reasoning capability?
• RQ4 Why do LLMs struggle with counterfactual reasoning tasks despite strong inductive abilities?
• RQ5 Can a framework like SolverLearner effectively isolate and evaluate pure inductive reasoning in LLMs?

Key findings

• LLMs achieve near-perfect inductive reasoning performance with an accuracy (ACC) of 1 in most tasks when evaluated via the SolverLearner framework.
• Despite strong inductive reasoning, LLMs exhibit notably weaker deductive reasoning capabilities, especially in counterfactual reasoning tasks such as base-9 arithmetic.
• GPT-4 demonstrates consistent inductive reasoning performance (ACC = 1) across all tasks, while GPT-3.5 fails to learn correct mappings in several cases, indicating model-scale dependency.
• The framework reveals that inductive reasoning is not inherently limited by the number of few-shot examples when the function is uniquely determinable (e.g., linear functions from two points).
• The study identifies a critical asymmetry: LLMs are highly capable of inductive reasoning but struggle with deductive reasoning in non-standard or counterfactual scenarios.
• The results suggest that current prompting techniques like Chain-of-Thought (COT) may not bridge the gap in deductive reasoning, as they are primarily designed to enhance deductive performance rather than assess inductive capacity.