[Paper Review] SatLM: Satisfiability-Aided Language Models Using Declarative Prompting

Xi Ye, Qiaochu Chen+2 more|arXiv (Cornell University)|May 16, 2023

Topic Modeling4 citations

TL;DR

SatLM introduces a satisfiability-aided language modeling approach that uses declarative prompting to improve reasoning in large language models (LLMs). Instead of generating imperative programs, LLMs parse natural language problems into logical constraints, which are solved using an off-the-shelf SAT solver, ensuring correctness and reducing planning errors. SatLM achieves state-of-the-art performance on GSM, LSAT, BoardgameQA, and StructuredRegex, outperforming program-aided LMs by 23% on GSM-Sys and setting new SOTA on multiple benchmarks.

ABSTRACT

Prior work has combined chain-of-thought prompting in large language models (LLMs) with programmatic representations to perform effective and transparent reasoning. While such an approach works well for tasks that only require forward reasoning (e.g., straightforward arithmetic), it is less effective for constraint solving problems that require more sophisticated planning and search. In this paper, we propose a new satisfiability-aided language modeling (SatLM) approach for improving the reasoning capabilities of LLMs. We use an LLM to generate a declarative task specification rather than an imperative program and leverage an off-the-shelf automated theorem prover to derive the final answer. This approach has two key advantages. The declarative specification is closer to the problem description than the reasoning steps are, so the LLM can parse it out of the description more accurately. Furthermore, by offloading the actual reasoning task to an automated theorem prover, our approach can guarantee the correctness of the answer with respect to the parsed specification and avoid planning errors in the solving process. We evaluate SATLM on 8 different datasets and show that it consistently outperforms program-aided LMs in the imperative paradigm. In particular, SATLM outperforms program-aided LMs by 23% on a challenging subset of the GSM arithmetic reasoning dataset; SATLM also achieves a new SoTA on LSAT and BoardgameQA, surpassing previous models that are trained on the respective training sets.

Motivation & Objective

• To address the limitations of chain-of-thought and program-aided prompting in complex reasoning tasks requiring sophisticated planning.
• To reduce both execution and planning errors in LLM-based reasoning by decoupling specification from execution.
• To improve reasoning accuracy by leveraging automated theorem provers to verify solutions against declarative problem specifications.
• To enable selective prediction by detecting unsatisfiable or ambiguous problem specifications.
• To achieve state-of-the-art performance on diverse reasoning benchmarks, including arithmetic, logical, and symbolic reasoning tasks.

Proposed method

• LLMs are prompted to convert natural language problem descriptions into a set of logical constraints (declarative specification), rather than imperative reasoning steps.
• The generated logical formulas are fed into a SAT solver (including SMT and first-order theorem provers) to determine the solution.
• The SAT solver ensures correctness by verifying that the solution satisfies all constraints, eliminating execution errors.
• The approach separates the LLM's role in parsing from the solver's role in planning and executing reasoning, reducing reliance on error-prone chain-of-thought generation.
• Few-shot prompting with self-consistency decoding is used to improve zero-shot generalization and robustness.
• The system can abstain from answering when the specification is unsatisfiable or ambiguous, improving reliability in selective prediction settings.

Figure 1 : Illustration of our Satisfiability-aided Language Modeling approach (right). We first parse an NL input into a declarative task specification (a set of logical constraints) using prompting (Section 3.1 ), then use a SAT solver to solve the problem (Section 3.2 ). The chain-of-thought stra

Experimental results

Research questions

• RQ1 Can declarative prompting combined with SAT solving improve reasoning performance in LLMs compared to imperative program-aided prompting?
• RQ2 Does offloading reasoning planning to a SAT solver reduce planning and execution errors in complex reasoning tasks?
• RQ3 Can the combination of LLMs and SAT solvers achieve state-of-the-art results on diverse reasoning benchmarks, including arithmetic and logical reasoning?
• RQ4 How does the approach perform in selective prediction settings where uncertain or ambiguous problems are rejected?
• RQ5 Does requiring LLMs to generate declarative specifications improve the quality of reasoning compared to standard chain-of-thought prompting?

Key findings

• SatLM outperforms program-aided LMs by 23% on the challenging GSM-Sys subset of the GSM dataset, demonstrating significant gains in arithmetic reasoning.
• SatLM achieves a new state-of-the-art (SOTA) performance of 84.8% on the GSM dataset using few-shot prompting and self-consistency decoding, matching models fine-tuned on the full training set.
• SatLM sets a new SOTA on LSAT (Zhong et al., 2022), outperforming previously trained models on the same benchmark.
• SatLM achieves SOTA performance on BoardgameQA (Kazemi et al., 2023) and StructuredRegex (Ye et al., 2020), indicating broad applicability across reasoning tasks.
• The approach reduces planning errors common in program-aided LMs, such as incorrect equation manipulation, by relying on the SAT solver for logical consistency.
• SatLM enables reliable selective prediction by detecting unsatisfiable or ambiguous problem specifications, enhancing accuracy in uncertain cases.

Figure 2 : Exemplar specifications for arithmetic reasoning problems generated by different approaches. CoT makes errors when parsing an equation; ProgLM produces an incorrect reasoning chain (both errors are highlighted in red ). By only using the LLMs to generate declarative specifications and rel