[Paper Review] From Gameplay to Symbolic Reasoning: Learning SAT Solver Heuristics in the Style of Alpha(Go) Zero

Fei Wang, Tiark Rompf|arXiv (Cornell University)|Feb 14, 2018

Machine Learning in Materials Science7 references3 citations

TL;DR

This paper proposes casting symbolic reasoning problems like Boolean Satisfiability (SAT) as games to leverage deep reinforcement learning for learning optimal decision heuristics. By modeling SAT solving as a sequential decision game and applying Alpha(Go) Zero-style reinforcement learning, the method achieves superior generalization and correctness compared to DeepQ and traditional heuristics, demonstrating a modular, efficient, and correct approach to symbolic reasoning with neural networks.

ABSTRACT

Despite the recent successes of deep neural networks in various fields such as image and speech recognition, natural language processing, and reinforcement learning, we still face big challenges in bringing the power of numeric optimization to symbolic reasoning. Researchers have proposed different avenues such as neural machine translation for proof synthesis, vectorization of symbols and expressions for representing symbolic patterns, and coupling of neural back-ends for dimensionality reduction with symbolic front-ends for decision making. However, these initial explorations are still only point solutions, and bear other shortcomings such as lack of correctness guarantees. In this paper, we present our approach of casting symbolic reasoning as games, and directly harnessing the power of deep reinforcement learning in the style of Alpha(Go) Zero on symbolic problems. Using the Boolean Satisfiability (SAT) problem as showcase, we demonstrate the feasibility of our method, and the advantages of modularity, efficiency, and correctness guarantees.

Motivation & Objective

• To address the challenge of integrating deep neural networks with symbolic reasoning, particularly in discrete, logic-based problems like SAT.
• To overcome limitations of existing approaches—such as lack of correctness guarantees, poor generalization, and reliance on point solutions—by redefining symbolic reasoning as a game.
• To leverage the decision-making power of deep reinforcement learning in a modular, correct-by-construction framework for symbolic problems.
• To evaluate whether Alpha(Go) Zero-style learning can generalize better than DeepQ in symbolic reasoning tasks, using SAT as a benchmark.
• To demonstrate that neural networks can learn high-level symbolic reasoning patterns (e.g., branching heuristics) while preserving symbolic interpretability and correctness.

Proposed method

• Representing SAT problems in Conjunctive Normal Form (CNF) as sparse adjacency matrices, where rows are clauses and columns are variables, with polarity encoded as (1,0) or (0,1).
• Using a convolutional neural network (CNN) to process the matrix representation and generate state embeddings for reinforcement learning.
• Extending the MiniSat SAT solver to function as a Gym-style environment, enabling interaction with reinforcement learning agents through reversible simulations.
• Employing Monte Carlo Tree Search (MCTS) within the Alpha(Go) Zero algorithm to guide exploration and improve policy and value network learning.
• Training two reinforcement learning algorithms—DeepQ and Alpha(Go) Zero—on the same SAT game environment, with the latter learning a policy vector (pi) and state value (v) for decision guidance.
• Using a reward signal based on the number of branching decisions to solve the problem, with lower counts indicating better performance.

Experimental results

Research questions

• RQ1 Can symbolic reasoning problems like SAT be effectively modeled as games to enable deep reinforcement learning for heuristic learning?
• RQ2 Does Alpha(Go) Zero-style reinforcement learning generalize better than DeepQ in symbolic reasoning tasks, despite both converging on training data?
• RQ3 Can a neural network trained via self-play in a game-like SAT environment learn heuristics that outperform hand-crafted, human-engineered heuristics like VSIGS?
• RQ4 To what extent does the modular design of the game-based framework allow for reuse across different symbolic reasoning problems?
• RQ5 How does the use of MCTS and value/policy networks in Alpha(Go) Zero contribute to better symbolic reasoning generalization compared to Q-value learning in DeepQ?

Key findings

• The Alpha(Go) Zero algorithm achieved near-optimal performance on the training set of 32 SAT problems and generalized well to the test set of 200 problems.
• DeepQ converged on the training set but failed to generalize to the test set, indicating overfitting to specific state-action pairs.
• Both reinforcement learning models outperformed the MiniSat solver using the VSIGS heuristic, as measured by a lower average number of branching decisions.
• The Alpha(Go) Zero model required significantly fewer average branching decisions than MiniSat, demonstrating superior heuristic quality.
• The method maintains correctness by restricting actions to only valid branching decisions, ensuring no malformed moves or incorrect solutions.
• The approach exhibits modularity, as the same neural network and game interface can be adapted to other symbolic reasoning problems with minimal changes.