[Paper Review] SATformer: Transformer-Based UNSAT Core Learning

Zhengyuan Shi, Min Li+6 more|arXiv (Cornell University)|Sep 2, 2022

Software Engineering Research4 citations

TL;DR

SATformer proposes a novel Transformer-based approach for Boolean satisfiability (SAT) solving by learning minimal unsatisfiable cores (MUCs) as supervision, using a graph neural network (GNN) for clause embeddings and a hierarchical Transformer to model multi-clause interactions. It achieves state-of-the-art performance among end-to-end learning-based SAT solvers, outperforming prior methods by focusing attention on clauses contributing to unsatisfiability.

ABSTRACT

This paper introduces SATformer, a novel Transformer-based approach for the Boolean Satisfiability (SAT) problem. Rather than solving the problem directly, SATformer approaches the problem from the opposite direction by focusing on unsatisfiability. Specifically, it models clause interactions to identify any unsatisfiable sub-problems. Using a graph neural network, we convert clauses into clause embeddings and employ a hierarchical Transformer-based model to understand clause correlation. SATformer is trained through a multi-task learning approach, using the single-bit satisfiability result and the minimal unsatisfiable core (MUC) for UNSAT problems as clause supervision. As an end-to-end learning-based satisfiability classifier, the performance of SATformer surpasses that of NeuroSAT significantly. Furthermore, we integrate the clause predictions made by SATformer into modern heuristic-based SAT solvers and validate our approach with a logic equivalence checking task. Experimental results show that our SATformer can decrease the runtime of existing solvers by an average of 21.33%.

Motivation & Objective

• To improve end-to-end learning-based SAT solvers by shifting from instance-level supervision to clause-level supervision using minimal unsatisfiable cores (MUCs).
• To model complex clause interactions beyond pairwise dependencies by introducing a hierarchical Transformer architecture.
• To enhance attention mechanisms so that the model focuses on clauses likely to form MUCs, improving unsatisfiability detection.
• To demonstrate that MUC-guided multi-task learning boosts overall SAT solving performance.

Proposed method

• A graph neural network (GNN) is used to generate clause embeddings from CNF-formatted formulas, capturing clause-level structural features.
• The model employs a hierarchical Transformer that groups clauses into higher-level units, enabling modeling of multi-clause relationships at varying granularities.
• Self-attention weights are dynamically adjusted to prioritize clauses that contribute to MUCs, guided by MUC prediction as a supervisory signal.
• Multi-task learning integrates SAT solving with MUC prediction, where the model jointly optimizes for both satisfiability and clause contribution to unsatisfiability.
• Clause ordering is randomized every 10 training iterations to improve robustness and generalization.

Experimental results

Research questions

• RQ1 Can learning minimal unsatisfiable cores (MUCs) as supervision improve the performance of end-to-end learning-based SAT solvers?
• RQ2 Does a hierarchical Transformer architecture better capture multi-clause interactions than standard pairwise self-attention in SAT solving?
• RQ3 How does MUC-guided attention improve the model’s ability to detect unsatisfiability?
• RQ4 To what extent does multi-task learning with MUC prediction enhance overall SAT solving accuracy?

Key findings

• SATformer achieves 94% accuracy on SR(3–10), 77% on SR(20), 68% on SR(40), and 61% on SR(60), significantly outperforming SATformer-NoHier, which degrades to random guessing on larger instances.
• The hierarchical structure enables correct prediction of unsatisfiable instances with reordered clauses (e.g., φU2), where a single-level model fails, demonstrating robustness to clause ordering.
• Removing MUC supervision (SATformer-NoCore) reduces accuracy across all datasets, confirming that MUC prediction is a key contributor to performance gains.
• SATformer-NoHier, which lacks hierarchical modeling, achieves only 70% accuracy on SR(3–10), indicating that pairwise attention alone is insufficient for complex clause interactions.