[Paper Review] Towards Neural-Guided Program Synthesis for Linear Temporal Logic Specifications

Alberto Rivas, Sheila A. McIlraith|arXiv (Cornell University)|Dec 31, 2019

Formal Methods in Verification24 references4 citations

TL;DR

This paper proposes a neural-guided search approach for Linear Temporal Logic (LTL) synthesis, reformulating the problem as an optimization task using deep Q-learning to guide search. The method learns a Q-function to improve search efficiency and successfully synthesizes correct-by-construction programs for small LTL specifications, demonstrating effective guidance with minimal training episodes despite the problem's 2EXP-complete complexity.

ABSTRACT

Synthesizing a program that realizes a logical specification is a classical problem in computer science. We examine a particular type of program synthesis, where the objective is to synthesize a strategy that reacts to a potentially adversarial environment while ensuring that all executions satisfy a Linear Temporal Logic (LTL) specification. Unfortunately, exact methods to solve so-called LTL synthesis via logical inference do not scale. In this work, we cast LTL synthesis as an optimization problem. We employ a neural network to learn a Q-function that is then used to guide search, and to construct programs that are subsequently verified for correctness. Our method is unique in combining search with deep learning to realize LTL synthesis. In our experiments the learned Q-function provides effective guidance for synthesis problems with relatively small specifications.

Motivation & Objective

• To address the scalability limitations of exact LTL synthesis methods, which are 2EXP-complete and infeasible for large problems.
• To explore whether deep reinforcement learning can provide effective guidance for search in LTL synthesis by learning a Q-function from experience.
• To combine search with deep learning to scale synthesis beyond traditional logical inference techniques.
• To evaluate the effectiveness of neural guidance in reducing search space exploration while maintaining correctness guarantees.
• To investigate the impact of architectural choices—such as using potentials, losing gameplays, and automata decompositions—on learning efficiency and solution quality.

Proposed method

• Reformulate LTL synthesis as a dynamic programming problem to enable optimization-based learning.
• Use a deep neural network to learn a Q-function that estimates the expected return of actions in state-action pairs during synthesis.
• Apply deep Q-learning with experience replay and target networks to stabilize training and improve policy learning.
• Integrate neural guidance into a heuristic search framework, using the Q-function to prioritize promising paths in the search space.
• Enhance learning with stronger supervision by incorporating potentials (i.e., heuristic state values) as additional reward signals.
• Use automata decompositions to handle large LTL formulas that cannot be converted into a single automaton due to computational constraints.

Experimental results

Research questions

• RQ1 Can a learned Q-function effectively guide search in LTL synthesis, reducing the need for exhaustive exploration of the exponentially large state space?
• RQ2 How do architectural enhancements—such as using potentials, reusing losing gameplays, and automata decompositions—affect the performance and convergence of the learning-based synthesis method?
• RQ3 To what extent can neural-guided search scale to LTL synthesis problems compared to traditional logical inference methods?
• RQ4 Does the use of neural guidance lead to correct-by-construction programs that satisfy complex LTL specifications with minimal training?
• RQ5 Can the method produce compact controllers, or does it consistently generate larger-than-minimal solutions?

Key findings

• The learned Q-function provided effective guidance for synthesis, enabling the system to solve multiple benchmarks from the SYNTCOMP 2019 suite with relatively small specifications.
• Using potentials as a supervision signal significantly improved learning performance, with DDQS[−, φ] and dec-DDQS[−, φ] configurations outperforming others.
• Reusing losing gameplays reduced the number of required training episodes and batch learning steps, accelerating convergence.
• The use of automata decompositions increased input size and required more training steps, but remained computationally feasible compared to single-automaton transformations.
• Controllers generated by the method were significantly larger than the minimal controllers reported in SYNTCOMP 2019, indicating room for improvement in policy optimization.
• Despite not achieving state-of-the-art performance, the approach demonstrated that combining deep learning with search is a viable and foundational path for scalable LTL synthesis.