[Paper Review] Supervisor Synthesis of POMDP based on Automata Learning

Xiaobin Zhang, Bo Wu+1 more|arXiv (Cornell University)|Mar 24, 2017

Formal Methods in Verification37 references4 citations

TL;DR

This paper proposes an L*-learning-based supervisor synthesis framework for Partially Observable Markov Decision Processes (POMDPs) using a specialized deterministic finite automaton (za-DFA) to enforce finite-horizon Probabilistic Computation Tree Logic (PCTL) specifications. The method automatically synthesizes a permissive, non-blocking supervisor by iteratively refining the za-DFA through membership queries and counterexample-driven learning, achieving sound and complete synthesis with guaranteed termination.

ABSTRACT

As a general and thus popular model for autonomous systems, partially observable Markov decision process (POMDP) can capture uncertainties from different sources like sensing noises, actuation errors, and uncertain environments. However, its comprehensiveness makes the planning and control in POMDP difficult. Traditional POMDP planning problems target to find the optimal policy to maximize the expectation of accumulated rewards. But for safety critical applications, guarantees of system performance described by formal specifications are desired, which motivates us to consider formal methods to synthesize supervisor for POMDP. With system specifications given by Probabilistic Computation Tree Logic (PCTL), we propose a supervisory control framework with a type of deterministic finite automata (DFA), za-DFA, as the controller form. While the existing work mainly relies on optimization techniques to learn fixed-size finite state controllers (FSCs), we develop an $L^*$ learning based algorithm to determine both space and transitions of za-DFA. Membership queries and different oracles for conjectures are defined. The learning algorithm is sound and complete. An example is given in detailed steps to illustrate the supervisor synthesis algorithm.

Motivation & Objective

• To address the challenge of synthesizing supervisors for POMDPs that guarantee satisfaction of formal PCTL specifications in safety-critical applications.
• To overcome the limitations of traditional optimization-based finite state controller (FSC) learning by enabling automatic discovery of both structure and transitions in the controller.
• To develop a sound and complete learning-based framework that synthesizes a permissive za-DFA supervisor without requiring prior knowledge of controller size.
• To integrate probabilistic model checking with automata learning by modifying POMCP and L* algorithms for POMDPs.
• To ensure termination and correctness through novel oracles for membership queries, conjecture validation, and counterexample handling.

Proposed method

• The framework uses a specialized deterministic finite automaton (za-DFA) as the supervisor, which encodes history-dependent control policies for POMDPs.
• An L*-learning algorithm is adapted to infer the za-DFA structure by issuing membership queries and processing counterexamples from three oracles: OracleP (POMDP model checking), OracleB (belief state satisfaction), and OracleS (counterexample selection for PCTL).
• The POMDP model checking step is accelerated by modifying the POMCP algorithm to compute the maximum probability of satisfying a PCTL formula over finite horizons.
• The observation table in L* is extended and refined iteratively to maintain closure and consistency, ensuring sound and complete learning.
• Counterexamples are used to refine the conjectured za-DFA, with OracleS identifying adversarial observation sequences that violate the PCTL specification.
• The final synthesized za-DFA is permissive, allowing multiple actions per state to preserve control freedom while ensuring PCTL satisfaction.

Experimental results

Research questions

• RQ1 Can L*-learning be effectively adapted to synthesize supervisors for POMDPs with formal PCTL specifications?
• RQ2 How can the learning process be made sound and complete when dealing with probabilistic systems and finite-horizon PCTL?
• RQ3 What modifications to membership queries and counterexample handling are required to ensure correct and efficient supervisor synthesis in POMDPs?
• RQ4 Can the proposed framework automatically determine both the state space and transition structure of the supervisor without prior size constraints?
• RQ5 How can the permissiveness of the synthesized supervisor be preserved while guaranteeing PCTL satisfaction?

Key findings

• The proposed L*-learning-based supervisor synthesis algorithm is sound and complete, guaranteeing termination and correctness for finite-horizon PCTL specifications.
• The framework successfully synthesizes a permissive za-DFA supervisor that allows multiple actions per state, enhancing control flexibility.
• In a detailed example, the algorithm converged in six iterations to a final za-DFA with a maximum satisfaction probability of 0.271, below the threshold of 0.28, confirming compliance with the specification.
• The algorithm correctly identified and refined counterexamples through iterative learning, including negative counterexamples such as 11, 124, 121, and 123, each leading to a refined hypothesis.
• The integration of POMCP with PCTL model checking enabled efficient computation of satisfaction probabilities, reducing computational overhead in the learning loop.
• The framework demonstrates the feasibility of combining automata learning with probabilistic model checking for formal synthesis in POMDPs, with potential for full software integration in the future.