[Paper Review] Identification of Unexpected Decisions in Partially Observable Monte-Carlo Planning: a Rule-Based Approach

Giulio Mazzi, Alberto Castellini+1 more|arXiv (Cornell University)|Dec 23, 2020

Adversarial Robustness in Machine Learning22 references4 citations

TL;DR

This paper proposes a rule-based, SMT-driven methodology to detect unexpected decisions in POMCP policies by analyzing belief-action-observation traces. By formulating expert-provided logical rules with tunable parameters and solving a MAX-SMT problem, the approach identifies policy deviations from expected behavior, achieving up to a 47% improvement in AUC over state-of-the-art anomaly detection methods.

ABSTRACT

Partially Observable Monte-Carlo Planning (POMCP) is a powerful online algorithm able to generate approximate policies for large Partially Observable Markov Decision Processes. The online nature of this method supports scalability by avoiding complete policy representation. The lack of an explicit representation however hinders interpretability. In this work, we propose a methodology based on Satisfiability Modulo Theory (SMT) for analyzing POMCP policies by inspecting their traces, namely sequences of belief-action-observation triplets generated by the algorithm. The proposed method explores local properties of policy behavior to identify unexpected decisions. We propose an iterative process of trace analysis consisting of three main steps, i) the definition of a question by means of a parametric logical formula describing (probabilistic) relationships between beliefs and actions, ii) the generation of an answer by computing the parameters of the logical formula that maximize the number of satisfied clauses (solving a MAX-SMT problem), iii) the analysis of the generated logical formula and the related decision boundaries for identifying unexpected decisions made by POMCP with respect to the original question. We evaluate our approach on Tiger, a standard benchmark for POMDPs, and a real-world problem related to mobile robot navigation. Results show that the approach can exploit human knowledge on the domain, outperforming state-of-the-art anomaly detection methods in identifying unexpected decisions. An improvement of the Area Under Curve up to 47\% has been achieved in our tests.

Motivation & Objective

• To address the lack of interpretability in online POMCP policies due to their implicit, belief-based representation.
• To enable human experts to detect policy behaviors that violate their domain assumptions, especially in safety-critical robotic applications.
• To develop a scalable, iterative method for identifying anomalous decisions in POMCP traces without requiring full policy retraining.
• To outperform existing anomaly detection methods by integrating human-provided logical constraints into the analysis process.
• To provide interpretable decision boundaries that highlight when and why POMCP deviates from expert expectations.

Proposed method

• The method uses parametric logical formulas, called rule templates, to express expert knowledge about expected agent behavior (e.g., 'move fast only if belief in cluttered path is low').
• Traces of belief-action-observation triplets from POMCP execution are analyzed to instantiate rule parameters via a MAX-SMT solver, maximizing the number of satisfied clauses.
• The approach formulates the parameter tuning task as a MAX-SMT problem, allowing for approximate solutions when rules are not fully satisfiable.
• Decision boundaries derived from the optimized rules are used to classify policy decisions as expected or unexpected based on belief states.
• Hellinger distance is used to quantify divergence between belief states and rule boundaries, aiding in anomaly ranking.
• The process is iterative: experts refine rules based on identified anomalies, enabling local policy behavior exploration.

Experimental results

Research questions

• RQ1 Can a rule-based, SMT-driven approach detect unexpected decisions in POMCP policies more effectively than state-of-the-art anomaly detection methods?
• RQ2 How can expert-provided qualitative assumptions about agent behavior be formalized and quantitatively validated using trace data?
• RQ3 To what extent can the integration of human knowledge improve the interpretability and detection accuracy of policy anomalies?
• RQ4 How do decision boundaries derived from learned rules compare to actual policy decisions in terms of belief state distribution and anomaly detection performance?
• RQ5 Can the method identify and rank anomalies even when the policy is slightly deviant but not fully violating a rule?

Key findings

• The proposed method achieved an improvement of up to 47% in Area Under Curve (AUC) compared to state-of-the-art anomaly detection methods when detecting unexpected decisions in POMCP policies.
• In the velocity regulation task with a deliberately misconfigured reward parameter (W=90), the method correctly identified 4 out of 3500 decisions as unexpected, all of which were confirmed as anomalous by expert analysis.
• The method detected unexpected decisions even when belief states were nearly uniform (e.g., p0≈p1≈p2≈0.33), indicating high confidence in incorrect actions despite low information.
• The t-SNE visualization showed that unexpected decisions were often isolated in belief space, particularly those with high Hellinger distance from the rule boundary, confirming spatial coherence of anomalies.
• The approach successfully identified 4 anomalies in the Tiger benchmark and 4 in the robot navigation task, all of which were consistent with expert expectations and deviated from the rule template.
• The MAX-SMT formulation allowed robust parameter estimation even when the rule template was not fully satisfiable, enabling effective anomaly detection in real-world policy traces.