[Paper Review] Lightning Does Not Strike Twice: Robust MDPs with Coupled Uncertainty

Shie Mannor, Ofir Mebel+1 more|arXiv (Cornell University)|Jun 18, 2012

Reinforcement Learning in Robotics17 references25 citations

TL;DR

This paper introduces a novel robust MDP framework that models parameter uncertainty through the 'Lightning Does Not Strike Twice' principle—limiting the number of state-parameter deviations to a bounded count—resulting in less conservative policies than traditional uncoupled uncertainty models. The approach enables tractable computation of optimal policies with probabilistic guarantees, offering a more realistic and efficient alternative for decision-making under uncertainty in control and learning systems.

ABSTRACT

We consider Markov decision processes under parameter uncertainty. Previous studies all restrict to the case that uncertainties among different states are uncoupled, which leads to conservative solutions. In contrast, we introduce an intuitive concept, termed "Lightning Does not Strike Twice," to model coupled uncertain parameters. Specifically, we require that the system can deviate from its nominal parameters only a bounded number of times. We give probabilistic guarantees indicating that this model represents real life situations and devise tractable algorithms for computing optimal control policies using this concept.

Motivation & Objective

• To address the over-conservatism in robust MDPs caused by uncoupled parameter uncertainty models.
• To model parameter uncertainty in a way that reflects real-world constraints where extreme deviations are rare.
• To develop a tractable computational framework for optimal policy computation under bounded deviation constraints.
• To provide probabilistic guarantees that the bounded deviation model reflects realistic system behavior.
• To improve decision-making robustness in control and reinforcement learning under uncertainty.

Proposed method

• Proposes a robust MDP formulation where the number of state-parameter deviations from nominal values is bounded by a constant.
• Models uncertainty as a constraint on the total number of state transitions that can deviate from nominal parameters.
• Uses a robust optimization framework to compute policies that are optimal under the worst-case deviation pattern within the bounded deviation limit.
• Employs dynamic programming and decomposition techniques to enable tractable computation of optimal policies.
• Derives probabilistic bounds on the likelihood of exceeding the deviation limit, linking the model to real-world plausibility.
• Applies the framework to both finite-horizon and infinite-horizon MDPs, ensuring scalability and practical applicability.

Experimental results

Research questions

• RQ1 How can we model parameter uncertainty in MDPs in a way that avoids the over-conservatism of traditional robust MDPs?
• RQ2 What is the impact of coupling uncertainty across states through a bounded number of deviations?
• RQ3 Can we compute optimal policies efficiently under this new uncertainty model?
• RQ4 How do the probabilistic guarantees of the bounded deviation model compare to standard robust MDPs?
• RQ5 Does the 'Lightning Does Not Strike Twice' principle reflect realistic system behavior in control and learning applications?

Key findings

• The proposed bounded deviation model significantly reduces policy conservatism compared to standard robust MDPs with uncoupled uncertainty.
• The framework enables tractable computation of optimal policies using dynamic programming and robust optimization techniques.
• Probabilistic guarantees are derived showing that the bounded deviation model aligns with real-world scenarios where extreme parameter shifts are rare.
• The method achieves better performance in terms of expected reward while maintaining robustness under worst-case deviations.
• The approach is applicable to both finite-horizon and infinite-horizon MDPs, demonstrating scalability and practical relevance.
• Empirical results show that the bounded deviation model leads to more aggressive and effective policies than traditional robust MDPs.