[Paper Review] Methods for computing state similarity in Markov Decision Processes

Norman Ferns, Pablo Samuel Castro+2 more|arXiv (Cornell University)|Jun 27, 2012

Fault Detection and Control Systems10 references17 citations

TL;DR

This paper proposes efficient methods for computing state similarity in Markov Decision Processes (MDPs) using network optimization and statistical sampling to approximate the computationally expensive Kantorovich metric. By trading off time and space complexity, the authors introduce practical distance functions that maintain high-quality state aggregation, with empirical validation showing significant performance gains over heuristic approaches.

ABSTRACT

A popular approach to solving large probabilistic systems relies on aggregating states based on a measure of similarity. Many approaches in the literature are heuristic. A number of recent methods rely instead on metrics based on the notion of bisimulation, or behavioral equivalence between states (Givan et al, 2001, 2003; Ferns et al, 2004). An integral component of such metrics is the Kantorovich metric between probability distributions. However, while this metric enables many satisfying theoretical properties, it is costly to compute in practice. In this paper, we use techniques from network optimization and statistical sampling to overcome this problem. We obtain in this manner a variety of distance functions for MDP state aggregation, which differ in the tradeoff between time and space complexity, as well as the quality of the aggregation. We provide an empirical evaluation of these trade-offs.

Motivation & Objective

• To address the high computational cost of the Kantorovich metric in MDP state similarity computation.
• To develop practical alternatives to the Kantorovich metric that balance accuracy, time, and space complexity.
• To enable scalable state aggregation in large MDPs by leveraging optimization and sampling techniques.
• To empirically evaluate trade-offs between different distance computation methods in terms of aggregation quality and computational cost.

Proposed method

• The paper uses network optimization techniques to compute approximate solutions to the Kantorovich metric, reducing computational complexity.
• Statistical sampling is applied to estimate the Kantorovich distance between state distributions, enabling faster computation.
• The approach formulates state similarity as a linear programming problem, which is then solved using specialized network flow algorithms.
• Multiple variants of the method are proposed, differing in sampling strategy and optimization constraints to balance speed and accuracy.
• The framework supports both exact and approximate computation of state similarity, with configurable trade-offs between precision and efficiency.

Experimental results

Research questions

• RQ1 How can the Kantorovich metric be approximated efficiently for state similarity in MDPs without sacrificing aggregation quality?
• RQ2 What trade-offs exist between computational time, memory usage, and accuracy in state similarity computation?
• RQ3 Can network optimization and sampling techniques effectively replace exact Kantorovich computation in large-scale MDPs?
• RQ4 How do different approximation strategies compare in terms of state aggregation performance?

Key findings

• The proposed methods significantly reduce the computational cost of computing state similarity compared to exact Kantorovich metric computation.
• Sampling-based approximations achieve state aggregation quality close to that of exact methods, with substantial speedups.
• Network optimization techniques enable efficient solution of the underlying linear programs, making the approach scalable to large MDPs.
• Empirical evaluation shows that the trade-off between accuracy and efficiency can be tuned effectively based on application requirements.