[Paper Review] Near Optimal Behavior via Approximate State Abstraction

David Abel, D Ellis Hershkowitz+1 more|arXiv (Cornell University)|Jan 15, 2017

Reinforcement Learning in Robotics25 references109 citations

TL;DR

The paper introduces four approximate state abstraction functions for MDPs, proving that abstract-optimal policies yield bounded suboptimality in the ground MDP, and demonstrates empirically that abstraction reduces task complexity with controlled loss.

ABSTRACT

The combinatorial explosion that plagues planning and reinforcement learning (RL) algorithms can be moderated using state abstraction. Prohibitively large task representations can be condensed such that essential information is preserved, and consequently, solutions are tractably computable. However, exact abstractions, which treat only fully-identical situations as equivalent, fail to present opportunities for abstraction in environments where no two situations are exactly alike. In this work, we investigate approximate state abstractions, which treat nearly-identical situations as equivalent. We present theoretical guarantees of the quality of behaviors derived from four types of approximate abstractions. Additionally, we empirically demonstrate that approximate abstractions lead to reduction in task complexity and bounded loss of optimality of behavior in a variety of environments.

Motivation & Objective

• Motivate and formalize the use of approximate state abstractions to tame the curse of dimensionality in planning and reinforcement learning.
• Propose four concrete abstraction families that trade off compression and bounded performance loss.
• Provide theoretical guarantees showing suboptimality is bounded and polynomial in the approximation parameter ε.
• Empirically evaluate how abstraction degree affects compression and resulting policy quality across diverse MDPs.

Proposed method

• Define abstract MDPs via state aggregation with weighted ground-state contributions to rewards and transitions.
• Introduce four approximate aggregation functions: ˜φ_{Q*,ε}, ˜φ_{model,ε}, ˜φ_{ bolt,ε}, and ˜φ_{mult,ε}.
• Prove a main bound: V_G^{π_G*}(s) − V_G^{π_GA}(s) ≤ 2ε η_f, with η_f depending on the abstraction type.
• Establish lemmas bounding Q-values and policy quality for each abstraction family.
• Show that as ε → 0, the bound collapses to zero, recovering exact abstraction properties.
• Outline connections to existing bisimulation and similarity-based abstractions.

Experimental results

Research questions

• RQ1 Can approximate state abstractions preserve near-optimal behavior when aggregating sufficiently similar ground states?
• RQ2 What are the theoretical bounds on suboptimality for the four proposed abstraction families in terms of ε and MDP parameters?
• RQ3 How do different abstraction criteria (Q*, model, Boltzmann, multinomial) compare in terms of compression and loss?
• RQ4 Do approximate abstractions yield practical reductions in task complexity while maintaining bounded performance loss across varied domains?

Key findings

• There exist four approximate state aggregation functions that yield bounded suboptimality when applying the abstract optimal policy to the ground MDP.
• The suboptimality bound is a function of ε and a problem-dependent factor η_f, showing polynomial dependence on ε for the four families.
• Approximate abstractions enable greater compression than exact abstractions, especially when no exact state equality exists.
• Theoretical results relate abstraction quality to bounds on value and Q-values between ground and abstract MDPs.
• Empirical results illustrate a trade-off between degree of compression and incurred error across multiple MDPs.
• The method preserves essential structure of the decision problem while keeping computation tractable.