[Paper Review] Adaptive Discretization for Model-Based Reinforcement Learning

Sean R. Sinclair, Tianyu Wang+3 more|arXiv (Cornell University)|Jul 1, 2020

Advanced Bandit Algorithms Research57 references9 citations

TL;DR

This paper proposes Adaptive Discretization for Model-Based Reinforcement Learning, which dynamically refines state-action space partitions based on data-driven need, reducing memory and computation while maintaining competitive regret bounds. The method achieves faster convergence and lower storage than fixed discretization, and empirically matches or exceeds model-free methods in performance with adaptive partitioning.

ABSTRACT

We introduce the technique of adaptive discretization to design an efficient model-based episodic reinforcement learning algorithm in large (potentially continuous) state-action spaces. Our algorithm is based on optimistic one-step value iteration extended to maintain an adaptive discretization of the space. From a theoretical perspective we provide worst-case regret bounds for our algorithm which are competitive compared to the state-of-the-art model-based algorithms. Moreover, our bounds are obtained via a modular proof technique which can potentially extend to incorporate additional structure on the problem. From an implementation standpoint, our algorithm has much lower storage and computational requirements due to maintaining a more efficient partition of the state and action spaces. We illustrate this via experiments on several canonical control problems, which shows that our algorithm empirically performs significantly better than fixed discretization in terms of both faster convergence and lower memory usage. Interestingly, we observe empirically that while fixed-discretization model-based algorithms vastly outperform their model-free counterparts, the two achieve comparable performance with adaptive discretization.

Motivation & Objective

• To address the inefficiency of fixed discretization in large or continuous state-action spaces by adapting the partitioning based on data.
• To design a model-based RL algorithm with provable worst-case regret bounds under Lipschitz continuity assumptions.
• To reduce storage and computational costs in online RL by maintaining only relevant, fine-grained partitions.
• To empirically evaluate whether adaptive discretization enables model-based RL to match or surpass model-free methods in performance.
• To provide a modular proof framework for regret analysis that may extend to structured problems.

Proposed method

• The algorithm uses optimistic one-step value iteration with an adaptive, data-dependent partition of the state and action spaces.
• It maintains estimates of rewards and transition probabilities, refining the partition only in regions with high uncertainty or value.
• The partitioning is guided by a bonus term that encourages exploration in under-sampled regions, similar to UCB principles.
• The method ensures that the number of partitions grows only where necessary, minimizing storage and computation.
• It leverages Lipschitz continuity of the dynamics and reward functions to bound estimation errors and derive regret guarantees.
• The algorithm is implemented with efficient data structures to support fast lookup and update of Q-values and counts.

Experimental results

Research questions

• RQ1 Can adaptive discretization reduce memory and computation costs in model-based RL without sacrificing regret performance?
• RQ2 How does the regret of adaptive discretization compare to state-of-the-art model-based algorithms in continuous MDPs?
• RQ3 Does adaptive discretization allow model-based RL to achieve performance comparable to model-free methods in practice?
• RQ4 Can the regret analysis be modularly extended to incorporate additional structural assumptions on the MDP?
• RQ5 How does the dynamic refinement of the partition affect convergence speed and final policy quality?

Key findings

• The proposed algorithm achieves worst-case regret bounds that are competitive with state-of-the-art model-based algorithms, scaling favorably with horizon and problem dimension.
• Adaptive discretization reduces memory usage significantly compared to fixed discretization, with partition sizes growing only in high-value regions.
• In experiments on oil discovery and ambulance routing, adaptive methods converged faster and used less memory than fixed-discretization baselines.
• The adaptive model-based algorithm empirically performs comparably to model-free methods, despite the latter typically being outperformed by fixed-discretization model-based methods.
• The adaptive partitioning strategy leads to finer grids in regions where optimal Q-values are high, improving learning efficiency.
• The model-free adaptive Q-learning algorithm maintains a finer partition than the model-based variant, suggesting different exploration behaviors.