[Paper Review] Classification-based Approximate Policy Iteration: Experiments and Extended Discussions

Amir massoud Farahmand, Doina Precup+2 more|arXiv (Cornell University)|Jul 2, 2014

Reinforcement Learning in Robotics1 references3 citations

TL;DR

This paper introduces Classification-based Approximate Policy Iteration (CAPI), a framework that jointly exploits regularities in both value functions and policies for sample-efficient reinforcement learning. By using a classifier to generalize greedy actions from value function estimates, CAPI achieves superior sample efficiency—demonstrated by solving a large HIV control task and outperforming baseline methods with orders of magnitude fewer samples—while maintaining theoretical guarantees on performance loss and estimation error.

ABSTRACT

Tackling large approximate dynamic programming or reinforcement learning problems requires methods that can exploit regularities, or intrinsic structure, of the problem in hand. Most current methods are geared towards exploiting the regularities of either the value function or the policy. We introduce a general classification-based approximate policy iteration (CAPI) framework, which encompasses a large class of algorithms that can exploit regularities of both the value function and the policy space, depending on what is advantageous. This framework has two main components: a generic value function estimator and a classifier that learns a policy based on the estimated value function. We establish theoretical guarantees for the sample complexity of CAPI-style algorithms, which allow the policy evaluation step to be performed by a wide variety of algorithms (including temporal-difference-style methods), and can handle nonparametric representations of policies. Our bounds on the estimation error of the performance loss are tighter than existing results. We also illustrate this approach empirically on several problems, including a large HIV control task.

Motivation & Objective

• To address the challenge of sample inefficiency in large-scale reinforcement learning by leveraging both value function and policy regularities.
• To develop a unified framework that generalizes existing classification-based RL methods and enables flexible policy representation.
• To provide theoretical guarantees on estimation error and performance loss for CAPI-style algorithms.
• To empirically validate CAPI's superiority in sample efficiency and stability across diverse environments, including high-dimensional control tasks.
• To demonstrate that CAPI can outperform value-based and policy-based methods when data is scarce, especially when policies have simple, structured forms.

Proposed method

• CAPI uses a generic value function estimator to approximate the action-value function, enabling flexible policy evaluation via various algorithms including TD-style and LSTD methods.
• A classifier is trained on estimated action-values to learn a policy, generalizing greedy actions across the state space and exploiting policy regularities.
• The framework supports nonparametric policy representations, such as decision trees or k-NN, allowing adaptive complexity based on data.
• Policy evaluation is decoupled from policy improvement, allowing the use of any value estimation method, including rollout-based or function approximation techniques.
• The method employs a weighted loss function in tree-based classifiers to emphasize action-gap differences, improving robustness to noisy value estimates.
• Model selection for policy complexity (e.g., k in k-NN or min-split in trees) is treated as a problem-dependent tuning task to balance bias and variance.

Experimental results

Research questions

• RQ1 Can a unified framework effectively exploit both value function and policy regularities in large-scale reinforcement learning?
• RQ2 How does CAPI compare in sample efficiency to standard value-based (e.g., Fitted Q-Iteration) and policy-based (e.g., DPI) methods?
• RQ3 What is the impact of policy space complexity (e.g., k in k-NN or tree depth) on CAPI’s performance and generalization?
• RQ4 Can CAPI achieve better performance than rollout-based or value-based methods when data is limited and expensive to collect?
• RQ5 What theoretical bounds can be established for the estimation error and performance loss in CAPI-style algorithms?

Key findings

• CAPI significantly outperformed Tree-FQI and Tree-DPI in sample efficiency on the pole-balancing task, solving the task with ~10^4 samples compared to Tree-DPI’s ~3×10^7 samples.
• With 15,000 samples, CAPI with ηπ=20 achieved perfect performance in all 50 runs, while Tree-FQI required ~3×10^4 samples to reach similar performance.
• The choice of K in KNN-CAPI had a noticeable impact on performance, especially in low-sample regimes, indicating that policy space complexity must be tuned per problem.
• CAPI with a small policy space (e.g., ηπ=20) achieved near-optimal performance even with limited data, demonstrating effective exploitation of policy regularities.
• Tree-CAPI outperformed Tree-FQI in early training stages, showing that policy-based generalization can be more sample-efficient than value-based function approximation in data-scarce settings.
• The computational cost of CAPI was higher than Tree-DPI, but the dramatic reduction in data requirements makes CAPI preferable when data collection is expensive.