[Paper Review] Learning RoboCup-Keepaway with Kernels

Tobias Jung, Daniel Polani|arXiv (Cornell University)|Jan 31, 2012

Reinforcement Learning in Robotics22 references3 citations

TL;DR

This paper proposes a kernel-based reinforcement learning approach using regularization networks with subset-of-regressors approximation to solve the high-dimensional, stochastic 3vs2 RoboCup-Keepaway problem. By employing efficient recursive least-squares policy iteration with supervised basis function selection, the method achieves superior performance over tilecoding while enabling real-time learning through dynamic, data-driven basis function inclusion.

ABSTRACT

We apply kernel-based methods to solve the difficult reinforcement learning problem of 3vs2 keepaway in RoboCup simulated soccer. Key challenges in keepaway are the high-dimensionality of the state space (rendering conventional discretization-based function approximation like tilecoding infeasible), the stochasticity due to noise and multiple learning agents needing to cooperate (meaning that the exact dynamics of the environment are unknown) and real-time learning (meaning that an efficient online implementation is required). We employ the general framework of approximate policy iteration with least-squares-based policy evaluation. As underlying function approximator we consider the family of regularization networks with subset of regressors approximation. The core of our proposed solution is an efficient recursive implementation with automatic supervised selection of relevant basis functions. Simulation results indicate that the behavior learned through our approach clearly outperforms the best results obtained earlier with tilecoding by Stone et al. (2005).

Motivation & Objective

• To address the challenges of high-dimensional state spaces and real-time learning in RoboCup-Keepaway, where conventional tilecoding fails due to exponential parameter growth.
• To develop a model-free, online reinforcement learning method that adapts to unknown stochastic dynamics in multi-agent settings.
• To enable efficient real-time learning by dynamically selecting only relevant basis functions using a supervised criterion.
• To outperform existing tilecoding-based approaches in terms of learning efficiency and final performance in the 3vs2 Keepaway task.

Proposed method

• Uses least-squares policy iteration (LSPE(λ)) with regularization networks as function approximators to estimate value functions in high-dimensional state spaces.
• Applies the subset of regressors method to approximate kernel functions using a reduced, dynamically selected set of basis functions.
• Employs a greedy, online basis selection strategy that prioritizes basis functions based on their contribution to reducing the learning cost, not just approximation error.
• Introduces a recursive update mechanism for efficient online learning, with per-step complexity dependent only on the number of selected basis functions.
• Incorporates a supervised selection criterion that evaluates the impact of each candidate basis function on the reduction of the original learning cost.
• Uses recursive matrix inversion and vector update formulas to maintain computational efficiency during online training and inference.

Experimental results

Research questions

• RQ1 Can kernel-based function approximation with dynamic basis selection outperform traditional tilecoding in high-dimensional, stochastic reinforcement learning tasks like RoboCup-Keepaway?
• RQ2 How effective is a supervised basis selection criterion in reducing the number of basis functions while maintaining or improving learning performance?
• RQ3 Can a recursive, least-squares-based policy iteration framework be made efficient enough for real-time online learning in a 100ms-per-action environment?
• RQ4 Does using the full state vector (13-dimensional) without simplifying assumptions lead to better performance than prior work that decouples state variables?
• RQ5 To what extent does the proposed method reduce parameter wastage in unvisited regions of the state space compared to grid-based methods?

Key findings

• The proposed kernel-based method significantly outperforms the tilecoding approach of Stone et al. (2005), achieving higher average episode lengths in the 3vs2 Keepaway task.
• The use of a supervised basis selection criterion reduces the number of basis functions required, leading to more efficient computation and lower per-step complexity.
• The recursive implementation enables real-time learning with a 100ms action cycle, satisfying the RoboCup server's timing constraints.
• The method effectively handles the high-dimensional state space (13-dimensional) without relying on simplifying assumptions about state variable independence.
• The subset-of-regressors approximation maintains high accuracy while drastically reducing computational overhead compared to full kernel methods.
• The learning performance is robust to environmental stochasticity and multi-agent cooperation dynamics, demonstrating generalization in complex, unknown environments.