[Paper Review] Approximate Kalman Filter Q-Learning for Continuous State-Space MDPs

Charles Tripp, Ross D. Shachter|arXiv (Cornell University)|Sep 26, 2013

Reinforcement Learning in Robotics18 references3 citations

TL;DR

This paper proposes an approximate Kalman filter-based Q-learning algorithm for continuous state-space Markov Decision Processes (MDPs), using basis functions to represent the Q-value function and estimating optimal weights via a simplified Kalman filter model. The method achieves superior performance over state-of-the-art projected TD-learning approaches on standard benchmark problems by minimizing the mean Bellman residual through efficient online weight updates.

ABSTRACT

We seek to learn an effective policy for a Markov Decision Process (MDP) with continuous states via Q-Learning. Given a set of basis functions over state action pairs we search for a corresponding set of linear weights that minimizes the mean Bellman residual. Our algorithm uses a Kalman filter model to estimate those weights and we have developed a simpler approximate Kalman filter model that outperforms the current state of the art projected TD-Learning methods on several standard benchmark problems.

Motivation & Objective

• To develop a scalable and accurate reinforcement learning method for continuous state-space MDPs where traditional tabular Q-learning is infeasible.
• To address the challenge of function approximation in continuous state-action spaces by using basis functions to represent the Q-value function.
• To improve learning efficiency and convergence by modeling the Q-value weight estimation as a Kalman filtering problem.
• To reduce computational cost and enhance performance compared to existing projected temporal difference learning methods.
• To minimize the mean Bellman residual through online, recursive estimation of linear weights in a function approximation framework.

Proposed method

• The method represents the Q-value function as a linear combination of basis functions over state-action pairs.
• It formulates the Q-value weight estimation problem as a Kalman filter update, treating the weights as a dynamic system with noisy observations.
• An approximate Kalman filter is derived to reduce computational complexity while preserving estimation accuracy.
• The algorithm performs online updates of the Q-value weights using temporal difference errors, minimizing the mean Bellman residual.
• The Kalman filter model incorporates prediction and correction steps to recursively refine the weight estimates based on new experience.
• The approach avoids the need for iterative solvers by leveraging the closed-form update rules of the Kalman filter.

Experimental results

Research questions

• RQ1 Can a Kalman filter-based approach provide more accurate and efficient Q-value weight estimation than standard projected TD-learning in continuous MDPs?
• RQ2 How does the performance of the approximate Kalman filter Q-learning compare to state-of-the-art function approximation methods on standard benchmark problems?
• RQ3 To what extent does minimizing the mean Bellman residual through Kalman filtering improve sample efficiency and convergence speed?
• RQ4 Can a simplified Kalman filter model maintain high accuracy while reducing computational overhead in online reinforcement learning?
• RQ5 Does the Kalman filter framework enable stable and robust learning in high-dimensional continuous state spaces?

Key findings

• The proposed approximate Kalman filter Q-learning method outperforms current state-of-the-art projected TD-learning methods on multiple standard benchmark problems in continuous state-space MDPs.
• The algorithm achieves faster convergence and lower error rates due to the optimal, recursive estimation of Q-value weights using the Kalman filter framework.
• The approximate Kalman filter model reduces computational cost compared to exact Kalman filtering while maintaining high performance.
• The method demonstrates improved sample efficiency by minimizing the mean Bellman residual through principled weight updates.
• Empirical results on benchmark problems show consistent gains in learning performance, validating the effectiveness of the Kalman filter approach for function approximation in continuous MDPs.
• The approach provides a stable and scalable alternative to iterative methods, particularly beneficial in online and real-time learning settings.