[Paper Review] Exploring compact reinforcement-learning representations with linear regression

Thomas J. Walsh, István Szita+2 more|arXiv (Cornell University)|May 9, 2012

Reinforcement Learning in Robotics14 references7 citations

TL;DR

This paper introduces a new online linear regression algorithm with improved sample complexity bounds that satisfies the KWIK (Knows What It Knows) framework. It enables efficient learning of compact representations in reinforcement learning, including reward functions in factored MDPs and outcome probabilities in Stochastic STRIPS and Object-Oriented MDPs—domains previously not known to be efficiently learnable in the RL setting.

ABSTRACT

This paper presents a new algorithm for online linear regression whose efficiency guarantees satisfy the requirements of the KWIK (Knows What It Knows) framework. The algorithm improves on the complexity bounds of the current state-of-the-art procedure in this setting. We explore several applications of this algorithm for learning compact reinforcement-learning representations. We show that KWIK linear regression can be used to learn the reward function of a factored MDP and the probabilities of action outcomes in Stochastic STRIPS and Object Oriented MDPs, none of which have been proven to be efficiently learnable in the RL setting before. We also combine KWIK linear regression with other KWIK learners to learn larger portions of these models, including experiments on learning factored MDP transition and reward functions together.

Motivation & Objective

• Address the challenge of learning compact, generalizable representations in reinforcement learning with sample efficiency.
• Overcome the lack of efficient learning guarantees for key components in structured MDPs such as factored MDPs and Object-Oriented MDPs.
• Develop a linear regression algorithm that meets the KWIK framework's efficiency requirements for online learning.
• Enable sample-efficient learning of reward functions and transition probabilities in structured RL settings.
• Combine KWIK linear regression with other KWIK learners to scale to larger model components in complex MDPs.

Proposed method

• Propose a novel online linear regression algorithm with improved sample complexity bounds compared to prior work.
• Integrate the algorithm within the KWIK learning framework, ensuring it only makes predictions with high confidence and can identify uncertainty.
• Use the KWIK linear regression algorithm to learn the reward function in factored MDPs by modeling rewards as linear functions of features.
• Apply the same algorithm to learn outcome probabilities in Stochastic STRIPS and Object-Oriented MDPs, where actions have probabilistic effects.
• Combine the KWIK linear regression learner with other KWIK learners (e.g., for value functions or transition models) to learn larger portions of structured MDPs.
• Ensure theoretical guarantees on sample efficiency and learning correctness under the KWIK framework.

Experimental results

Research questions

• RQ1 Can a KWIK-compliant linear regression algorithm be designed with improved sample complexity bounds compared to existing methods?
• RQ2 Is it possible to use KWIK linear regression to efficiently learn the reward function in factored MDPs?
• RQ3 Can KWIK linear regression be applied to learn action outcome probabilities in Stochastic STRIPS and Object-Oriented MDPs?
• RQ4 Can the combination of KWIK linear regression with other KWIK learners enable efficient learning of joint transition and reward functions in structured MDPs?
• RQ5 Are there previously unlearnable components in structured RL settings that become efficiently learnable using this approach?

Key findings

• The proposed KWIK linear regression algorithm achieves better sample complexity bounds than the state-of-the-art in the KWIK framework.
• The algorithm enables efficient, sample-efficient learning of the reward function in factored MDPs, a setting previously not known to be efficiently learnable in RL.
• The method successfully learns outcome probabilities in Stochastic STRIPS and Object-Oriented MDPs, domains that lacked prior efficient learning guarantees in RL.
• Combining KWIK linear regression with other KWIK learners allows for joint learning of transition and reward functions in factored MDPs, demonstrating scalability to larger model components.
• The theoretical framework ensures that predictions are made only when confident, and uncertainty is explicitly handled, improving learning reliability.
• The approach provides the first provably efficient learning method for reward and transition models in several structured MDP formalisms, expanding the scope of learnable RL representations.