[Paper Review] Transfer with Model Features in Reinforcement Learning

Lucas Lehnert, Michael L. Littman|arXiv (Cornell University)|Jul 4, 2018

Reinforcement Learning in Robotics11 references4 citations

TL;DR

This paper introduces Model Features, a representation that clusters behaviorally equivalent states to enable efficient transfer in reinforcement learning across tasks with varying transition and reward functions. By formulating a novel optimization objective and proving convergence to a model reduction, the method enables accurate value function approximation and transfer performance even when dynamics differ between tasks.

ABSTRACT

A key question in Reinforcement Learning is which representation an agent can learn to efficiently reuse knowledge between different tasks. Recently the Successor Representation was shown to have empirical benefits for transferring knowledge between tasks with shared transition dynamics. This paper presents Model Features: a feature representation that clusters behaviourally equivalent states and that is equivalent to a Model-Reduction. Further, we present a Successor Feature model which shows that learning Successor Features is equivalent to learning a Model-Reduction. A novel optimization objective is developed and we provide bounds showing that minimizing this objective results in an increasingly improved approximation of a Model-Reduction. Further, we provide transfer experiments on randomly generated MDPs which vary in their transition and reward functions but approximately preserve behavioural equivalence between states. These results demonstrate that Model Features are suitable for transfer between tasks with varying transition and reward functions.

Motivation & Objective

• To address the challenge of transferring knowledge efficiently between reinforcement learning tasks where transition dynamics and reward functions differ.
• To develop a feature representation that clusters behaviorally equivalent states, preserving essential dynamics for reward prediction.
• To formalize a learning objective that converges to a model reduction, ensuring minimal loss of predictive power.
• To demonstrate that Model Features enable effective transfer even when only the reward function changes, extending beyond prior methods reliant on shared dynamics.
• To provide theoretical bounds showing that minimizing the proposed objective leads to increasingly accurate approximations of a model reduction.

Proposed method

• Proposes Model Features as a feature representation that assigns identical vectors to behaviorally equivalent states, effectively performing model reduction.
• Introduces a novel optimization objective to learn features that minimize the difference between original and compressed transition dynamics.
• Derives theoretical bounds showing that minimizing the objective leads to a better approximation of a model reduction, with error decreasing as the objective is minimized.
• Adapts the Successor Features architecture to learn Model Features, enabling end-to-end training with deep neural networks.
• Uses the Successor Feature framework to show that learning Model Features is equivalent to learning a model-reduced MDP.
• Employs matrix norms and spectral bounds to analyze approximation error, proving that value function errors are bounded by reward and transition approximation errors.

Experimental results

Research questions

• RQ1 Can a feature representation be learned that clusters behaviorally equivalent states and enables transfer across tasks with different transition and reward functions?
• RQ2 Does minimizing the proposed optimization objective lead to a better approximation of a model reduction in MDPs?
• RQ3 How does the performance of Model Features compare to existing methods like Successor Features in transfer learning scenarios with varying dynamics?
• RQ4 To what extent can Model Features preserve predictive accuracy for value functions under model reduction?
• RQ5 Is there a theoretical guarantee that the learned features converge to a valid model reduction as the optimization objective is minimized?

Key findings

• Model Features successfully cluster behaviorally equivalent states, as demonstrated in a 30×3 grid world where columns are collapsed into single states without loss of predictive power.
• The proposed optimization objective ensures that minimizing it leads to increasingly accurate approximations of a model reduction, with theoretical bounds on error propagation.
• The method achieves bounded value function approximation error, with the error upper bound decreasing as the reward and transition approximation errors decrease.
• Transfer experiments on randomly generated MDPs show that Model Features enable effective knowledge transfer even when transition functions differ between tasks.
• Theoretical analysis proves that the value function approximation error is bounded by the reward approximation error and the transition approximation error, scaled by the discount factor.
• Model Features outperform standard Successor Features in transfer scenarios where dynamics differ, as they are not restricted to shared transition functions.