[Paper Review] Policy Error Bounds for Model-Based Reinforcement Learning with Factored Linear Models

Bernardo Ávila Pires, Csaba Szepesvári|arXiv (Cornell University)|Feb 19, 2016

Reinforcement Learning in Robotics18 references10 citations

TL;DR

This paper establishes novel policy error bounds for model-based reinforcement learning using factored linear models, introducing bounds in weighted norms—previously unexplored in MBRL—while reducing sensitivity to the discount factor and eliminating measure mismatch issues. The key contribution is a new analysis framework using Banach lattices and mixed norms that ensures value selection operators are non-expansive, yielding tighter, more robust performance guarantees than prior approaches.

ABSTRACT

In this paper we study a model-based approach to calculating approximately optimal policies in Markovian Decision Processes. In particular, we derive novel bounds on the loss of using a policy derived from a factored linear model, a class of models which generalize numerous previous models out of those that come with strong computational guarantees. For the first time in the literature, we derive performance bounds for model-based techniques where the model inaccuracy is measured in weighted norms. Moreover, our bounds show a decreased sensitivity to the discount factor and, unlike similar bounds derived for other approaches, they are insensitive to measure mismatch. Similarly to previous works, our proofs are also based on contraction arguments, but with the main differences that we use carefully constructed norms building on Banach lattices, and the contraction property is only assumed for operators acting on "compressed" spaces, thus weakening previous assumptions, while strengthening previous results.

Motivation & Objective

• To derive performance bounds for policies derived from factored linear models in model-based reinforcement learning.
• To address the limitations of existing bounds that rely on supremum norms, which are overly conservative in data-sparse or non-uniformly distributed settings.
• To eliminate sensitivity to measure mismatch—common in approximate linear programming and approximate dynamic programming—by leveraging a novel operator framework.
• To generalize previous results by recovering existing supremum norm bounds under milder assumptions.
• To show that model-based approaches can avoid the error amplification issues seen in ALP and ADP, particularly under data distribution shifts.

Proposed method

• The authors introduce a factored linear model structure where the transition kernel is approximated as a product of two linear operators: a right factor R mapping to a compressed space W, and a left factor Q mapping back to the action-value space.
• They define key operators including the Bellman operator T_Q on the compressed space, the value selection operator M' on W^A, and the full return operator T_QR.
• A novel mixed norm structure is constructed using Banach lattice theory, ensuring that the value selection operator M' is non-expansive under these norms.
• The analysis uses contraction arguments on compressed spaces, weakening the need for strong global contraction assumptions while strengthening the resulting bounds.
• The framework allows for performance bounds in supremum, weighted supremum, and weighted L^p norms, with model error measured in weighted norms.
• Theoretical bounds are derived under mild boundedness and homomorphism conditions on the model components, with a key condition on the swapped operator Q R ensuring norm control.

Experimental results

Research questions

• RQ1 Can policy error bounds in model-based RL be derived using weighted norms instead of the traditional supremum norm, which is known to be overly pessimistic in practice?
• RQ2 Does the use of factored linear models in MBRL allow for reduced sensitivity to the discount factor compared to prior approaches?
• RQ3 Can the proposed framework eliminate the measure mismatch problem that plagues approximate linear programming and approximate dynamic programming?
• RQ4 Is it possible to derive bounds that are both tighter and more robust by leveraging the structure of Banach lattices and mixed norms?
• RQ5 How do the new bounds compare to existing ones in terms of recovery of classical results and generalization to new settings?

Key findings

• The paper derives the first policy error bounds for model-based RL in which model inaccuracy is measured in weighted norms, enabling more natural alignment with learning objectives.
• The bounds show reduced sensitivity to the discount factor γ, a significant improvement over previous approaches.
• The analysis is insensitive to measure mismatch, avoiding the hard-to-control error terms that arise in ALP and ADP when sampling distributions differ from the optimal policy’s stationary distribution.
• The framework recovers existing supremum norm bounds for factored linear models under the join-homomorphism condition, validating consistency with prior work.
• The use of Banach lattice theory enables a new definition of mixed norms for action-value functions, ensuring that value selection operators are non-expansive, which is critical for stability.
• The condition on the swapped operator Q R is shown to be more tractable than previous global contraction assumptions, as it constrains a low-dimensional operator.