[Paper Review] Offline Reinforcement Learning with Fisher Divergence Critic Regularization

Ilya Kostrikov, Jonathan Tompson+2 more|arXiv (Cornell University)|Mar 14, 2021

Reinforcement Learning in Robotics32 references25 citations

TL;DR

Fisher-BRC introduces a log-behavior-policy based critic with a gradient penalty that realizes Fisher divergence regularization, yielding state-of-the-art offline RL performance with faster convergence and improved stability.

ABSTRACT

Many modern approaches to offline Reinforcement Learning (RL) utilize behavior regularization, typically augmenting a model-free actor critic algorithm with a penalty measuring divergence of the policy from the offline data. In this work, we propose an alternative approach to encouraging the learned policy to stay close to the data, namely parameterizing the critic as the log-behavior-policy, which generated the offline data, plus a state-action value offset term, which can be learned using a neural network. Behavior regularization then corresponds to an appropriate regularizer on the offset term. We propose using a gradient penalty regularizer for the offset term and demonstrate its equivalence to Fisher divergence regularization, suggesting connections to the score matching and generative energy-based model literature. We thus term our resulting algorithm Fisher-BRC (Behavior Regularized Critic). On standard offline RL benchmarks, Fisher-BRC achieves both improved performance and faster convergence over existing state-of-the-art methods.

Motivation & Objective

• Motivate a regulator for the critic in offline RL to prevent extrapolation on unseen actions.
• Propose a critic parameterization that ties Q-values to the behavior policy via a log-density term plus a learnable offset.
• Derive a gradient penalty regularizer for the offset that is equivalent to Fisher divergence regularization.
• Show empirical advantages of Fisher-BRC over existing offline RL methods in terms of performance and efficiency.

Proposed method

• Parameterize the critic as Q(s,a)=Oθ(s,a)+log μ(a|s), where μ is the behavior policy learned via behavioral cloning.
• Regularize the critic by a gradient penalty on Oθ: minimize J(Oθ+log μ) + λ E_{s,a∼D, a∼πφ(·|s)}[||∇a Oθ(s,a)||^2].
• Connect the gradient penalty to Fisher divergence between the Boltzmann policy exp(Q)/Z and the behavior policy μ, avoiding explicit normalization.
• Train the actor with entropy regularization, using the offset-based critic to drive actions toward the data while allowing generalization beyond the dataset.
• Relate Fisher-BRC to BRAC and CQL, noting computational advantages over log-sum-exp based objectives.

Experimental results

Research questions

• RQ1 Can a log-behavior-policy based critic with offset gradient regularization yield robust offline RL performance?
• RQ2 Does the gradient penalty on the critic’s offset implement Fisher divergence regularization and provide benefits over traditional divergence-penalized policies?
• RQ3 Do Fisher-BRC methods offer faster convergence and better stability compared to state-of-the-art offline RL baselines on standard benchmarks?

Key findings

• Fisher-BRC achieves competitive to state-of-the-art results across D4RL benchmarks, with more consistent performance than several baselines.
• The gradient penalty is crucial: λ=0 degrades performance, while very large λ over-constrains the policy.
• Fisher-BRC converges faster than CQL and BRAC in gradient steps and wall-clock time.
• The method yields strong results particularly on medium and expert datasets, demonstrating robustness across tasks.
• The approach reduces computational burden relative to CQL by avoiding expensive log-sum-exp computations.
• Empirical results show F-BRC matching or exceeding baselines on most tasks, with improved convergence behavior.