[Paper Review] Critic Regularized Regression

Ziyu Wang, Alexander Novikov+9 more|arXiv (Cornell University)|Jun 26, 2020

Reinforcement Learning in RoboticsComputer Science34 references90 citations

TL;DR

CRR is an offline reinforcement learning algorithm that filters actions using a learned critic to steer policy learning from fixed datasets, outperforming several state-of-the-art offline RL methods across diverse high-dimensional tasks.

ABSTRACT

Offline reinforcement learning (RL), also known as batch RL, offers the prospect of policy optimization from large pre-recorded datasets without online environment interaction. It addresses challenges with regard to the cost of data collection and safety, both of which are particularly pertinent to real-world applications of RL. Unfortunately, most off-policy algorithms perform poorly when learning from a fixed dataset. In this paper, we propose a novel offline RL algorithm to learn policies from data using a form of critic-regularized regression (CRR). We find that CRR performs surprisingly well and scales to tasks with high-dimensional state and action spaces -- outperforming several state-of-the-art offline RL algorithms by a significant margin on a wide range of benchmark tasks.

Motivation & Objective

• Motivate learning policies from fixed, offline datasets to address data collection cost and safety.
• Develop a simple, implementable offline RL method compatible with standard actor-critic frameworks.
• Mitigate extrapolation and overestimation issues by filtering actions via a learned critic.
• Show that CRR scales to high-dimensional state and action spaces and diverse data qualities.

Proposed method

• Formulate policy learning as a critic-filtered regression where the policy is trained on data actions whose Q-values are not outperformed by the current policy.
• Use a distributional Q-function and a non-negative filter f that increases with Q to restrict policy updates to data-supported actions.
• Explore different filtering functions f, including binary and exponential weighting based on estimated advantages.
• Implement CRR with deep networks, target networks, and a distributional critic to stabilize training.
• Introduce Critic Weighted Policy (CWP) to improve action selection at test time by reweighting actions with Q-values.
• Provide an algorithmic procedure (Algorithm 1) for CRR updates and discuss advantages of exponential weighting (Eq. 4) over simple BC-like filtering.

Experimental results

Research questions

• RQ1 Can CRR effectively restrict policy updates to the support of the offline dataset to avoid poor Q-estimates in offline RL?
• RQ2 How do different advantage estimators and filtering functions affect performance in offline RL?
• RQ3 Does CRR scale to high-dimensional state and action spaces and diverse data qualities?
• RQ4 What impact does Critic Weighted Policy (CWP) have on test-time performance compared to standard CRR?

Key findings

• CRR outperforms several state-of-the-art offline RL algorithms on a wide range of benchmark tasks.
• CRR scales to high-dimensional state and action spaces and handles datasets of diverse or low-quality data.
• Different CRR variants (exp, binary, binary max) perform robustly across tasks, with some variants excelling in easier vs harder domains.
• CWP generally improves CRR performance at test time across domains.
• CRR often surpasses D4PG, BCQ, ABM, and Behavior Cloning on challenging manipulation and locomotion tasks.