[Paper Review] OptiDICE: Offline Policy Optimization via Stationary Distribution Correction Estimation

Jongmin Lee, Wonseok Jeon+3 more|arXiv (Cornell University)|Jun 21, 2021

Reinforcement Learning in Robotics4 citations

TL;DR

OptiDICE proposes a novel offline reinforcement learning algorithm that mitigates distributional shift by directly estimating stationary distribution corrections between the behavior data and the optimal policy, avoiding reliance on policy gradients or value overestimation. It formulates the problem as a convex optimization via closed-form solutions to a minimax subproblem, achieving state-of-the-art performance on D4RL benchmarks with minimal hyperparameter tuning.

ABSTRACT

We consider the offline reinforcement learning (RL) setting where the agent aims to optimize the policy solely from the data without further environment interactions. In offline RL, the distributional shift becomes the primary source of difficulty, which arises from the deviation of the target policy being optimized from the behavior policy used for data collection. This typically causes overestimation of action values, which poses severe problems for model-free algorithms that use bootstrapping. To mitigate the problem, prior offline RL algorithms often used sophisticated techniques that encourage underestimation of action values, which introduces an additional set of hyperparameters that need to be tuned properly. In this paper, we present an offline RL algorithm that prevents overestimation in a more principled way. Our algorithm, OptiDICE, directly estimates the stationary distribution corrections of the optimal policy and does not rely on policy-gradients, unlike previous offline RL algorithms. Using an extensive set of benchmark datasets for offline RL, we show that OptiDICE performs competitively with the state-of-the-art methods.

Motivation & Objective

• To address the critical challenge of distributional shift in offline reinforcement learning, where behavior and target policies differ, leading to value overestimation.
• To eliminate the need for complex value underestimation techniques that introduce additional hyperparameters in existing offline RL algorithms.
• To develop a method that optimizes policies directly in the space of stationary distributions, rather than via Q-functions or policy gradients.
• To achieve stable and sample-efficient offline policy optimization using principled distribution correction estimation.

Proposed method

• OptiDICE estimates the stationary distribution correction (SDC) ratio between the behavior policy's data distribution and the optimal policy's stationary distribution.
• It formulates the SDC estimation as a minimax optimization problem that avoids sampling from the target policy, enabling stable training.
• The method derives a closed-form solution to the inner minimax subproblem, transforming the overall problem into an unconstrained convex optimization.
• The algorithm uses this convex formulation to directly optimize the policy in the stationary distribution space, avoiding bootstrapped value overestimation.
• It employs two policy extraction methods: minimax and mean squared error (MSE) minimization, both grounded in the estimated SDCs.
• The approach is designed to be hyperparameter-efficient, with only a single hyperparameter (e.g., α in importance-weighted BC) to tune.

Experimental results

Research questions

• RQ1 Can stationary distribution correction estimation provide a principled alternative to value underestimation in offline RL?
• RQ2 Can a policy optimization method based solely on stationary distribution corrections achieve competitive performance without relying on policy gradients or Q-function bootstrapping?
• RQ3 How does the proposed convex optimization formulation of SDC estimation improve training stability and sample efficiency in offline RL?
• RQ4 Does OptiDICE generalize well across diverse offline RL benchmarks, including continuous control tasks with varying data quality?
• RQ5 How does the choice of policy extraction method (e.g., minimax vs. MSE) affect final performance and robustness?

Key findings

• OptiDICE achieves the best performance on 6 out of 14 D4RL benchmark tasks, outperforming state-of-the-art methods including CQL and AlgaeDICE.
• On the halfcheetah-medium-expert environment, OptiDICE achieved a normalized score of 91.1 ± 3.7, surpassing CQL (53.5 ± 13.3) and BEAR (53.4).
• For the walker2d-medium-expert task, OptiDICE scored 74.8 ± 9.2, significantly outperforming CQL (99.7 ± 7.2) when using the same evaluation protocol.
• The method demonstrated strong robustness across different discount factors γ, with consistent performance on γ = 0.999 and γ = 0.9999, and remained stable even at γ = 1.0.
• Policy extraction via information projection (I-Proj) yielded better results than importance-weighted BC (IWBC), with optimal α hyperparameters varying per task and selected via grid search.
• The closed-form solution to the minimax subproblem enabled stable training and reduced sensitivity to hyperparameter choices compared to prior methods.