[Paper Review] Confidence-Conditioned Value Functions for Offline Reinforcement Learning

Joey Hong, Aviral Kumar+1 more|arXiv (Cornell University)|Dec 8, 2022

Reinforcement Learning in Robotics4 citations

TL;DR

This paper proposes Confidence-Conditioned Value Learning (CCVL), an offline reinforcement learning method that learns value functions conditioned on confidence levels to enable dynamic conservatism during evaluation. By conditioning Q-values on desired confidence levels, CCVL allows policies to adapt their pessimism using online observations, outperforming static conservative methods like CQL on Atari environments with significant gains in sample efficiency and final performance.

ABSTRACT

Offline reinforcement learning (RL) promises the ability to learn effective policies solely using existing, static datasets, without any costly online interaction. To do so, offline RL methods must handle distributional shift between the dataset and the learned policy. The most common approach is to learn conservative, or lower-bound, value functions, which underestimate the return of out-of-distribution (OOD) actions. However, such methods exhibit one notable drawback: policies optimized on such value functions can only behave according to a fixed, possibly suboptimal, degree of conservatism. However, this can be alleviated if we instead are able to learn policies for varying degrees of conservatism at training time and devise a method to dynamically choose one of them during evaluation. To do so, in this work, we propose learning value functions that additionally condition on the degree of conservatism, which we dub confidence-conditioned value functions. We derive a new form of a Bellman backup that simultaneously learns Q-values for any degree of confidence with high probability. By conditioning on confidence, our value functions enable adaptive strategies during online evaluation by controlling for confidence level using the history of observations thus far. This approach can be implemented in practice by conditioning the Q-function from existing conservative algorithms on the confidence.We theoretically show that our learned value functions produce conservative estimates of the true value at any desired confidence. Finally, we empirically show that our algorithm outperforms existing conservative offline RL algorithms on multiple discrete control domains.

Motivation & Objective

• To address the limitation of fixed conservatism in existing conservative offline RL methods, which rely on static hyperparameters for pessimism.
• To enable dynamic adaptation of conservatism during online evaluation using observed data, rather than fixed offline tuning.
• To develop a practical and theoretically grounded method for learning value functions that provide conservative estimates at any desired confidence level.
• To improve sample efficiency and final performance in offline RL by allowing policies to adjust their risk tolerance based on real-time observation history.
• To extend offline RL to support online fine-tuning by learning both lower- and upper-bound value estimates.

Proposed method

• Proposes confidence-conditioned value functions that map state-action pairs to Q-values conditioned on a confidence level δ, representing the probability that the estimate is a lower-bound on the true return.
• Derives a novel Bellman backup that simultaneously learns conservative Q-values for all δ by enforcing a confidence-aware regularization objective.
• Implements the method by conditioning the Q-network on δ using parameter sharing and distributional RL-style head architectures, enabling end-to-end training.
• Adapts the confidence level δ at inference time based on online observation history, allowing policies to reduce pessimism when evidence suggests overestimation.
• Extends the framework to learn both lower- and upper-bound estimates, facilitating online fine-tuning with improved exploration.
• Uses conservative Q-learning (CQL)-style regularization to ensure conservative estimates across all confidence levels without relying on anti-exploration bonuses.

Experimental results

Research questions

• RQ1 Can offline RL policies dynamically adjust their level of conservatism during evaluation based on observed data, rather than relying on a fixed hyperparameter?
• RQ2 Does learning value functions conditioned on confidence levels lead to better performance than fixed-conservatism methods in offline RL?
• RQ3 Can confidence-adaptive policies improve sample efficiency and final performance in low-data regimes?
• RQ4 Is the proposed method theoretically sound, ensuring conservative estimates at any confidence level under tabular settings?
• RQ5 Can the confidence-conditioned framework be extended to support online fine-tuning with improved exploration?

Key findings

• CCVL outperforms CQL and REM on all three data regimes (1%, 5%, 10%) in Atari environments, achieving a final normalized IQM of 59.1 on 1% data, compared to 56.9 for CQL.
• On 5% data, CCVL achieves 110.1 normalized IQM, significantly outperforming CQL (105.7) and Fixed-CCVL (105.9), demonstrating the benefit of adaptive confidence.
• In online fine-tuning, CCVL achieves the highest final performance on 4 out of 5 Atari games, with a 159.5 score on Asterix (vs. 52.7 for CQL), showing strong improvement with additional online data.
• The ablation study confirms that both confidence-conditioning and confidence adaptation are essential: CCVL outperforms both Fixed-CCVL and AEVL, which lack adaptive confidence control.
• CCVL’s performance is robust across seeds, with 95% bootstrap CIs showing consistent superiority over baselines, especially in low-data regimes.
• The method enables effective online fine-tuning by learning both lower- and upper-bound estimates, allowing conservative behavior offline and optimistic exploration online.