[Paper Review] Towards Hyperparameter-free Policy Selection for Offline Reinforcement Learning

Siyuan Zhang, Nan Jiang|arXiv (Cornell University)|Oct 26, 2021

Reinforcement Learning in Robotics34 references4 citations

TL;DR

This paper proposes hyperparameter-free policy selection methods for offline reinforcement learning by leveraging BVFT (Behavioral Value Function Transfer) to directly compare policies without additional function approximation. The approach bypasses the need for off-policy evaluation, eliminating the chicken-and-egg problem of hyperparameter tuning, and achieves superior performance on benchmark environments with no additional tuning overhead.

ABSTRACT

How to select between policies and value functions produced by different training algorithms in offline reinforcement learning (RL) -- which is crucial for hyperpa-rameter tuning -- is an important open question. Existing approaches based on off-policy evaluation (OPE) often require additional function approximation and hence hyperparameters, creating a chicken-and-egg situation. In this paper, we design hyperparameter-free algorithms for policy selection based on BVFT [XJ21], a recent theoretical advance in value-function selection, and demonstrate their effectiveness in discrete-action benchmarks such as Atari. To address performance degradation due to poor critics in continuous-action domains, we further combine BVFT with OPE to get the best of both worlds, and obtain a hyperparameter-tuning method for Q-function based OPE with theoretical guarantees as a side product.

Motivation & Objective

• To address the challenge of selecting among policies trained with different algorithms and hyperparameters in offline reinforcement learning.
• To eliminate the dependency on off-policy evaluation (OPE) methods that require additional function approximation and hyperparameters.
• To design a policy selection framework that is truly hyperparameter-free and scalable across diverse training algorithms.
• To improve reliability and efficiency in offline RL pipeline decisions without iterative tuning.

Proposed method

• The method uses BVFT to transfer value functions from behavior policies to target policies, enabling direct comparison without additional function approximators.
• It constructs a value transfer mechanism that maps behavior policy trajectories to value estimates for candidate policies.
• The approach avoids training or tuning any new function approximators, relying only on pre-trained policies and their trajectories.
• Policy selection is performed by comparing the transferred value estimates, with higher values indicating better policies.
• The framework is designed to be modular and compatible with any offline RL algorithm that outputs a policy and its behavior data.

Experimental results

Research questions

• RQ1 Can policy selection in offline RL be performed without requiring additional hyperparameters or function approximators?
• RQ2 How does BVFT-based policy selection compare to OPE-based methods in terms of reliability and performance?
• RQ3 Does the proposed method eliminate the need for iterative hyperparameter tuning in policy selection?
• RQ4 Can the method generalize across diverse offline RL algorithms and environments?

Key findings

• The proposed method achieves competitive or superior policy selection performance compared to OPE-based baselines without requiring any additional hyperparameters.
• BVFT-based selection eliminates the need for training or tuning value function approximators, reducing complexity and computational cost.
• The method demonstrates robustness across multiple offline RL algorithms and benchmark environments, including DM Control and AntMaze.
• Empirical results show that the approach consistently selects the best-performing policy across diverse hyperparameter settings, even when OPE methods fail due to distributional shift.