[Paper Review] Quantity vs. Quality: On Hyperparameter Optimization for Deep Reinforcement Learning

Lars Hertel, Pierre Baldi+1 more|arXiv (Cornell University)|Jul 29, 2020

Advanced Bandit Algorithms Research34 references4 citations

TL;DR

This paper investigates hyperparameter optimization in deep reinforcement learning, comparing quantity-driven methods (e.g., Successive Halving) against quality-focused approaches using repetitions and Bayesian optimization. It finds that Bayesian optimization with a noise-robust acquisition function (qNEI) outperforms all others, while pruning and repeated evaluations do not improve performance across random seeds.

ABSTRACT

Reinforcement learning algorithms can show strong variation in performance between training runs with different random seeds. In this paper we explore how this affects hyperparameter optimization when the goal is to find hyperparameter settings that perform well across random seeds. In particular, we benchmark whether it is better to explore a large quantity of hyperparameter settings via pruning of bad performers, or if it is better to aim for quality of collected results by using repetitions. For this we consider the Successive Halving, Random Search, and Bayesian Optimization algorithms, the latter two with and without repetitions. We apply these to tuning the PPO2 algorithm on the Cartpole balancing task and the Inverted Pendulum Swing-up task. We demonstrate that pruning may negatively affect the optimization and that repeated sampling does not help in finding hyperparameter settings that perform better across random seeds. From our experiments we conclude that Bayesian optimization with a noise robust acquisition function is the best choice for hyperparameter optimization in reinforcement learning tasks.

Motivation & Objective

• To evaluate whether hyperparameter optimization in deep reinforcement learning should prioritize exploring many configurations (quantity) or improving measurement quality via repetitions.
• To assess the impact of performance variability across random seeds on hyperparameter selection and optimization outcomes.
• To determine whether model-free methods like Random Search and Successive Halving, or model-based Bayesian optimization, are more effective under high observation noise.
• To investigate whether repeated evaluations per hyperparameter setting improve the robustness of hyperparameter selection across random seeds.
• To identify the most effective hyperparameter optimization strategy for achieving consistent performance across random seeds in deep RL tasks.

Proposed method

• Employs Bayesian optimization with noise-robust acquisition functions, specifically qNEI (quasi-Newton Expected Improvement), to model uncertainty in hyperparameter evaluations.
• Compares four main hyperparameter optimization strategies: Random Search, Successive Halving (ASHA), and Bayesian optimization using Expected Improvement (EI) and Lower Confidence Bound (LCB), both with and without repeated evaluations.
• Applies these methods to tune PPO2 on two continuous control tasks: CartPole balancing and Inverted Pendulum Swing-up, with hyperparameters including log-learning-rate and log-entropy-coefficient.
• Uses repeated evaluations (K=1, 3, 5) per hyperparameter configuration to estimate the true expected reward, with performance averaged across 15 hyperparameter optimization runs.
• Employs a fixed computational budget, evaluating 25, 50, and 100 agents per run, and reports median and mean rewards across 20 training runs per best-found configuration.
• Analyzes results using boxplots and statistical summaries to compare distributional performance across methods and tasks.

Experimental results

Research questions

• RQ1 Does pruning hyperparameter configurations via Successive Halving lead to better performance than Bayesian optimization in the presence of high variance across random seeds?
• RQ2 Does increasing the number of repeated evaluations per hyperparameter setting improve the robustness and accuracy of hyperparameter selection in deep reinforcement learning?
• RQ3 How do different acquisition functions in Bayesian optimization (e.g., EI vs. LCB vs. qNEI) perform when observations are noisy due to random seed variation?
• RQ4 Is there a significant performance difference between model-free and model-based hyperparameter optimization methods when the goal is to find settings that generalize across random seeds?
• RQ5 Can repeated evaluations in Bayesian optimization improve the selection of hyperparameters that yield high mean performance across random seeds?

Key findings

• Bayesian optimization with the noise-robust qNEI acquisition function (BoTorch-qNEI) achieved the highest median and mean performance on both the CartPole and Inverted Pendulum tasks.
• On the Inverted Pendulum Swing-up task, BoTorch-qNEI achieved a mean reward of -387.3 after 100 evaluations, significantly outperforming all other methods including GPyOpt-LCB (-442.4) and Random Search x5 (-444.6).
• Random Search with a single evaluation outperformed Random Search with three or five repetitions when computational budgets were matched, indicating that repetition does not improve performance.
• Successive Halving (ASHA) did not outperform Bayesian optimization and showed inconsistent performance, particularly under high variance.
• Repetitions did not improve Bayesian optimization performance, as GPyOpt-EI x3 and GPyOpt-LCB x3 showed no advantage over their single-evaluation counterparts.
• The distribution of outcomes for BoTorch-qNEI was consistently superior and less overlapping with other methods, especially after 50 and 100 evaluations, indicating robustness and reliability.