[Paper Review] Evolving Rewards to Automate Reinforcement Learning

Aleksandra Faust, Anthony Francis+1 more|arXiv (Cornell University)|May 18, 2019

Reinforcement Learning in Robotics26 references22 citations

TL;DR

This paper proposes AutoRL, an evolutionary reinforcement learning framework that automates reward engineering by treating reward tuning as hyperparameter optimization. It evolves proxy rewards using a population-based search over parameterized reward functions, achieving superior performance—up to 489% improvement—on complex Mujoco tasks compared to hand-tuned and hyperparameter-tuned baselines, especially with SAC on Humanoid.

ABSTRACT

Many continuous control tasks have easily formulated objectives, yet using them directly as a reward in reinforcement learning (RL) leads to suboptimal policies. Therefore, many classical control tasks guide RL training using complex rewards, which require tedious hand-tuning. We automate the reward search with AutoRL, an evolutionary layer over standard RL that treats reward tuning as hyperparameter optimization and trains a population of RL agents to find a reward that maximizes the task objective. AutoRL, evaluated on four Mujoco continuous control tasks over two RL algorithms, shows improvements over baselines, with the the biggest uplift for more complex tasks. The video can be found at: \url{https://youtu.be/svdaOFfQyC8}.

Motivation & Objective

• Automate the tedious process of reward shaping in continuous control tasks, which currently requires extensive manual tuning.
• Address the challenge of sparse or poorly structured rewards that hinder sample-efficient learning in high-dimensional control.
• Reduce reliance on hand-designed multi-objective rewards by discovering effective proxy rewards through automated optimization.
• Evaluate whether evolutionary reward search can outperform traditional hyperparameter tuning and manual reward engineering across diverse RL algorithms and tasks.
• Demonstrate that optimizing for simpler, single-task objectives (e.g., distance traveled) can yield policies comparable to those trained on complex, standard returns.

Proposed method

• Apply an evolutionary algorithm (EA) to search over parameterized reward functions, treating reward weights as hyperparameters to be optimized.
• Train a population of RL agents in parallel, where each agent uses a different reward configuration derived from a parameterized version of the standard environment reward.
• Use the task objective (e.g., distance traveled, height reached) or standard return as the fitness signal to evaluate and select the best-performing reward configurations.
• Optimize reward parameters via evolutionary strategies such as mutation and selection, iteratively improving the reward function based on agent performance.
• Integrate the evolved reward into standard RL algorithms (SAC and PPO) for training, enabling end-to-end comparison with hand-tuned and hyperparameter-tuned baselines.
• Conduct cross-evaluation between policies trained on task objectives and standard returns to assess generalization and performance parity.

Experimental results

Research questions

• RQ1 Can an automated evolutionary approach to reward design outperform manual reward shaping and hyperparameter tuning in continuous control tasks?
• RQ2 Does optimizing for a simpler, single-task objective (e.g., distance traveled) yield policies that perform as well as those trained on complex, multi-objective standard returns?
• RQ3 How does AutoRL’s reward search compare to traditional hyperparameter tuning in terms of sample efficiency and final policy performance?
• RQ4 Can AutoRL generalize across different RL algorithms (SAC vs. PPO) and environments (Ant, Walker2D, HumanoidStandup, Humanoid)?
• RQ5 Does the evolved reward lead to faster convergence and higher-quality policies than baseline methods under the same training budget?

Key findings

• AutoRL achieved a 489% performance gain over hyperparameter tuning on the Humanoid task using SAC when optimizing for a single-task objective (distance traveled).
• Policies trained with AutoRL on single-task objectives performed comparably to those trained on standard multi-objective returns, reducing the need for complex reward engineering.
• AutoRL outperformed both hand-tuned and hyperparameter-tuned baselines across all four Mujoco environments, with the largest gains on more complex tasks.
• For SAC, AutoRL produced the fastest and most stable training, with agents achieving higher average returns and better behavioral consistency than PPO-based baselines.
• On HumanoidStandup, AutoRL with SAC was the only method to fully stand, while others only achieved a consistent crouch, indicating superior policy quality.
• Cross-evaluation showed that policies optimized on task objectives generalized well to standard return evaluation, confirming that simpler objectives can effectively proxy complex reward functions.