[Paper Review] Hybrid Reward Architecture for Reinforcement Learning

Harm van Seijen, Mehdi Fatemi+4 more|arXiv (Cornell University)|Jun 13, 2017

Reinforcement Learning in Robotics64 references188 citations

TL;DR

The paper introduces Hybrid Reward Architecture (HRA), decomposing environment rewards into multiple components and learning separate value functions for each, then aggregating them to improve learning in complex domains like Ms. Pac-Man, achieving above-human performance.

ABSTRACT

One of the main challenges in reinforcement learning (RL) is generalisation. In typical deep RL methods this is achieved by approximating the optimal value function with a low-dimensional representation using a deep network. While this approach works well in many domains, in domains where the optimal value function cannot easily be reduced to a low-dimensional representation, learning can be very slow and unstable. This paper contributes towards tackling such challenging domains, by proposing a new method, called Hybrid Reward Architecture (HRA). HRA takes as input a decomposed reward function and learns a separate value function for each component reward function. Because each component typically only depends on a subset of all features, the corresponding value function can be approximated more easily by a low-dimensional representation, enabling more effective learning. We demonstrate HRA on a toy-problem and the Atari game Ms. Pac-Man, where HRA achieves above-human performance.

Motivation & Objective

• Motivate learning in high-complexity RL domains where the optimal value function is hard to approximate with a low-dimensional representation.
• Propose decomposing the environment reward into multiple component rewards learned by separate agents (heads).
• Demonstrate that aggregating component value functions can yield superior policies and faster learning compared to single-head baselines.

Proposed method

• Decompose the environment reward R_env into n component rewards R_k(s,a,s') for k=1..n (Eq. 4).
• Train a separate RL agent for each component reward, sharing a common network but with multiple heads representing Q_k(s,a;θ).
• Aggregate heads to form Q_HRA(s,a;θ) = sum_k Q_k(s,a;θ) (Eq. 5).
• Update each head with its own target y_k,i = R_k(s,a,s') + γ max_a' Q_k(s',a';θ_{i-1}) (Eq. 7).
• Explore training targets: Q_env* (default, consistent) vs. Q_HRA^ν (semi-consistent, learned via expected SARSA with random policy) (Eq. 8).
• Optionally incorporate domain knowledge (feature filtering, terminal-state handling, pseudo-rewards, etc.) to improve learning efficiency (Section 3.2).

Experimental results

Research questions

• RQ1 Can decomposing the environment reward into multiple simpler components and learning separate value functions accelerate learning in hard RL domains?
• RQ2 Does aggregating the component value functions yield competitive or superior policies compared to standard single-head methods?
• RQ3 How does HRA perform on toy tasks and on challenging Atari games like Ms. Pac-Man, including with added domain knowledge?
• RQ4 What training targets (Q_env* vs. Q_HRA^ν) are most effective in guiding learning?
• RQ5 How can domain knowledge further enhance HRA performance?

Key findings

• HRA outperforms a DQN baseline on the fruit collection task and on Ms. Pac-Man, especially when leveraging domain knowledge.
• Using Q_HRA^ν (random-policy-based target) can provide a semi-consistent training target that aids learning in navigation-based domains.
• In the fruit task, adding domain knowledge (removing irrelevant features, identifying terminal states, using pseudo-rewards) yields large performance gains, with tabular representations enabling near-optimal behavior.
• On Ms. Pac-Man, HRA with reward decomposition and GVFs achieves substantially higher scores than A3C baselines and human benchmarks under both fixed-start and random-start evaluations, including surpassing human high-scores and reaching very high maximum scores with executive-memory-inspired augmentation.
• HRA scales by creating many GVFs with small state-spaces (GVF state-space ~10^3 per head), enabling effective learning where a single compact representation struggles (Ms. Pac-Man).