[Paper Review] Discrete and Continuous Action Representation for Practical RL in Video Games

Olivier Delalleau, Maxim Peter+2 more|arXiv (Cornell University)|Dec 23, 2019

Human Pose and Action RecognitionComputer Science26 references39 citations

TL;DR

The paper introduces Hybrid SAC, an extension of Soft Actor-Critic that can handle discrete, continuous, and mixed discrete-continuous actions, and evaluates its performance on parameterized action benchmarks and a commercial video game; it also studies normalizing flows with SAC.

ABSTRACT

While most current research in Reinforcement Learning (RL) focuses on improving the performance of the algorithms in controlled environments, the use of RL under constraints like those met in the video game industry is rarely studied. Operating under such constraints, we propose Hybrid SAC, an extension of the Soft Actor-Critic algorithm able to handle discrete, continuous and parameterized actions in a principled way. We show that Hybrid SAC can successfully solve a highspeed driving task in one of our games, and is competitive with the state-of-the-art on parameterized actions benchmark tasks. We also explore the impact of using normalizing flows to enrich the expressiveness of the policy at minimal computational cost, and identify a potential undesired effect of SAC when used with normalizing flows, that may be addressed by optimizing a different objective.

Motivation & Objective

• Motivate reinforcement learning use in industry settings with data and runtime constraints typical of video game development.
• Develop a practical off-policy algorithm (Hybrid SAC) that handles discrete, continuous, and mixed actions within SAC.
• Assess how normalizing flows affect policy expressiveness and learning under SAC objectives in realistic tasks.
• Compare Hybrid SAC to state-of-the-art methods on parameterized action benchmarks.
• Demonstrate applicability to a real Ubisoft game scenario to show industry relevance.

Proposed method

• Propose a general action representation decomposing pi(a|s) into pi(a^d|s) and pi(a^c|s,a^d) to accommodate mixed action types.
• Extend SAC to Hybrid SAC by learning both a discrete action distribution and a continuous action conditioned on the discrete choice, with a critic predicting Q-values for discrete actions.
• Use a weighted entropy bonus alpha^d H(pi(a^d|s)) + alpha^c sum_{a^d} pi(a^d|s) H(pi(a^c|s,a^d)) to balance exploration across discrete and continuous parts.
• Provide network architectures where the actor outputs either separate discrete distributions and continuous parameters or shared representations for multiple components.
• Explore several practical policy parameterizations and discuss when to duplicate continuous components per discrete action to simplify learning.
• Experiment with normalizing flows on top of Gaussian policies to enrich expressiveness, while noting potential collapses under SAC objectives.

Experimental results

Research questions

• RQ1 Can Hybrid SAC effectively learn policies with mixed discrete and continuous actions in practical video game tasks?
• RQ2 How does the proposed action decomposition affect data efficiency and learning performance compared to purely discrete or purely continuous baselines?
• RQ3 Do normalizing flows improve or degrade SAC performance in practice, and under what objective formulations?
• RQ4 What architectural and parameterization choices best scale to industry-like constraints and parameterized action spaces?
• RQ5 How does Hybrid SAC perform on parameterized action benchmarks relative to state-of-the-art methods like MP-DQN?

Key findings

• Hybrid SAC achieves competitive performance on parameterized action benchmarks, matching MP-DQN on Platform and close to MP-DQN on Goal, with some gaps on HFO.
• In a commercial game, Hybrid SAC demonstrated successful high-speed driving with discrete hand brake plus continuous acceleration and steering.
• Using normalizing flows with SAC did not consistently outperform the Gaussian policy on Roboschool benchmarks, suggesting the need for alternative objectives beyond KL for flows.
• A potential issue was identified where entropy bonuses could cause continuous parameter collapse for small discrete action probabilities, with preliminary attempts to address this aligning with MP-DQN results.
• Experiments show the importance of policy representation choice; duplicating continuous parameters per discrete action can aid learning when the discrete set is manageable.