[Paper Review] HyAR: Addressing Discrete-Continuous Action Reinforcement Learning via Hybrid Action Representation

Boyan Li, Hongyao Tang+6 more|arXiv (Cornell University)|Sep 12, 2021

Reinforcement Learning in Robotics10 citations

TL;DR

This paper proposes HyAR, a hybrid action representation framework that learns a compact, disentangled latent space for discrete-continuous action spaces using a conditional VAE and action embedding table. By modeling dependencies between discrete and continuous components and training representations via unsupervised dynamics prediction, HyAR enables effective deep reinforcement learning in complex, high-dimensional hybrid action environments, outperforming prior methods in both sample efficiency and final performance.

ABSTRACT

Discrete-continuous hybrid action space is a natural setting in many practical problems, such as robot control and game AI. However, most previous Reinforcement Learning (RL) works only demonstrate the success in controlling with either discrete or continuous action space, while seldom take into account the hybrid action space. One naive way to address hybrid action RL is to convert the hybrid action space into a unified homogeneous action space by discretization or continualization, so that conventional RL algorithms can be applied. However, this ignores the underlying structure of hybrid action space and also induces the scalability issue and additional approximation difficulties, thus leading to degenerated results. In this paper, we propose Hybrid Action Representation (HyAR) to learn a compact and decodable latent representation space for the original hybrid action space. HyAR constructs the latent space and embeds the dependence between discrete action and continuous parameter via an embedding table and conditional Variantional Auto-Encoder (VAE). To further improve the effectiveness, the action representation is trained to be semantically smooth through unsupervised environmental dynamics prediction. Finally, the agent then learns its policy with conventional DRL algorithms in the learned representation space and interacts with the environment by decoding the hybrid action embeddings to the original action space. We evaluate HyAR in a variety of environments with discrete-continuous action space. The results demonstrate the superiority of HyAR when compared with previous baselines, especially for high-dimensional action spaces.

Motivation & Objective

• To address the limitations of existing RL methods in handling discrete-continuous hybrid action spaces, which often ignore structural dependencies or suffer from scalability issues.
• To develop a unified, compact, and semantically smooth latent representation space that preserves the dependence between discrete actions and their continuous parameters.
• To improve sample efficiency and policy generalization in high-dimensional hybrid action spaces by leveraging unsupervised dynamics prediction during representation learning.
• To enable conventional DRL algorithms to operate effectively in the learned latent space while decoding back to the original action space for environment interaction.

Proposed method

• HyAR constructs a hybrid action representation using an embedding table for discrete actions and a conditional Variational Auto-Encoder (VAE) to model the dependence between discrete actions and their continuous parameters.
• The latent space is trained to be semantically smooth through unsupervised prediction of environmental dynamics, enhancing policy generalization.
• A two-stage training process is employed: first, a warm-up phase collects demonstrations to pre-train the representation; second, the policy is trained in the latent space using standard DRL algorithms like TD3.
• The agent decodes the learned latent embeddings back into the original hybrid action space for interaction with the environment.
• The method supports both structured and non-structured hybrid action spaces by formulating them under the Parameterized Action MDP (PAMDP) framework.

Experimental results

Research questions

• RQ1 Can a compact and disentangled latent representation effectively model the dependence between discrete actions and their continuous parameters in hybrid action spaces?
• RQ2 Does learning semantically smooth representations via unsupervised dynamics prediction improve sample efficiency and policy performance in hybrid action RL?
• RQ3 How does HyAR compare to existing methods like PADDPG, HPPO, and PDQN in terms of scalability and performance on high-dimensional hybrid action tasks?
• RQ4 To what extent does the choice of representation training frequency affect policy learning in HyAR?

Key findings

• HyAR outperforms baseline methods, including PADDPG, HPPO, and PDQN, across all evaluated environments, especially in high-dimensional action spaces.
• The unfixed HyAR variant, which continues updating the representation during policy training, consistently outperforms the fixed representation baseline, which performs poorly in complex environments like Goal and Hard Goal.
• A moderate training frequency for subsequent representation updates (e.g., every 10 episodes) yields the best performance, balancing representation stability and adaptability.
• In environments like Hard Move with 8 discrete actions, HyAR achieves significantly higher final rewards and faster convergence than fixed representation baselines.
• The ablation study confirms that modeling the dependence between discrete and continuous components is critical for performance, as methods ignoring this structure underperform.