[Paper Review] Reinforcement Learning in Large Discrete Action Spaces.

Gabriel Dulac-Arnold, Richard Evans+2 more|arXiv (Cornell University)|Dec 24, 2015

Reinforcement Learning in Robotics17 references287 citations

TL;DR

This paper proposes a reinforcement learning method for large discrete action spaces by embedding actions into a continuous space and using approximate nearest-neighbor search for efficient action selection. The approach enables sub-linear inference complexity and generalization across actions, successfully scaling to tasks with up to one million actions where prior methods fail.

ABSTRACT

Being able to reason in an environment with a large number of discrete actions is essential to bringing reinforcement learning to a larger class of problems. Recommender systems, industrial plants and language models are only some of the many real-world tasks involving large numbers of discrete actions for which current methods are difficult or even often impossible to apply. An ability to generalize over the set of actions as well as sub-linear complexity relative to the size of the set are both necessary to handle such tasks. Current approaches are not able to provide both of these, which motivates the work in this paper. Our proposed approach leverages prior information about the actions to embed them in a continuous space upon which it can generalize. Additionally, approximate nearest-neighbor methods allow for logarithmic-time lookup complexity relative to the number of actions, which is necessary for time-wise tractable training. This combined approach allows reinforcement learning methods to be applied to large-scale learning problems previously intractable with current methods. We demonstrate our algorithm’s abilities on a series of tasks having up to one million actions.

Motivation & Objective

• Address the challenge of applying reinforcement learning to environments with extremely large discrete action spaces, such as recommender systems and language models.
• Overcome the limitations of existing methods that scale poorly with action set size and lack generalization across actions.
• Enable efficient, scalable training by achieving logarithmic-time action lookup relative to action count.
• Leverage prior knowledge about actions to create meaningful continuous representations for generalization.
• Demonstrate feasibility and performance on large-scale tasks with up to one million actions.

Proposed method

• Embed discrete actions into a continuous vector space using prior knowledge about their structure or semantics.
• Train a reinforcement learning agent that generalizes over the continuous action embedding space rather than individual actions.
• Use approximate nearest-neighbor (ANN) search to efficiently retrieve the most promising actions during inference.
• Achieve logarithmic-time complexity for action lookup, making training tractable even with millions of actions.
• Integrate the action embedding and ANN components into a standard RL framework, such as Q-learning or policy gradient methods.
• Train the agent end-to-end, jointly optimizing the policy and the action embedding space using gradient-based methods.

Experimental results

Research questions

• RQ1 Can action embeddings combined with approximate nearest-neighbor search enable efficient reinforcement learning in large discrete action spaces?
• RQ2 To what extent does learning in a continuous action embedding space improve generalization across actions compared to per-action learning?
• RQ3 How does the computational complexity of the proposed method scale with increasing action set size?
• RQ4 Can the method be applied to real-world tasks with up to one million actions, and how does it compare to baseline methods?
• RQ5 Does the use of prior knowledge about actions significantly improve sample efficiency and final performance?

Key findings

• The proposed method successfully trains reinforcement learning agents on environments with up to one million discrete actions, a regime previously intractable for standard RL algorithms.
• The use of action embeddings enables generalization across semantically similar actions, improving sample efficiency and policy performance.
• Approximate nearest-neighbor search reduces action lookup time to logarithmic complexity relative to the number of actions, enabling tractable training.
• The method outperforms baseline approaches that do not use action embeddings or efficient search, especially as action space size increases.
• The integration of prior knowledge into action embeddings leads to faster convergence and higher final return in large-scale tasks.
• The approach maintains strong performance even when action space size grows beyond the practical limits of standard RL methods.