[Paper Review] Learn What Not to Learn: Action Elimination with Deep Reinforcement Learning

Tom Zahavy, Matan Haroush+3 more|arXiv (Cornell University)|Sep 6, 2018

Reinforcement Learning in Robotics54 references77 citations

TL;DR

The paper introduces AE-DQN, a deep RL architecture that jointly learns a DQN and an Action Elimination Network (AEN) to prune suboptimal actions using an elimination signal, speeding up learning in large action spaces such as text-based games like Zork.

ABSTRACT

Learning how to act when there are many available actions in each state is a challenging task for Reinforcement Learning (RL) agents, especially when many of the actions are redundant or irrelevant. In such cases, it is sometimes easier to learn which actions not to take. In this work, we propose the Action-Elimination Deep Q-Network (AE-DQN) architecture that combines a Deep RL algorithm with an Action Elimination Network (AEN) that eliminates sub-optimal actions. The AEN is trained to predict invalid actions, supervised by an external elimination signal provided by the environment. Simulations demonstrate a considerable speedup and added robustness over vanilla DQN in text-based games with over a thousand discrete actions.

Motivation & Objective

• Motivate learning policies in environments with extremely large action spaces where many actions are redundant or irrelevant.
• Propose a framework that uses an elimination signal to prune admissible actions, reducing sample complexity and improving robustness in deep RL.
• Develop and evaluate a dual-network architecture (DQN + AEN) that jointly learns action values and elimination rules in NLP-based tasks.

Proposed method

• Introduce Action-Elimination Deep Q-Network (AE-DQN) that trains a DQN and an Action Elimination Network (AEN) concurrently.
• Use the AEN to predict invalid actions via an elimination signal provided by the environment; derive an admissible action set for the DQN.
• Represent states and actions with NLP-oriented CNNs; use the last-layer activations of AEN as features for a linear contextual bandit to decide elimination.
• Construct a batch-updates framework where a contextual bandit model is periodically updated from AEN activations to decouple elimination from the MDP learning.
• Define action elimination criteria based on concentration bounds from linear contextual bandits to ensure valid-action preservation with high probability.
• Provide an algorithm (AE-DQN) that interleaves standard Q-learning updates with elimination-based action pruning, using epsilon-greedy exploration over admissible actions.
• Evaluate on text-based Zork domain with thousands of actions, including subdomains Egg Quest and Troll Quest, and compare against vanilla DQN and baselines.

Experimental results

Research questions

• RQ1 Does action elimination via a learned elimination signal speed up learning in large action spaces compared with standard DQN?
• RQ2 Can a decoupled contextual bandit model reliably identify invalid actions with high probability without eliminating valid actions?
• RQ3 How does AE-DQN perform in text-based games with combinatorially large action spaces, and how robust is it to hyperparameters?
• RQ4 What is the impact of action elimination on sample efficiency and final performance in large-scale NLP-action environments?

Key findings

• AE-DQN outperforms vanilla DQN in large action space settings (e.g., Zork) by learning faster and achieving higher cumulative rewards.
• Action elimination yields robustness to hyperparameter settings, particularly when the action set is very large (e.g., Egg Troll subdomains).
• Using AEN activations to form a contextual bandit enables reliable elimination of suboptimal actions, reducing exploration of invalid actions.
• The modular elimination approach reduces sample complexity and converges to high-performing policies when sufficient elimination accuracy is achieved.
• In Zork and its subdomains, AE-DQN achieves state-of-the-art or competitive results compared to prior work using much larger action templates.