[Paper Review] Q-Networks for Binary Vector Actions

Naoto Yoshida|arXiv (Cornell University)|Dec 4, 2015

Reinforcement Learning in Robotics8 references3 citations

TL;DR

This paper proposes a Q-network architecture that models the action-value function as a linear function of binary vector actions, enabling efficient greedy and softmax action selection in large discrete action spaces. By structuring the network to output a state-dependent scalar and vector, the method supports tractable Q-learning without Monte Carlo sampling, demonstrating effectiveness in grid world and blocker tasks with binary-coded actions.

ABSTRACT

In this paper reinforcement learning with binary vector actions was investigated. We suggest an effective architecture of the neural networks for approximating an action-value function with binary vector actions. The proposed architecture approximates the action-value function by a linear function with respect to the action vector, but is still non-linear with respect to the state input. We show that this approximation method enables the efficient calculation of greedy action selection and softmax action selection. Using this architecture, we suggest an online algorithm based on Q-learning. The empirical results in the grid world and the blocker task suggest that our approximation architecture would be effective for the RL problems with large discrete action sets.

Motivation & Objective

• Address the challenge of applying Q-learning to large discrete action spaces where action counts grow exponentially with binary vector length.
• Overcome the intractability of greedy action selection and exact Q-learning in such settings due to nonlinearity in standard function approximators.
• Develop a neural network architecture that enables efficient, exact computation of greedy and softmax actions without sampling.
• Enable off-policy Q-learning with function approximation in large discrete action domains using a tractable, non-Monte Carlo approach.

Proposed method

• Propose a Q-network architecture where the action-value function is modeled as $ Q_{\theta}(s,a) = \Psi_{\theta}(s) + a^\top \phi_{\theta}(s) $, linear in the binary action vector $ a $.
• Use a multilayer perceptron (MLP) to compute $ \Psi_{\theta}(s) $ and $ \phi_{\theta}(s) $, with the latter being a state-dependent vector.
• Enable efficient greedy action selection by computing $ \arg\max_a Q_{\theta}(s,a) = \arg\max_i \phi^i_{\theta}(s) $, avoiding full enumeration over all actions.
• Support softmax action selection via the same linear structure, allowing differentiable policy optimization.
• Apply stochastic gradient descent to update network parameters using the standard Q-learning TD error: $ \Delta\theta = (r + \gamma \max_{\hat{a}} Q_{\theta}(s^\prime,\hat{a}) - Q_{\theta}(s,a)) \frac{\partial Q_{\theta}(s,a)}{\partial\theta} $.
• Use the architecture in an online Q-learning algorithm without requiring Gibbs sampling or other approximations.

Experimental results

Research questions

• RQ1 Can a linear-in-action function approximation enable efficient and exact greedy action selection in large discrete action spaces with binary vector actions?
• RQ2 Is it possible to design a Q-network architecture that supports both greedy and softmax action selection without Monte Carlo sampling?
• RQ3 Can off-policy Q-learning be effectively applied to large discrete action spaces using this architecture, avoiding intractable optimization steps?
• RQ4 How does the proposed method compare to standard Q-learning and energy-based approaches in terms of sample efficiency and convergence in large action space environments?

Key findings

• The proposed architecture enables exact greedy action selection via $ \arg\max_i \phi^i_{\theta}(s) $, avoiding the exponential cost of enumerating all binary actions.
• The method supports efficient softmax action selection through the same linear structure, enabling differentiable policy learning.
• In the grid world environment, the agent achieved stable learning and reached near-optimal performance using the proposed Q-network.
• In the blocker task with 64 discrete actions (12-bit binary actions), the agent learned to coordinate and score points, with agent-wise $ \epsilon $-greedy exploration outperforming standard $ \epsilon $-greedy.
• The agent-wise $ \epsilon $-greedy policy helped escape local optima, showing improved robustness in multi-agent coordination.
• Empirical results confirm that the method enables tractable, off-policy Q-learning in large discrete action spaces without Monte Carlo approximations.