[Paper Review] Learning a Generic Value-Selection Heuristic Inside a Constraint Programming Solver

Marty, Tom, François, Tristan+4 more|arXiv (Cornell University)|Jun 16, 2017

Reinforcement Learning in Robotics28 references521 citations

TL;DR

This paper proposes Value-Decomposition Networks (VDN), a deep reinforcement learning architecture that learns to decompose a team-level value function into individual agent value functions, enabling cooperative multi-agent learning with only a single joint reward signal. By back-propagating the total Q-value gradient through individual networks, VDN mitigates the 'lazy agent' problem and spurious rewards, outperforming centralized and independent learning baselines across multiple partially-observable environments, especially when combined with weight sharing and information channels.

ABSTRACT

Constraint programming is known for being an efficient approach to solving combinatorial problems. Important design choices in a solver are the branching heuristics, designed to lead the search to the best solutions in a minimum amount of time. However, developing these heuristics is a time-consuming process that requires problem-specific expertise. This observation has motivated many efforts to use machine learning to automatically learn efficient heuristics without expert intervention. Although several generic variable-selection heuristics are available in the literature, the options for value-selection heuristics are more scarce. We propose to tackle this issue by introducing a generic learning procedure that can be used to obtain a value-selection heuristic inside a constraint programming solver. This has been achieved thanks to the combination of a deep Q-learning algorithm, a tailored reward signal, and a heterogeneous graph neural network. Experiments on graph coloring, maximum independent set, and maximum cut problems show that this framework competes with the well-known impact-based and activity-based search heuristics and can find solutions close to optimality without requiring a large number of backtracks.

Motivation & Objective

• To address the 'lazy agent' problem and spurious rewards in cooperative multi-agent reinforcement learning with only a joint reward signal.
• To develop a method that enables individual agents to learn effective policies without explicit reward shaping or full environment observability.
• To improve sample efficiency and coordination in partially-observable multi-agent environments using a learnable value decomposition mechanism.
• To evaluate the effectiveness of VDN in comparison to centralized and independent learning approaches across diverse benchmark tasks.
• To investigate the impact of auxiliary components such as weight sharing, role information, and communication channels on VDN performance.

Proposed method

• Proposes a novel value-decomposition network (VDN) that learns an additive decomposition of the joint Q-function into individual agent Q-functions.
• Uses end-to-end deep learning to back-propagate the total Q-value gradient through individual networks, enabling joint optimization of individual value functions.
• Employs a shared neural network architecture across agents (weight sharing) to improve sample efficiency and generalization.
• Introduces role information and information channels to enhance coordination, particularly in asymmetric tasks.
• Applies the VDN architecture in a centralized training with decentralized execution (CTDE) paradigm, allowing independent inference while training jointly.
• Uses experience replay and target networks to stabilize training, similar to deep Q-networks (DQN) but adapted for multi-agent settings.

Experimental results

Research questions

• RQ1 Can a learnable value decomposition mechanism outperform centralized and independent learning in cooperative multi-agent reinforcement learning with only a joint reward?
• RQ2 How does value decomposition mitigate the 'lazy agent' problem caused by partial observability and non-stationarity?
• RQ3 What is the impact of weight sharing, role information, and communication channels on the performance of value-decomposed agents?
• RQ4 Can VDN generalize across diverse partially-observable multi-agent environments without task-specific reward shaping?
• RQ5 Does value decomposition lead to more interpretable and disentangled value functions that reflect individual contributions to team rewards?

Key findings

• VDN consistently outperforms both centralized and independent learning baselines across all seven benchmark tasks, achieving significantly higher normalized area under the curve (AUC) and final performance.
• In the Fetch task with a single corridor, VDN with weight sharing and role information achieves perfect performance, while the non-sharing variant suffers from the 'lazy agent' problem.
• The addition of information channels improves learning speed and coordination, particularly in asymmetric tasks like Checkers, where agents have different reward magnitudes.
• The learned Q-decomposition successfully disentangles team rewards into agent-specific contributions, as shown in Figure 6, where agent Q-functions spike independently at pickup and drop-off events.
• VDN with shared weights and role information solves the one-corridor Fetch task perfectly, while independent learners and centralized agents fail to coordinate effectively.
• The low-level communication channels (processed by LSTMs) outperform high-level centralized communication, suggesting that early fusion of multi-agent observations enhances learning.