[Paper Review] Exploratory Combinatorial Optimization with Reinforcement Learning

Thomas D. Barrett, William R. Clements+2 more|arXiv (Cornell University)|Sep 9, 2019

Metaheuristic Optimization Algorithms Research35 references21 citations

TL;DR

This paper proposes ECO-DQN, a reinforcement learning framework for exploratory combinatorial optimization that allows agents to iteratively improve solutions by dynamically adding or removing vertices from a solution subset, unlike prior methods that build solutions irreversibly. The method achieves state-of-the-art performance on the Maximum Cut problem by enabling reversible decisions and leveraging graph neural networks to guide exploration, with strong generalization to unseen graph distributions.

ABSTRACT

Many real-world problems can be reduced to combinatorial optimization on a graph, where the subset or ordering of vertices that maximize some objective function must be found. With such tasks often NP-hard and analytically intractable, reinforcement learning (RL) has shown promise as a framework with which efficient heuristic methods to tackle these problems can be learned. Previous works construct the solution subset incrementally, adding one element at a time, however, the irreversible nature of this approach prevents the agent from revising its earlier decisions, which may be necessary given the complexity of the optimization task. We instead propose that the agent should seek to continuously improve the solution by learning to explore at test time. Our approach of exploratory combinatorial optimization (ECO-DQN) is, in principle, applicable to any combinatorial problem that can be defined on a graph. Experimentally, we show our method to produce state-of-the-art RL performance on the Maximum Cut problem. Moreover, because ECO-DQN can start from any arbitrary configuration, it can be combined with other search methods to further improve performance, which we demonstrate using a simple random search.

Motivation & Objective

• To address the limitation of irreversible solution construction in existing RL-based combinatorial optimization methods.
• To enable agents to explore and improve solutions at test time by allowing both addition and removal of vertices from the solution set.
• To develop a flexible framework applicable to any graph-based combinatorial optimization problem where solutions are subsets of vertices.
• To improve performance on the Maximum Cut problem through continuous exploration and dynamic decision-making.
• To demonstrate generalization to unseen graph distributions and compatibility with other search heuristics.

Proposed method

• The framework uses a deep Q-network (DQN) with a message-passing neural network (MPNN) to encode graph structure and compute Q-values for each vertex.
• At each step, the agent selects a vertex to either add to or remove from the current solution set, enabling reversible, exploratory behavior.
• The Q-value for each vertex is computed using a graph embedding that aggregates information from neighbors and the global graph state.
• The MPNN uses three rounds of message passing with learnable weights to update vertex embeddings based on local neighborhood and edge features.
• The Q-value head combines a global graph embedding with individual vertex embeddings to predict action values.
• Training uses experience replay, target networks, and clipped Q-value predictions to stabilize learning.

Experimental results

Research questions

• RQ1 Can a reinforcement learning agent achieve better performance on combinatorial optimization problems by allowing reversible decisions during inference?
• RQ2 Does the ability to explore the solution space dynamically improve solution quality compared to irreversible, incremental construction?
• RQ3 Can the proposed ECO-DQN framework generalize to graphs of different sizes and structures than those seen during training?
• RQ4 How does ECO-DQN perform relative to S2V-DQN, a prior state-of-the-art method, on the Maximum Cut problem?
• RQ5 Can ECO-DQN be effectively combined with other search heuristics, such as random initialization, to further improve performance?

Key findings

• ECO-DQN achieves state-of-the-art performance on the Maximum Cut problem, outperforming the S2V-DQN baseline.
• The performance gain is attributed to the agent's ability to reverse earlier decisions and explore the solution space dynamically.
• ECO-DQN generalizes well to graphs of up to 2000 vertices, even when trained on smaller, differently structured graphs.
• The method maintains similar computational cost per step as S2V-DQN, with a time per action of 6.64±0.09 ms on 500-node graphs.
• Combining ECO-DQN with random initialization improves performance significantly, demonstrating compatibility with other search strategies.
• Ablation studies confirm that both reversible actions and proper reward shaping are essential for the observed performance gains.