[Paper Review] Learning Combinatorial Optimization Algorithms over Graphs

Hanjun Dai, Elias B. Khalil+3 more|arXiv (Cornell University)|Apr 5, 2017

Optimization and Search ProblemsComputer Science28 references971 citations

TL;DR

This paper presents S2V-DQN, a framework that learns greedy heuristics for graph-based NP-hard optimization problems by combining reinforcement learning with graph embeddings, generalizing across problem types and sizes.

ABSTRACT

The design of good heuristics or approximation algorithms for NP-hard combinatorial optimization problems often requires significant specialized knowledge and trial-and-error. Can we automate this challenging, tedious process, and learn the algorithms instead? In many real-world applications, it is typically the case that the same optimization problem is solved again and again on a regular basis, maintaining the same problem structure but differing in the data. This provides an opportunity for learning heuristic algorithms that exploit the structure of such recurring problems. In this paper, we propose a unique combination of reinforcement learning and graph embedding to address this challenge. The learned greedy policy behaves like a meta-algorithm that incrementally constructs a solution, and the action is determined by the output of a graph embedding network capturing the current state of the solution. We show that our framework can be applied to a diverse range of optimization problems over graphs, and learns effective algorithms for the Minimum Vertex Cover, Maximum Cut and Traveling Salesman problems.

Motivation & Objective

• Motivate automating the design of heuristics for recurring graph optimization problems.
• Propose a greedy meta-algorithm whose actions are determined by a graph embedding network.
• Develop an end-to-end learning framework that optimizes the original problem objective via reinforcement learning.
• Demonstrate generalization across problem sizes and graph types for multiple classic problems.
• Show that learned heuristics can outperform traditional approaches on both synthetic and real-world datasets.

Proposed method

• Formulate greedy algorithms on graphs with a state S and candidate actions V\S.
• Represent the policy with a graph embedding network Structure2Vec to produce node embeddings.
• Parameterize the Q-function ficiently as AQ(h(S),v;Θ) using node and graph embeddings.
• Train end-to-end with n-step Q-learning and fitted Q-iteration to handle delayed rewards.
• Apply the framework to MVC, MAXCUT, and TSP and compare against PN-AC and classical heuristics.
• Demonstrate scalability and generalization to graphs larger than training data.

Experimental results

Research questions

• RQ1 Can a learned greedy policy generalize to unseen graph instances drawn from the same distribution?
• RQ2 Can a graph-structured embedding paired with reinforcement learning yield effective heuristics for MVC, MAXCUT, and TSP?
• RQ3 How does the learned method compare with hand-crafted heuristics and prior neural approaches on synthetic and real-world graphs?
• RQ4 Do the learned policies scale to graphs much larger than those seen during training?

Key findings

• S2V-DQN achieves significantly better average approximation ratios than competing methods across MVC, MAXCUT, and TSP.
• On MVC, S2V-DQN achieves an approximation ratio close to 1, approaching optimal performance.
• S2V-DQN generalizes well to graphs up to 1200 nodes when trained on smaller graphs.
• The method remains efficient with a polynomial time complexity of O(k|E|) and competitive runtimes on large graphs.
• Real-world datasets show S2V-DQN outperforming best competitors by notable margins (e.g., MVC MemeTracker, MAXCUT Physics, TSP TSPLIB).
• The learned policies demonstrate the ability to discover sensible new heuristics for MVC and MAXCUT.