[Paper Review] ML4CO: Is GCNN All You Need? Graph Convolutional Neural Networks Produce Strong Baselines For Combinatorial Optimization Problems, If Tuned and Trained Properly, on Appropriate Data

Amin Banitalebi-Dehkordi, Yong Zhang|arXiv (Cornell University)|Dec 22, 2021

Advanced Graph Neural Networks4 citations

TL;DR

This paper demonstrates that a well-tuned Graph Convolutional Neural Network (GCNN) can achieve state-of-the-art performance on combinatorial optimization problems by learning variable selection in branch-and-bound algorithms. The Huawei EI-OROAS team achieved second place in the NeurIPS 2021 ML4CO competition, with consistent top rankings during weekly evaluations, proving that GCNNs outperform traditional heuristics when properly trained on problem-specific data.

ABSTRACT

The 2021 NeurIPS Machine Learning for Combinatorial Optimization (ML4CO) competition was designed with the goal of improving state-of-the-art combinatorial optimization solvers by replacing key heuristic components with machine learning models. The competition's main scientific question was the following: is machine learning a viable option for improving traditional combinatorial optimization solvers on specific problem distributions, when historical data is available? This was motivated by the fact that in many practical scenarios, the data changes only slightly between the repetitions of a combinatorial optimization problem, and this is an area where machine learning models are particularly powerful at. This paper summarizes the solution and lessons learned by the Huawei EI-OROAS team in the dual task of the competition. The submission of our team achieved the second place in the final ranking, with a very close distance to the first spot. In addition, our solution was ranked first consistently for several weekly leaderboard updates before the final evaluation. We provide insights gained from a large number of experiments, and argue that a simple Graph Convolutional Neural Network (GCNNs) can achieve state-of-the-art results if trained and tuned properly.

Motivation & Objective

• To investigate whether machine learning, specifically GCNNs, can improve heuristic components in combinatorial optimization solvers for problem distributions with repeated structural patterns.
• To evaluate the effectiveness of GCNNs in learning variable selection for branch-and-bound algorithms in mixed-integer linear programs (MILPs).
• To identify key hyperparameters and data engineering practices that significantly impact model performance in real-world optimization settings.
• To provide practical guidelines for practitioners to maximize the performance of neural networks in combinatorial optimization.

Proposed method

• The team employed a Graph Convolutional Neural Network (GCNN) to predict the best variable to branch on during the branch-and-bound process in MILP solving.
• The model was trained on data collected from SCIP solver logs, using a reinforcement learning setup with a reward signal based on solving time reduction.
• Data collection involved sampling from MILP instances using varying probabilities of strong branching and different time limits to generate diverse training trajectories.
• Hyperparameter tuning was performed across learning rate, batch size, GCN architecture depth, and training sample size to optimize validation performance.
• The model was evaluated using a standardized benchmark of MILP instances from three domains: item placement, load balancing, and anonymous problems.
• The team used a multi-objective evaluation strategy, tracking top-k variable selection accuracy (top-1 to top-10) alongside overall reward (solving time improvement).

Experimental results

Research questions

• RQ1 Can a simple GCNN achieve state-of-the-art performance in combinatorial optimization when properly trained and tuned?
• RQ2 How do hyperparameters such as training sample size, time limit, and strong branching probability affect the final model performance?
• RQ3 To what extent does top-1 variable selection accuracy correlate with overall solving time improvement (i.e., reward)?
• RQ4 Does increasing training data size always lead to better model generalization in this setting?
• RQ5 What role does data engineering play in achieving high performance with neural networks for optimization?

Key findings

• The GCNN-based model achieved second place in the final NeurIPS 2021 ML4CO competition, with a very close margin to the first-place solution.
• The model consistently ranked first in multiple weekly leaderboard updates before the final evaluation, indicating strong generalization and robustness.
• Larger training set sizes did not necessarily lead to higher rewards, challenging the common assumption that more data always improves performance.
• Top-5 and top-10 variable selection accuracy frequently exceeded 90%, suggesting that GCNNs can reliably identify high-quality branching candidates.
• The model’s performance was highly sensitive to hyperparameters such as strong branching probability and time limits during data collection, indicating that data quality and diversity are critical.
• The study highlights that data-centric AI—careful data collection and preprocessing—can be as important as model architecture in achieving high performance in combinatorial optimization.