[Paper Review] Self-Improvement for Neural Combinatorial Optimization: Sample without Replacement, but Improvement

Jonathan Pirnay, Dominik G. Grimm|arXiv (Cornell University)|Mar 22, 2024

Neural Networks and Applications4 citations

TL;DR

This paper proposes Gumbeldore (GD), a self-improving framework for neural combinatorial optimization that trains models via supervised imitation of the best solutions sampled from the current policy across multiple rounds. By combining batch-wise stochastic beam search with advantage-based policy updates and nucleus sampling, GD achieves state-of-the-art performance on TSP, CVRP, and especially JSSP—outperforming prior methods without reinforcement learning or expert trajectories.

ABSTRACT

Current methods for end-to-end constructive neural combinatorial optimization usually train a policy using behavior cloning from expert solutions or policy gradient methods from reinforcement learning. While behavior cloning is straightforward, it requires expensive expert solutions, and policy gradient methods are often computationally demanding and complex to fine-tune. In this work, we bridge the two and simplify the training process by sampling multiple solutions for random instances using the current model in each epoch and then selecting the best solution as an expert trajectory for supervised imitation learning. To achieve progressively improving solutions with minimal sampling, we introduce a method that combines round-wise Stochastic Beam Search with an update strategy derived from a provable policy improvement. This strategy refines the policy between rounds by utilizing the advantage of the sampled sequences with almost no computational overhead. We evaluate our approach on the Traveling Salesman Problem and the Capacitated Vehicle Routing Problem. The models trained with our method achieve comparable performance and generalization to those trained with expert data. Additionally, we apply our method to the Job Shop Scheduling Problem using a transformer-based architecture and outperform existing state-of-the-art methods by a wide margin.

Motivation & Objective

• Address the high computational cost and training instability of policy gradient methods in neural combinatorial optimization.
• Reduce reliance on expensive expert solutions from exact solvers by enabling self-supervised training from model-generated trajectories.
• Improve sample efficiency in solution search by avoiding duplicates and enhancing diversity through round-wise beam search with advantage-based updates.
• Enable strong generalization in large-scale architectures (e.g., transformers) by avoiding the need for complex RL fine-tuning.
• Demonstrate that a simple, problem-agnostic training loop based on iterative self-imitation can match or exceed performance of expert-supervised and RL-based methods.

Proposed method

• Sample multiple solutions per instance using the current policy in each training epoch, with beam width k=32 and four rounds of batch-wise stochastic beam search.
• Select the best-performing solution from each instance’s sampled set as a pseudo-expert trajectory for supervised learning.
• Update the policy using a provable policy improvement strategy that adjusts sequence probabilities based on their estimated advantage (objective value deviation).
• Introduce a dynamic nucleus sampling strategy with increasing top-p values over rounds to balance exploration and exploitation.
• Maintain a search tree across rounds to avoid sampling with replacement and improve diversity and convergence.
• Apply the method to a transformer-based architecture for JSSP with job- and machine-wise attention mechanisms and ALiBi positional encoding.

Experimental results

Research questions

• RQ1 Can a self-imitation loop using only model-generated solutions achieve performance comparable to expert-supervised training in neural combinatorial optimization?
• RQ2 How can sampling efficiency be maximized in self-improving training by minimizing redundant and duplicate solutions?
• RQ3 To what extent can a simple, problem-agnostic training scheme based on supervised imitation outperform complex policy gradient methods in large-scale problems?
• RQ4 Can the combination of stochastic beam search and advantage-based policy updates significantly improve solution quality with minimal computational overhead?
• RQ5 Does the proposed method enable strong generalization in large transformer-based architectures that are typically infeasible to train with policy gradient methods?

Key findings

• Gumbeldore achieves performance comparable to expert-supervised training on TSP and CVRP, even with no access to exact solver solutions.
• On the Job Shop Scheduling Problem, the method outperforms existing state-of-the-art approaches by a wide margin, demonstrating strong generalization and scalability.
• The use of advantage-based updates and dynamic nucleus sampling significantly improves sample efficiency, reducing the need for large numbers of samples.
• The method enables training of large transformer-based models for JSSP without reinforcement learning, overcoming the computational infeasibility of policy gradient fine-tuning.
• The self-imitation loop with beam search and progressive policy updates leads to consistent and measurable improvement across training epochs.
• The approach is problem-agnostic and effective across diverse combinatorial optimization problems, including TSP, CVRP, and JSSP, with minimal architectural or hyperparameter tuning.