[Paper Review] Neural Combinatorial Optimization with Heavy Decoder: Toward Large Scale Generalization

Fu Luo, Xi Lin+3 more|arXiv (Cornell University)|Oct 12, 2023

Vehicle Routing Optimization Methods20 citations

TL;DR

Introduces LEHD, a Light Encoder and Heavy Decoder model, to generalize neural combinatorial optimization to large-scale TSP and CVRP by dynamically updating node relationships during construction. Trains with a data-efficient supervised scheme and improves solutions via Random Re-Construct at inference.

ABSTRACT

Neural combinatorial optimization (NCO) is a promising learning-based approach for solving challenging combinatorial optimization problems without specialized algorithm design by experts. However, most constructive NCO methods cannot solve problems with large-scale instance sizes, which significantly diminishes their usefulness for real-world applications. In this work, we propose a novel Light Encoder and Heavy Decoder (LEHD) model with a strong generalization ability to address this critical issue. The LEHD model can learn to dynamically capture the relationships between all available nodes of varying sizes, which is beneficial for model generalization to problems of various scales. Moreover, we develop a data-efficient training scheme and a flexible solution construction mechanism for the proposed LEHD model. By training on small-scale problem instances, the LEHD model can generate nearly optimal solutions for the Travelling Salesman Problem (TSP) and the Capacitated Vehicle Routing Problem (CVRP) with up to 1000 nodes, and also generalizes well to solve real-world TSPLib and CVRPLib problems. These results confirm our proposed LEHD model can significantly improve the state-of-the-art performance for constructive NCO. The code is available at https://github.com/CIAM-Group/NCO_code/tree/main/single_objective/LEHD.

Motivation & Objective

• Address the poor generalization of existing constructive NCO models to large-scale problems.
• Propose a LEHD architecture that dynamically models node relations during solution construction.
• Develop data-efficient supervised training via partial-solution augmentation.
• Introduce a flexible inference improvement mechanism (Random Re-Construct) to refine solutions.

Proposed method

• Propose a Light Encoder and Heavy Decoder (LEHD) architecture where the light encoder has one attention layer and the heavy decoder has L attention layers.
• During decoding, dynamically re-embeds the starting node, destination node, and available nodes at each step to capture scale-independent relations.
• Train via a supervised scheme that constructs partial solutions with random sizes and directions (learn to construct partial solutions).
• Use a greedy inference along with Random Re-Construct (RRC) to iteratively improve solutions under a budget.
• Compare LEHD against classical solvers and multiple NCO baselines on TSP and CVRP across scales up to 1000 nodes.

Experimental results

Research questions

• RQ1 Can LEHD generalize from small-scale training to large-scale TSP/CVRP instances (up to 1000 nodes) without problem-specific tailoring?
• RQ2 Does a dynamic heavy decoder improve generalization more than a heavy encoder design for constructive NCO?
• RQ3 Can data-efficient supervised training via partial-solution augmentation match or exceed RL-based approaches in this setting?
• RQ4 Does Random Re-Construct reliably improve solution quality within a fixed inference budget?

Key findings

• LEHD achieves robust generalization to large-scale TSP/CVRP, solving up to 1000 nodes with competitive gaps.
• With 1000 RCC iterations, LEHD achieves near-optimal gaps on TSP1000 and sub-1% gaps on CVRP1000 under certain budgets.
• LEHD can outperform several learning-based baselines and even rival LKH3 on CVRP2xx–CVRP5xx with sufficient RCC iterations.
• RRC and partial-solution training contribute to superior performance versus POMO or random sampling approaches.
• Ablations show the advantage of the heavy decoder over heavy encoder designs for generalization on large-scale instances.