[Paper Review] Learning Generalizable Models for Vehicle Routing Problems via Knowledge Distillation

Jieyi Bi, Yining Ma+5 more|arXiv (Cornell University)|Oct 14, 2022

Vehicle License Plate RecognitionEngineering25 citations

TL;DR

The paper introduces Adaptive Multi-Distribution Knowledge Distillation (AMDKD) to train lightweight student models that generalize across multiple VRP distributions by distilling knowledge from multiple distribution-specific teachers.

ABSTRACT

Recent neural methods for vehicle routing problems always train and test the deep models on the same instance distribution (i.e., uniform). To tackle the consequent cross-distribution generalization concerns, we bring the knowledge distillation to this field and propose an Adaptive Multi-Distribution Knowledge Distillation (AMDKD) scheme for learning more generalizable deep models. Particularly, our AMDKD leverages various knowledge from multiple teachers trained on exemplar distributions to yield a light-weight yet generalist student model. Meanwhile, we equip AMDKD with an adaptive strategy that allows the student to concentrate on difficult distributions, so as to absorb hard-to-master knowledge more effectively. Extensive experimental results show that, compared with the baseline neural methods, our AMDKD is able to achieve competitive results on both unseen in-distribution and out-of-distribution instances, which are either randomly synthesized or adopted from benchmark datasets (i.e., TSPLIB and CVRPLIB). Notably, our AMDKD is generic, and consumes less computational resources for inference.

Motivation & Objective

• Address cross-distribution generalization in neural VRP solvers, which typically train and test on the same distribution (often Uniform).
• Propose a generic learning framework that transfers diverse policies from multiple exemplar distributions into a single generalist student model.
• Demonstrate that AMDKD yields competitive performance on unseen in-distribution and out-of-distribution instances while being computationally efficient.
• Show applicability to representative construction models (AM and POMO) and potential boost when combined with Efficient Active Search (EAS).
• Provide analysis validating the effectiveness and generality of the proposed distillation scheme.

Proposed method

• Propose Adaptive Multi-Distribution Knowledge Distillation (AMDKD) that trains a lightweight student by distilling from multiple distribution-specific teachers.
• Use exemplar distributions (Uniform, Cluster, Mixed) to train teachers and perform on-policy distillation where the student learns sequential node selection from a single selected teacher per epoch.
• Introduce an adaptive distribution selection strategy that updates the probability of choosing a distribution based on the student’s validation performance, focusing on harder distributions.
• Reduce the student model size (e.g., node embedding dimension from 128 to 64) to balance generalization and inference speed, while allowing larger students when beneficial.
• Apply AMDKD to two backbone models (AM and POMO) to obtain AMDKD-AM and AMDKD-POMO variants and report competitive generalization performance.
• Optionally couple the AMDKD student with Efficient Active Search (EAS) during inference to achieve state-of-the-art results.

Experimental results

Research questions

• RQ1 Can a single student model generalize across VRP distributions by learning from multiple distribution-specific teachers?
• RQ2 Does an adaptive distillation strategy that prioritizes harder distributions improve cross-distribution generalization?
• RQ3 How does AMDKD perform compared with baseline neural VRP models and other generalization techniques on unseen distributions and benchmark datasets?
• RQ4 Is the AMDKD approach robust when applied to different backbone architectures (AM and POMO) and problem variants (TSP and CVRP)?

Key findings

• AMDKD produces a lightweight student that outperforms its teacher models on unseen distributions for both TSP and CVRP.
• The student model size can be substantially reduced (e.g., AM from 0.68M to 0.26M; POMO from 1.20M to 0.49M) with maintained or improved generalization.
• AMDKD achieves competitive generalization on unseen in-distribution and out-of-distribution instances compared with baselines such as HAC, LCP, DACT, DROP, GANCO, and PSRO/LIH.
• AMDKD variants (AMDKD-AM and AMDKD-POMO) often outperform their respective backbones across sizes, while maintaining faster inference.
• Coupling AMDKD with Efficient Active Search (EAS) yields a new state-of-the-art on the evaluated benchmarks.