[Paper Review] Meta-SAGE: Scale Meta-Learning Scheduled Adaptation with Guided Exploration for Mitigating Scale Shift on Combinatorial Optimization

Jiwoo Son, Minsu Kim+2 more|arXiv (Cornell University)|Jun 5, 2023

Reinforcement Learning in Robotics4 citations

TL;DR

Meta-SAGE proposes a test-time adaptation framework for deep reinforcement learning models in combinatorial optimization, combining a scale meta-learner (SML) for scale-aware context embedding and scheduled adaptation with guided exploration (SAGE) that introduces decaying locality bias. It achieves state-of-the-art performance on large-scale TSP, CVRP, and related problems with significantly reduced adaptation iterations and improved scalability.

ABSTRACT

This paper proposes Meta-SAGE, a novel approach for improving the scalability of deep reinforcement learning models for combinatorial optimization (CO) tasks. Our method adapts pre-trained models to larger-scale problems in test time by suggesting two components: a scale meta-learner (SML) and scheduled adaptation with guided exploration (SAGE). First, SML transforms the context embedding for subsequent adaptation of SAGE based on scale information. Then, SAGE adjusts the model parameters dedicated to the context embedding for a specific instance. SAGE introduces locality bias, which encourages selecting nearby locations to determine the next location. The locality bias gradually decays as the model is adapted to the target instance. Results show that Meta-SAGE outperforms previous adaptation methods and significantly improves scalability in representative CO tasks. Our source code is available at https://github.com/kaist-silab/meta-sage

Motivation & Objective

• Address the scalability gap in deep reinforcement learning (DRL) models for combinatorial optimization (CO) by enabling efficient adaptation to larger problem scales.
• Overcome the inefficiency of existing test-time adaptation methods when facing large-scale distributional shifts between source (small-scale) and target (large-scale) problems.
• Improve transferability of pre-trained DRL models to unseen large-scale CO instances without requiring retraining on large-scale data.
• Enhance adaptation efficiency by introducing structured, scale-aware parameter updates and guided exploration strategies.
• Achieve high performance on large-scale CO tasks such as TSP, CVRP, prize-collecting TSP, and orienteering problem with minimal adaptation overhead.

Proposed method

• Introduce a scale meta-learner (SML) that generates scale-conditioned context embeddings by meta-optimizing for subsequent SAGE adaptation, using a bi-level optimization structure.
• Design scheduled adaptation with guided exploration (SAGE), which applies locality bias to encourage selection of nearby nodes during inference, with the bias gradually decaying over adaptation steps.
• Decouple the adaptation process into two scheduling targets: locality bias decay and SoftMax temperature adjustment, enabling controlled exploration and exploitation trade-offs.
• Use a masking mechanism in the DRL policy to enforce constraints (e.g., capacity limits in CVRP), preserving feasibility during adaptation.
• Leverage pre-trained models (e.g., Sym-NCO) as initialization, enabling zero-shot transfer to large-scale problems via context-aware adaptation.
• Formulate the SML objective as a bi-level optimization problem that amortizes the parameter update process, ensuring effective initialization for SAGE.

Experimental results

Research questions

• RQ1 Can a meta-learned context embedding mechanism improve the efficiency of test-time adaptation for large-scale combinatorial optimization problems?
• RQ2 Does introducing a decaying locality bias in the adaptation process enhance convergence speed and solution quality?
• RQ3 To what extent does scheduled adaptation with dual scheduling (locality bias and SoftMax temperature) reduce the number of adaptation steps required for high performance?
• RQ4 How does Meta-SAGE compare to existing adaptation baselines in terms of scalability and solution quality on large-scale TSP and CVRP?
• RQ5 Can the proposed method generalize to non-Euclidean CO problems with appropriate pre-trained models?

Key findings

• Meta-SAGE achieves a 0.68% optimal gap on CVRP with 1,000 nodes, significantly outperforming prior adaptation methods.
• On TSP with 500 nodes, Meta-SAGE reduces the optimal gap to 17.22% with full component usage, compared to 18.63% without SML.
• The inclusion of SML reduces the L1 distance between input and adapted embeddings by 11% on average (TSP: 8,066.13 → 7,175.99; CVRP: 10,361.51 → 9,285.22), demonstrating improved embedding alignment.
• Ablation studies confirm that both SML and SAGE components are essential: removing SML or SAGE components leads to significant performance degradation.
• SML performs best when the input scale matches the target scale, indicating strong scale-awareness in context embedding generation.
• Meta-SAGE shows feasibility on real-world CVRP benchmarks, suggesting robustness to instance-wise distributional shifts beyond scale shifts.