[Paper Review] Large Language Model-Enhanced Algorithm Selection: Towards Comprehensive Algorithm Representation

Xingyu Wu, Yan Zhong+3 more|arXiv (Cornell University)|Nov 22, 2023

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes AS-LLM, a novel algorithm selection framework that leverages Large Language Models (LLMs) to extract comprehensive, high-dimensional algorithm features from code and description text, combining them with problem features via similarity-based matching. The method achieves state-of-the-art performance and strong generalization across diverse algorithms and problems, with theoretical bounds confirming its robustness to distribution shifts.

ABSTRACT

Algorithm selection, a critical process of automated machine learning, aims to identify the most suitable algorithm for solving a specific problem prior to execution. Mainstream algorithm selection techniques heavily rely on problem features, while the role of algorithm features remains largely unexplored. Due to the intrinsic complexity of algorithms, effective methods for universally extracting algorithm information are lacking. This paper takes a significant step towards bridging this gap by introducing Large Language Models (LLMs) into algorithm selection for the first time. By comprehending the code text, LLM not only captures the structural and semantic aspects of the algorithm, but also demonstrates contextual awareness and library function understanding. The high-dimensional algorithm representation extracted by LLM, after undergoing a feature selection module, is combined with the problem representation and passed to the similarity calculation module. The selected algorithm is determined by the matching degree between a given problem and different algorithms. Extensive experiments validate the performance superiority of the proposed model and the efficacy of each key module. Furthermore, we present a theoretical upper bound on model complexity, showcasing the influence of algorithm representation and feature selection modules. This provides valuable theoretical guidance for the practical implementation of our method.

Motivation & Objective

• To address the underexplored role of algorithm features in automated algorithm selection, which are often treated as black boxes despite their potential to improve model performance.
• To overcome the limitations of manual or handcrafted algorithm feature engineering, which lack universality and fail to capture semantic and contextual information in algorithms.
• To develop a universal, automated method for algorithm representation that captures structural, semantic, and contextual features across diverse programming languages and algorithm types.
• To enable bidirectional modeling of problem-algorithm relationships by integrating algorithm representations with problem features for improved selection accuracy and generalization.
• To theoretically bound the generalization error of the model under distribution shifts in both problems and algorithms, validating its robustness.

Proposed method

• Leverages pre-trained Large Language Models (LLMs) to encode algorithm code or textual descriptions into high-dimensional, semantically rich representations that capture syntax, semantics, and library function usage.
• Applies a feature selection module to identify and retain the most informative algorithm features, reducing dimensionality while preserving discriminative power.
• Combines the selected algorithm representation with problem features using a cosine similarity calculation module to measure matching degree between problems and candidate algorithms.
• Uses the similarity score as input to a final classifier (MLP) to predict the optimal algorithm for a given problem instance.
• Employs a unified framework that supports both code-based and description-based algorithm input, enabling broad applicability even when source code is unavailable.
• Provides theoretical upper bounds on generalization error for both problem and algorithm distribution shifts, validating the model’s robustness.

Experimental results

Research questions

• RQ1 Can LLMs effectively extract comprehensive, universal algorithm features from code and textual descriptions, surpassing traditional handcrafted features?
• RQ2 How does incorporating algorithm features improve the performance and generalization capability of algorithm selection models compared to relying solely on problem features?
• RQ3 What is the contribution of each component in the AS-LLM framework—LLM-based feature extraction, feature selection, and similarity-based matching—to overall performance?
• RQ4 To what extent can the model generalize to unseen algorithms not present in the training data, especially when algorithm features are extracted via LLMs?
• RQ5 What theoretical guarantees can be provided for the generalization error of algorithm selection models under joint distribution shifts in problems and algorithms?

Key findings

• AS-LLM outperforms existing algorithm selection methods across multiple ASLib benchmarks, with significant gains in PAR10 scores, especially in scenarios with diverse algorithm types.
• The ablation study confirms that removing the algorithm feature module (AS-LLM-AF) leads to the largest performance drop, validating the critical role of algorithm features in selection accuracy.
• The feature selection module improves model convergence speed and final performance, as shown by faster-decreasing training loss curves compared to the full model without it.
• The cosine similarity module contributes meaningfully to performance, particularly in zero-shot generalization to new algorithms, as removing it (AS-LLM-CS) results in measurable performance degradation.
• AS-LLM demonstrates strong generalization over algorithms: when individual algorithms are excluded from training, the resulting PAR10 scores remain close to the full training baseline, indicating robustness to algorithm distribution shifts.
• LLM-extracted algorithm features consistently outperform manually engineered features (e.g., code lines, AST features) in both isolation and combination, highlighting the superiority of LLM-based representation.