[论文解读] AutoML in the Age of Large Language Models: Current Challenges, Future Opportunities and Risks

Alexander Tornede, Difan Deng+9 more|arXiv (Cornell University)|Jun 13, 2023

Natural Language Processing Techniques被引用 7

一句话总结

tldr: 本综述分析 AutoML 与大语言模型（LLMs）如何互相促进，在其交汇处详细探讨挑战、机遇与风险。

ABSTRACT

The fields of both Natural Language Processing (NLP) and Automated Machine Learning (AutoML) have achieved remarkable results over the past years. In NLP, especially Large Language Models (LLMs) have experienced a rapid series of breakthroughs very recently. We envision that the two fields can radically push the boundaries of each other through tight integration. To showcase this vision, we explore the potential of a symbiotic relationship between AutoML and LLMs, shedding light on how they can benefit each other. In particular, we investigate both the opportunities to enhance AutoML approaches with LLMs from different perspectives and the challenges of leveraging AutoML to further improve LLMs. To this end, we survey existing work, and we critically assess risks. We strongly believe that the integration of the two fields has the potential to disrupt both fields, NLP and AutoML. By highlighting conceivable synergies, but also risks, we aim to foster further exploration at the intersection of AutoML and LLMs.

研究动机与目标

• Explain how AutoML can enhance LLM lifecycles (pre-training, fine-tuning, inference).
• Identify core challenges that prevent holistic AutoML for LLMs.
• Explore how LLMs can improve AutoML tools and workflows.
• Assess risks of integrating AutoML with LLMs and propose safeguards.

提出的方法

• Survey existing work on AutoML for LLMs and LLM-assisted AutoML.
• Critically assess technical challenges across LLM lifecycle stages (pre-training, fine-tuning, inference).
• Discuss potential meta-learning, multi-fidelity, and gradient-based AutoML approaches relevant to LLMs.
• Highlight human-machine interaction enhancements via LLMs within AutoML tools.
• Outline risks such as evaluation issues, hallucinations, and resource demands.

Figure 1: AutoML can be used in all stages of the LLM lifecycle and needs to be adjusted to the different objectives, hyperparameters, and design decisions of each stage. The graphic depicts exemplary objectives, subjects of optimization, and associated hyperparameters. Due to computational constrai

实验结果

研究问题

• RQ1 What are the main challenges to holistically optimizing the LLM lifecycle with AutoML?
• RQ2 How can AutoML techniques be adapted (or extended) to handle pre-training, fine-tuning (including RLHF/alignment), and inference for LLMs?
• RQ3 What opportunities do LLMs offer to improve AutoML tools, human-AI interaction, and meta-learning components?
• RQ4 What risks arise from integrating AutoML and LLMs, and how can they be mitigated?

主要发现

• AutoML currently cannot holistically optimize the full LLM lifecycle due to cost, multi-stage objectives, and differing learning paradigms.
• LLMs can enhance AutoML via improved human–machine interaction, better data-driven prompts, and meta-learning from unstructured data.
• Integration brings risks like inadequate evaluation, potential hallucinations in AutoML-assisted components, and increasing resource demands.
• AutoRL and multi-fidelity approaches show promise but demand careful configuration and validation in the LLM context.
• Prompt engineering and in-context learning present avenues where AutoML can automate or augment LLM-driven optimization.
• Incentivizing multi-objective optimization is essential to balance performance, fairness/bias, and computational costs.

Figure 2: Overview of options where LLMs can be integrated into the AutoML process.