[Paper Review] Interactive Multi-fidelity Learning for Cost-effective Adaptation of Language Model with Sparse Human Supervision

Jiaxin Zhang, Zhuohang Li+2 more|arXiv (Cornell University)|Oct 31, 2023

Topic Modeling4 citations

TL;DR

This paper proposes Interactive Multi-fidelity Learning (IMFL), a cost-effective framework for fine-tuning small domain-specific language models using a strategic blend of low-fidelity LLM-generated annotations and high-fidelity human annotations. By employing an exploration-exploitation query strategy with prompt retrieval and variable batch sizing, IMFL achieves performance comparable to 5× human annotation with only 1/3 the human labeling budget on financial and medical tasks.

ABSTRACT

Large language models (LLMs) have demonstrated remarkable capabilities in various tasks. However, their suitability for domain-specific tasks, is limited due to their immense scale at deployment, susceptibility to misinformation, and more importantly, high data annotation costs. We propose a novel Interactive Multi-Fidelity Learning (IMFL) framework for the cost-effective development of small domain-specific LMs under limited annotation budgets. Our approach formulates the domain-specific fine-tuning process as a multi-fidelity learning problem, focusing on identifying the optimal acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance. We further propose an exploration-exploitation query strategy that enhances annotation diversity and informativeness, incorporating two innovative designs: 1) prompt retrieval that selects in-context examples from human-annotated samples to improve LLM annotation, and 2) variable batch size that controls the order for choosing each fidelity to facilitate knowledge distillation, ultimately enhancing annotation quality. Extensive experiments on financial and medical tasks demonstrate that IMFL achieves superior performance compared with single fidelity annotations. Given a limited budget of human annotation, IMFL significantly outperforms the human annotation baselines in all four tasks and achieves very close performance as human annotations on two of the tasks. These promising results suggest that the high human annotation costs in domain-specific tasks can be significantly reduced by employing IMFL, which utilizes fewer human annotations, supplemented with cheaper and faster LLM (e.g., GPT-3.5) annotations to achieve comparable performance.

Motivation & Objective

• Address the high cost of human annotation in fine-tuning domain-specific language models despite their need for expert knowledge.
• Overcome the limitations of relying solely on LLM-generated annotations, which suffer from hallucinations and misinformation in high-stakes domains.
• Develop a cost-effective fine-tuning framework that optimally balances low-fidelity LLM annotations and high-fidelity human annotations under strict annotation budgets.
• Improve annotation quality and model performance by leveraging in-context learning and dynamic batch scheduling in a multi-fidelity learning setting.

Proposed method

• Formulate domain-specific fine-tuning as an interactive multi-fidelity learning problem, treating LLMs and humans as sources of annotation with different fidelities.
• Design an exploration-exploitation query strategy: LLMs explore for diversity, while human annotations exploit for informativeness.
• Introduce prompt retrieval to select in-context examples from human-annotated data, improving the accuracy of LLM-generated annotations.
• Implement variable batch size to control the order of annotation acquisition, enabling better knowledge distillation between LLM and human annotations.
• Use iterative active learning to select the most uncertain or informative samples for human annotation, minimizing redundant effort.
• Apply knowledge distillation to transfer knowledge from the final model to the LLM-annotator, refining future LLM predictions.

Figure 1: (a) Proposed Interactive Multi-Fidelity Learning Framework ( IMFL ). IMFL aims at solving the best acquisition strategy that balances between low-fidelity automatic LLM annotations and high-fidelity human annotations to maximize model performance given limited annotation budgets. (b) IMFL

Experimental results

Research questions

• RQ1 Can a multi-fidelity learning framework reduce human annotation costs in domain-specific language model fine-tuning without sacrificing performance?
• RQ2 How does combining low-fidelity LLM annotations with sparse high-fidelity human annotations compare to single-fidelity baselines in terms of model performance?
• RQ3 To what extent can prompt retrieval and variable batch sizing improve the quality of LLM-generated annotations in low-resource, high-stakes domains?
• RQ4 Can the proposed framework achieve performance close to 5× human annotation budgets using only 1/3 of the human labeling effort?
• RQ5 How does the interaction between LLM and human annotators affect model generalization and robustness in financial and medical NLP tasks?

Key findings

• IMFL significantly outperforms 3× human annotation baselines across all four evaluated tasks (financial and medical NLP), demonstrating superior cost-performance trade-offs.
• On two of the four tasks, IMFL achieves performance within close range of the 5× human annotation upper bound, indicating strong efficiency gains.
• The integration of prompt retrieval improves LLM annotation accuracy by selecting relevant in-context examples from human-annotated data.
• Variable batch sizing enhances knowledge distillation, leading to better alignment between LLM and human annotations and improved model generalization.
• The framework reduces human annotation costs by up to 66% while maintaining or exceeding performance of full human-annotation baselines.
• IMFL is robust across diverse domains, showing consistent gains in both financial and medical NLP tasks under limited annotation budgets.

Figure 3: Illustration of exploration-exploitation query strategy with core components and steps.