[論文レビュー] INSTRUCTEVAL: Towards Holistic Evaluation of Instruction-Tuned Large Language Models

Yew Ken Chia, Pengfei Hong+2 more|arXiv (Cornell University)|Jun 7, 2023

Topic Modeling被引用数 26

ひとこと要約

InstructEval は、問題解決、ライティング、および人間の価値観への整合性の観点から instruction-tuned LLMs を評価する包括的ベンチマークスイートを提供し、事前学習の影響、命令データ、トレーニング方法を分析します。

ABSTRACT

Instruction-tuned large language models have revolutionized natural language processing and have shown great potential in applications such as conversational agents. These models, such as GPT-4, can not only master language but also solve complex tasks in areas like mathematics, coding, medicine, and law. Despite their impressive capabilities, there is still a lack of comprehensive understanding regarding their full potential, primarily due to the black-box nature of many models and the absence of holistic evaluation studies. To address these challenges, we present INSTRUCTEVAL, a more comprehensive evaluation suite designed specifically for instruction-tuned large language models. Unlike previous works, our evaluation involves a rigorous assessment of models based on problem-solving, writing ability, and alignment to human values. We take a holistic approach to analyze various factors affecting model performance, including the pretraining foundation, instruction-tuning data, and training methods. Our findings reveal that the quality of instruction data is the most crucial factor in scaling model performance. While open-source models demonstrate impressive writing abilities, there is substantial room for improvement in problem-solving and alignment. We are encouraged by the rapid development of models by the open-source community, but we also highlight the need for rigorous evaluation to support claims made about these models. Through INSTRUCTEVAL, we aim to foster a deeper understanding of instruction-tuned models and advancements in their capabilities. INSTRUCTEVAL is publicly available at https://github.com/declare-lab/instruct-eval.

研究の動機と目的

• Assess the holistic capabilities of instruction-tuned LLMs beyond traditional benchmarks.
• Analyze how pretraining foundation, instruction data, and training methods influence performance.
• Identify which factors most effectively scale model capabilities.
• Provide open-source access to a comprehensive evaluation framework and leaderboard.

提案手法

• Define a holistic evaluation suite covering problem-solving, writing, and alignment to human values.
• Use multiple objective and subjective evaluation methods including automatic and human-in-the-loop assessments.
• Compare over 60 open-source instructed LLMs using standardized benchmarks (MMLU, BBH, DROP, CRASS, HumanEval, HHH, and an impact writing dataset).
• Analyze effects of foundation model size, instruction data quality, and training method (supervised vs RLHF, parameter-efficient fine-tuning).
• Investigate few-shot versus zero-shot performance and in-context learning effects across tasks.]
• research_questions:[
• How do instruction-tuning factors (foundational model, data quality, and training method) affect problem-solving, writing, and alignment performance?
• What is the relative importance of instruction data versus pretraining foundation in scaling performance?
• Can open-source instructed LLMs match or approach closed-source models in writing and alignment, and where do they lag in problem-solving?
• Do few-shot demonstrations consistently improve performance across tasks and models?

Figure 1: Overview of InstructEval , our holistic evaluation suite for Instructed LLMs

実験結果

リサーチクエスチョン

• RQ1 How do instruction-tuning factors (foundational model, data quality, and training method) affect problem-solving, writing, and alignment performance?
• RQ2 What is the relative importance of instruction data versus pretraining foundation in scaling performance?
• RQ3 Can open-source instructed LLMs match or approach closed-source models in writing and alignment, and where do they lag in problem-solving?
• RQ4 Do few-shot demonstrations consistently improve performance across tasks and models?

主な発見

• Instruction data quality is the most crucial factor for scaling performance.
• Open-source instructed LLMs excel in writing but show substantial gaps in problem-solving and alignment.
• Mimicking closed-source models via synthetic instructions yields limited benefits and can propagate biases/noise.
• Training method (e.g., RLHF) helps but generally has smaller impact than instruction data; parameter-efficient tuning scales well with model size.
• There is not a uniform benefit of few-shot demonstrations across tasks; benefits are task-dependent and sometimes negative.