[Paper Review] LLMeBench: A Flexible Framework for Accelerating LLMs Benchmarking

Fahim Dalvi, Maram Hasanain+11 more|arXiv (Cornell University)|Aug 9, 2023

Mathematics, Computing, and Information Processing4 citations

TL;DR

LLMeBench is an open-source, modular framework that accelerates and simplifies benchmarking of large language models (LLMs) across diverse NLP tasks and languages. It enables users to easily integrate custom datasets, models (via API), and evaluation metrics, with built-in caching, zero- and few-shot prompting, and support for 31 tasks across 11 languages using 53 datasets.

ABSTRACT

The recent development and success of Large Language Models (LLMs) necessitate an evaluation of their performance across diverse NLP tasks in different languages. Although several frameworks have been developed and made publicly available, their customization capabilities for specific tasks and datasets are often complex for different users. In this study, we introduce the LLMeBench framework, which can be seamlessly customized to evaluate LLMs for any NLP task, regardless of language. The framework features generic dataset loaders, several model providers, and pre-implements most standard evaluation metrics. It supports in-context learning with zero- and few-shot settings. A specific dataset and task can be evaluated for a given LLM in less than 20 lines of code while allowing full flexibility to extend the framework for custom datasets, models, or tasks. The framework has been tested on 31 unique NLP tasks using 53 publicly available datasets within 90 experimental setups, involving approximately 296K data points. We open-sourced LLMeBench for the community (https://github.com/qcri/LLMeBench/) and a video demonstrating the framework is available online. (https://youtu.be/9cC2m_abk3A)

Motivation & Objective

• Address the lack of user-friendly, customizable frameworks for benchmarking LLMs across diverse NLP tasks and low-resource languages.
• Reduce the time and complexity of setting up LLM evaluations by enabling plug-and-play integration of custom tasks, datasets, and models.
• Support zero-shot and few-shot prompting with intelligent example selection using Maximal Marginal Relevance (MMR) to improve few-shot learning efficiency.
• Minimize costs and API call overhead through an efficient caching mechanism that stores and reuses model outputs.
• Facilitate secure, private evaluation by allowing local dataset loading and on-premise model inference, enhancing data privacy for sensitive domains.

Proposed method

• Uses a modular pipeline architecture with four core components: Dataset, Model, Evaluation, and Asset modules, connected via key-value dictionary passing.
• Employs a customizable data loader that supports local or remote datasets, ensuring data privacy and flexibility.
• Implements a prompt engineering module that generates task-specific prompts, supporting both zero-shot and few-shot settings.
• Applies a Maximal Marginal Relevance (MMR) algorithm to automatically select diverse and representative few-shot examples from a user-defined training set.
• Integrates a persistent caching system that stores model outputs to prevent redundant API calls, reducing latency and cost.
• Provides a Benchmark Driver to orchestrate the full evaluation pipeline, including logging, post-processing, and result aggregation.

Figure 1: The architecture of the LLMeBench framework. The dotted boxes represent the core implemented modules of the architecture. Customization for new tasks, datasets and models can be done on Dataset , Model , Evaluation , and Asset modules.

Experimental results

Research questions

• RQ1 How can LLM benchmarking be made more accessible and customizable for researchers and practitioners working with low-resource or non-English NLP tasks?
• RQ2 To what extent can a caching mechanism reduce API call overhead and improve the efficiency of LLM evaluation workflows?
• RQ3 How effective is the MMR-based few-shot example selection strategy in improving few-shot learning performance across diverse NLP tasks?
• RQ4 Can a unified, modular framework support seamless integration of diverse models (e.g., GPT, BLOOM), datasets, and evaluation metrics without requiring deep code changes?
• RQ5 What is the scalability and reusability of a benchmarking framework that supports 31 predefined task recipes across 11 languages and 53 datasets?

Key findings

• LLMeBench enables the addition of a new custom dataset in under 10 minutes, significantly reducing setup time for new evaluation tasks.
• The framework has been rigorously tested across 90 experimental setups, involving 296,000 data points, across 31 unique NLP tasks and 53 publicly available datasets.
• The caching mechanism effectively prevents redundant API calls, reducing both time and cost in large-scale evaluations.
• The MMR-based few-shot example selection method improves the diversity and relevance of in-context examples, enhancing few-shot learning performance.
• The framework supports zero- and few-shot prompting across 140 pre-defined prompts, serving as a community resource for prompt engineering.
• LLMeBench is designed for extensibility and has been validated with models including GPT and BLOOM, with plans to support offline and multi-configuration model inference in future versions.

Figure 6: Summary and examples of the 53 datasets, 31 tasks, 3 models and metrics currently implemented and validated in LLMeBench.