[Paper Review] Chatbot Arena: An Open Platform for Evaluating LLMs by Human Preference

Régin, Florian, De Maria, Elisabetta+1 more|arXiv (Cornell University)|Jan 1, 2024

FinTech, Crowdfunding, Digital Finance94 citations

TL;DR

Chatbot Arena introduces an open, live benchmark platform that evaluates large language models (LLMs) via crowdsourced pairwise comparisons of model responses to real user questions. By leveraging human preference votes from over 240,000 interactions, the platform uses statistical models like Bradley-Terry and E-values to rank models efficiently and reliably, establishing a credible, dynamic alternative to static, accuracy-based benchmarks with strong consistency against expert ratings.

ABSTRACT

Constraint Programming (CP) and Machine Learning (ML) face challenges in text generation due to CP’s struggle with implementing "meaning" and ML’s difficulty with structural constraints. This paper proposes a solution by combining both approaches and embedding a Large Language Model (LLM) in CP. The LLM handles word generation and meaning, while CP manages structural constraints. This approach builds on GenCP, an improved version of On-the-fly Constraint Programming Search (OTFS) using LLM-generated domains. Compared to Beam Search (BS), a standard NLP method, this combined approach (GenCP with LLM) is faster and produces better results, ensuring all constraints are satisfied. This fusion of CP and ML presents new possibilities for enhancing text generation under constraints.

Motivation & Objective

• To address the limitations of static, accuracy-based benchmarks in evaluating LLMs' real-world alignment with human preferences.
• To create a scalable, open, and live evaluation platform that reflects real user interactions and diverse, fresh prompts.
• To ensure high-quality human preference data by validating crowd votes against expert raters and analyzing prompt diversity and quality.
• To develop efficient, sample-optimal sampling algorithms that accelerate convergence in model ranking while preserving statistical validity.
• To establish a publicly accessible, open-source benchmark that becomes a widely referenced standard in the LLM research and industry community.

Proposed method

• Employs a live, anonymous pairwise battle format where users vote between two LLM responses without knowing model identities.
• Collects real user questions and responses through a public website, ensuring diversity and freshness of prompts.
• Applies the Bradley-Terry model to estimate relative preference scores between models from pairwise vote data.
• Uses E-values from Vovk & Wang (2021) to assess statistical significance and reliability of model comparisons.
• Implements active learning-inspired sampling algorithms to select model pairs that maximize information gain and accelerate ranking convergence.
• Validates data quality by comparing crowd-sourced votes with expert annotations, confirming high agreement and consistency.

Experimental results

Research questions

• RQ1 How effective is a live, crowdsourced pairwise comparison system in capturing nuanced human preferences for LLM responses?
• RQ2 To what extent do crowd-sourced votes on Chatbot Arena align with expert human raters in evaluating model quality?
• RQ3 How diverse and representative are the user-generated prompts collected on the platform in reflecting real-world LLM use cases?
• RQ4 Can statistical models like Bradley-Terry and E-values reliably estimate model rankings from noisy, real-world human preference data?
• RQ5 How does the active sampling strategy improve sample efficiency in model evaluation compared to random or fixed pairing?

Key findings

• The platform has collected over 240,000 human preference votes from approximately 90,000 users across more than 100 languages since April 2023.
• Crowd-sourced votes show strong agreement with expert evaluations, confirming the reliability and credibility of the human preference data.
• User-generated prompts are sufficiently diverse and challenging, effectively covering a broad spectrum of real-world LLM applications and use cases.
• The active sampling algorithm significantly improves sample efficiency, accelerating convergence in model ranking while maintaining statistical validity.
• The platform has become one of the most widely cited LLM leaderboards in the research and industry community, with over 1 million website visits as of March 2024.
• The team plans to publicly release a human preference dataset containing over 100,000 pairwise votes from the platform, supporting future research.