[Paper Review] EvalAI: Towards Better Evaluation Systems for AI Agents

Deshraj Jain, Rishabh Jain+7 more|arXiv (Cornell University)|Feb 10, 2019

Machine Learning and Data Classification21 references44 citations

TL;DR

EvalAI is an open-source platform for scalable evaluation and comparison of ML/AI models and agents, enabling human-in-the-loop and remote evaluation in dynamic environments with customizable pipelines.

ABSTRACT

We introduce EvalAI, an open source platform for evaluating and comparing machine learning (ML) and artificial intelligence algorithms (AI) at scale. EvalAI is built to provide a scalable solution to the research community to fulfill the critical need of evaluating machine learning models and agents acting in an environment against annotations or with a human-in-the-loop. This will help researchers, students, and data scientists to create, collaborate, and participate in AI challenges organized around the globe. By simplifying and standardizing the process of benchmarking these models, EvalAI seeks to lower the barrier to entry for participating in the global scientific effort to push the frontiers of machine learning and artificial intelligence, thereby increasing the rate of measurable progress in this domain.

Motivation & Objective

• Motivate the need for robust evaluation platforms beyond static datasets and isolated models.
• Describe the desiderata for modern AI evaluation, including human-in-the-loop evaluation and environment-driven benchmarking.
• Introduce EvalAI as an extensible, open-source solution addressing custom pipelines, multiple phases, and remote evaluation.
• Demonstrate EvalAI’s architecture and capabilities through case studies across multimodal and embodied AI tasks.

Proposed method

• Proposes an extensible evaluation platform with support for arbitrary evaluation phases and dataset splits.
• Uses containerization (Docker) and a web backend (Django) with REST APIs to manage submissions and results.
• Implements a remote evaluation pipeline that decouples web servers and worker pools via message queues (SQS).
• Supports human-in-the-loop evaluation by pairing AMT workers with agents in real time and collecting interaction data.
• Allows organizers to submit model code and artifacts (Docker images, S3 assets) for evaluation in dynamic environments.

Experimental results

Research questions

• RQ1 How can an evaluation platform support both static and dynamic, environment-based AI tasks at scale?
• RQ2 What architectural choices enable flexible, multi-phase, multi-split benchmarks with human-in-the-loop and remote evaluation?
• RQ3 How does EvalAI perform in real-world challenges involving multimodal tasks and embodied AI?,

Key findings

• EvalAI substantially speeds up evaluation in VQA challenges, achieving about a 12x speedup over prior setups.
• The platform supports multiple challenge phases and data splits, enabling continued evaluation and privacy-controlled leaderboards.
• Remote evaluation allows organizers to run heavy computations on private clusters while maintaining centralized leadership and submission handling.
• Human-in-the-loop evaluation is feasible at scale, demonstrated in visual dialog tasks via real-time worker-agent interactions.
• Case studies show practical applicability across VQA, Visual Dialog, Embodied Question Answering, and fastMRI challenges.