[Paper Review] MLModelScope: A Distributed Platform for ML Model Evaluation and Benchmarking at Scale

Abdul Dakkak, Cheng Li+2 more|arXiv (Cornell University)|Sep 25, 2019

Machine Learning in Materials Science28 references4 citations

TL;DR

MLModelScope is an open-source, framework- and hardware-agnostic platform that enables scalable, repeatable, and fair evaluation and benchmarking of machine learning models. It supports distributed execution via web, command-line, and library interfaces, and demonstrates how pipeline configurations and hardware/software stacks impact model accuracy and performance.

ABSTRACT

Machine Learning (ML) and Deep Learning (DL) innovations are being introduced at such a rapid pace that researchers are hard-pressed to analyze and study them. The complicated procedures for evaluating innovations, along with the lack of standard and efficient ways of specifying and provisioning ML/DL evaluation, is a major pain point for the community. This paper proposes MLModelScope, an open-source, framework/hardware agnostic, extensible and customizable design that enables repeatable, fair, and scalable model evaluation and benchmarking. We implement the distributed design with support for all major frameworks and hardware, and equip it with web, command-line, and library interfaces. To demonstrate MLModelScope's capabilities we perform parallel evaluation and show how subtle changes to model evaluation pipeline affects the accuracy and HW/SW stack choices affect performance.

Motivation & Objective

• To address the lack of standardized, efficient, and reproducible methods for evaluating ML/DL innovations.
• To design a distributed, extensible, and customizable platform that supports all major ML frameworks and hardware accelerators.
• To enable fair and scalable model evaluation through standardized pipelines and reproducible configurations.
• To provide multiple access interfaces—web, CLI, and library—for broad usability across research and production workflows.
• To empirically demonstrate the impact of evaluation pipeline design and hardware/software stack choices on model accuracy and performance.

Proposed method

• Designing a distributed architecture that decouples model evaluation logic from underlying hardware and frameworks.
• Implementing a plugin-based extensibility model to support integration with major ML frameworks (e.g., PyTorch, TensorFlow) and hardware accelerators.
• Providing standardized configuration schemas for defining evaluation pipelines, enabling reproducibility and versioning.
• Supporting parallel execution of model evaluations across heterogeneous hardware via a distributed task scheduler.
• Exposing the platform through a web interface for interactive exploration, CLI for automation, and a programmatic library API for integration.
• Basing evaluation fairness on consistent data loading, preprocessing, and metric computation across all test runs.

Experimental results

Research questions

• RQ1 How can we standardize and scale ML model evaluation across diverse frameworks and hardware platforms?
• RQ2 To what extent do subtle changes in the evaluation pipeline affect model accuracy and performance metrics?
• RQ3 How do hardware and software stack choices influence model inference and training performance in real-world settings?
• RQ4 Can a unified, extensible platform reduce the complexity and increase reproducibility of ML benchmarking?
• RQ5 What are the performance and accuracy trade-offs when using different evaluation pipeline configurations?

Key findings

• MLModelScope enables parallel evaluation of models across heterogeneous hardware, significantly reducing benchmarking time.
• Subtle changes in the evaluation pipeline—such as data shuffling or normalization—can lead to measurable differences in reported model accuracy.
• Hardware and software stack choices, including GPU types and framework versions, result in measurable performance variations in model inference and training.
• The platform’s standardized configuration and interface design improve reproducibility and reduce configuration-related errors in benchmarking workflows.
• The web and CLI interfaces allow researchers to efficiently manage and monitor large-scale evaluation campaigns.
• The library interface enables seamless integration into existing ML research and CI/CD pipelines.