[Paper Review] PMLB v1.0: An open source dataset collection for benchmarking machine learning methods

Joseph D. Romano, Trang T. Le+8 more|arXiv (Cornell University)|Nov 30, 2020

Explainable Artificial Intelligence (XAI)4 citations

TL;DR

PMLB v1.0 is an open-source, centralized repository of diverse, publicly available benchmark datasets designed to streamline machine learning method evaluation. It provides standardized, programmatic access via Python and R interfaces, enabling rapid benchmarking across diverse data science workflows with improved usability and community-driven enhancements over prior versions.

ABSTRACT

Motivation: Novel machine learning and statistical modeling studies rely on standardized comparisons to existing methods using well-studied benchmark datasets. Few tools exist that provide rapid access to many of these datasets through a standardized, user-friendly interface that integrates well with popular data science workflows. Results: This release of PMLB provides the largest collection of diverse, public benchmark datasets for evaluating new machine learning and data science methods aggregated in one location. v1.0 introduces a number of critical improvements developed following discussions with the open-source community. Availability: PMLB is available at https://github.com/EpistasisLab/pmlb. Python and R interfaces for PMLB can be installed through the Python Package Index and Comprehensive R Archive Network, respectively.

Motivation & Objective

• To address the lack of standardized, easily accessible benchmark datasets for evaluating new machine learning methods.
• To provide a centralized, user-friendly interface that integrates seamlessly with popular data science workflows in Python and R.
• To improve accessibility and usability of benchmark datasets through community-driven enhancements and versioned releases.
• To support reproducible research by offering a curated, diverse collection of well-studied datasets across multiple domains.

Proposed method

• Aggregation of diverse, publicly available datasets from multiple sources into a single, standardized repository.
• Implementation of a consistent, programmatic interface for programmatic access using Python and R packages.
• Incorporation of metadata, data provenance, and preprocessing notes to ensure reproducibility and usability.
• Versioning of the dataset collection (v1.0) to ensure stability and reproducibility across benchmarking experiments.
• Adoption of open-source licensing and community feedback to guide feature development and dataset curation.
• Integration with package managers (PyPI and CRAN) for easy installation and deployment in data science pipelines.

Experimental results

Research questions

• RQ1 How can a centralized, standardized dataset collection improve reproducibility and efficiency in machine learning benchmarking?
• RQ2 What are the key usability and integration challenges in existing benchmark dataset repositories, and how can they be addressed?
• RQ3 To what extent can community feedback improve the design and utility of open-source benchmark datasets?
• RQ4 How does a unified interface for Python and R enhance interoperability and adoption in machine learning research?

Key findings

• PMLB v1.0 provides the largest publicly available collection of diverse benchmark datasets in a single, standardized repository.
• The dataset collection is accessible via both Python and R through official package managers (PyPI and CRAN), enabling broad adoption.
• The release includes critical usability improvements based on community feedback, enhancing reliability and integration with data science workflows.
• The repository supports reproducible benchmarking by including metadata, provenance, and consistent preprocessing across datasets.
• The project is hosted as an open-source initiative with a DOI, ensuring persistent access and academic citability.
• The availability of a stable v1.0 release enables consistent benchmarking across studies and facilitates method comparison in machine learning research.