[Paper Review] JARVIS-Leaderboard: A Large Scale Benchmark of Materials Design Methods

Kamal Choudhary, Daniel Wines+36 more|arXiv (Cornell University)|Jun 20, 2023

Machine Learning in Materials Science193 references4 citations

TL;DR

JARVIS-Leaderboard is an open-source, community-driven benchmark platform integrating diverse materials design methods—AI, electronic structure, force fields, quantum computation, and experiments—across multiple data modalities, including perfect and defective materials. It enables reproducible, transparent, and standardized evaluation of methods, with over 1,200 contributions and comprehensive performance comparisons using metrics like MAE and MAD/MAE ratios across 5,572 materials in the JARVIS-DFT 3D dataset.

ABSTRACT

Lack of rigorous reproducibility and validation are major hurdles for scientific development across many fields. Materials science in particular encompasses a variety of experimental and theoretical approaches that require careful benchmarking. Leaderboard efforts have been developed previously to mitigate these issues. However, a comprehensive comparison and benchmarking on an integrated platform with multiple data modalities with both perfect and defect materials data is still lacking. This work introduces JARVIS-Leaderboard, an open-source and community-driven platform that facilitates benchmarking and enhances reproducibility. The platform allows users to set up benchmarks with custom tasks and enables contributions in the form of dataset, code, and meta-data submissions. We cover the following materials design categories: Artificial Intelligence (AI), Electronic Structure (ES), Force-fields (FF), Quantum Computation (QC) and Experiments (EXP). For AI, we cover several types of input data, including atomic structures, atomistic images, spectra, and text. For ES, we consider multiple ES approaches, software packages, pseudopotentials, materials, and properties, comparing results to experiment. For FF, we compare multiple approaches for material property predictions. For QC, we benchmark Hamiltonian simulations using various quantum algorithms and circuits. Finally, for experiments, we use the inter-laboratory approach to establish benchmarks. There are 1281 contributions to 274 benchmarks using 152 methods with more than 8 million data-points, and the leaderboard is continuously expanding. The JARVIS-Leaderboard is available at the website: https://pages.nist.gov/jarvis_leaderboard

Motivation & Objective

• Address the critical lack of reproducibility and validation in materials science by establishing a unified benchmarking infrastructure.
• Integrate diverse materials design methods—including AI, electronic structure, force fields, quantum computation, and experimental data—into a single, extensible platform.
• Enable community contributions of datasets, code, and metadata to enhance transparency, reproducibility, and method validation.
• Provide standardized evaluation metrics and comparison tools (e.g., Jupyter notebooks) for consistent performance assessment across methods.
• Support both ideal and defective materials data to reflect real-world complexity and improve method robustness.

Proposed method

• Develop a centralized, open-access platform hosted on the JARVIS infrastructure at NIST to host benchmarking tasks across five domains: AI, electronic structure, force fields, quantum computation, and experiments.
• Implement standardized evaluation protocols using metrics such as mean absolute error (MAE) and mean absolute deviation to MAE ratio (MAD/MAE) for cross-method comparison.
• Integrate multiple data modalities including atomic structures, atomistic images, spectra, text, and experimental results from inter-laboratory round-robin studies.
• Support diverse input types in AI benchmarks, including Magpie and Voronoi-tessellation descriptors, and compare with neural network models.
• Enable reproducibility via Jupyter and Google Colab notebooks for visualization and performance analysis of benchmark results.
• Facilitate contributions through a structured submission pipeline for datasets, code, and metadata, ensuring version control and provenance tracking.

Figure 1: Leaderboard snapshot showing an example output for AI based formation energy per atom model on the JARVIS-DFT (dft_3d) dataset. The benchmark has seven contributions so far and they are sorted based on the mean absolute error (MAE) values. Lower MAE values indicate higher accuracy. Links t

Experimental results

Research questions

• RQ1 How do different AI models (e.g., descriptor-based vs. neural networks) perform in predicting materials properties like formation energy across diverse data inputs?
• RQ2 What is the consistency and accuracy of electronic structure methods across different software packages, pseudopotentials, and materials, relative to experimental data?
• RQ3 How do force field models compare in predicting bulk modulus and other mechanical properties across a range of materials?
• RQ4 What is the performance of various quantum circuits and algorithms in simulating electronic band structures using Hamiltonian simulations?
• RQ5 How reproducible are experimental results across multiple laboratories, particularly for complex properties like CO2 capture in zeolites?

Key findings

• The best descriptor-based model for AI regression tasks was a tree-based model using Magpie and Voronoi-tessellation features, outperforming neural networks in all tested benchmarks.
• For the JARVIS-DFT 3D dataset (5,572 materials), the AI benchmark showed consistent performance with MAE values enabling clear method ranking across input modalities.
• Electronic structure benchmarks revealed significant variation in bandgap predictions across software packages and pseudopotentials, with deviations from experimental values up to several eV in some cases.
• Quantum computation benchmarks demonstrated that different quantum circuits and algorithms produced varying levels of accuracy in simulating aluminum’s electronic band structure, with performance dependent on circuit depth and qubit mapping.
• Inter-laboratory experimental comparisons for CO2 capture in ZSM-5 zeolite showed measurable variability across labs, highlighting the need for standardized protocols.
• The MAD/MAE ratio served as a robust metric for performance comparison, with values indicating that some models exhibited higher variance relative to their mean error, especially in high-dimensional or noisy data regimes.

Figure 2: A flow-chart showing the processes involved in uploading a new contribution to the leaderbaord. The jarvis_populate_data.py scripts generate a benchmark dataset. A user can apply their method, train models, or run experiments on that dataset and prepare a csv.zip, a metadata.json file, and