[Paper Review] Are large language models superhuman chemists?

Adrian Mirza, Nawaf Alampara+33 more|arXiv (Cornell University)|Apr 1, 2024

History and advancements in chemistry17 citations

TL;DR

The paper introduces ChemBench, a benchmark of 2,788 chemistry QA pairs to evaluate LLMs; leading models outperform top human chemists on average, but still struggle on certain tasks and with confidence calibration.

ABSTRACT

Large language models (LLMs) have gained widespread interest due to their ability to process human language and perform tasks on which they have not been explicitly trained. However, we possess only a limited systematic understanding of the chemical capabilities of LLMs, which would be required to improve models and mitigate potential harm. Here, we introduce "ChemBench," an automated framework for evaluating the chemical knowledge and reasoning abilities of state-of-the-art LLMs against the expertise of chemists. We curated more than 2,700 question-answer pairs, evaluated leading open- and closed-source LLMs, and found that the best models outperformed the best human chemists in our study on average. However, the models struggle with some basic tasks and provide overconfident predictions. These findings reveal LLMs' impressive chemical capabilities while emphasizing the need for further research to improve their safety and usefulness. They also suggest adapting chemistry education and show the value of benchmarking frameworks for evaluating LLMs in specific domains.

Motivation & Objective

• Create a standardized benchmark (ChemBench) to evaluate chemical knowledge, reasoning, and intuition in LLMs beyond property prediction.
• Assess how current state-of-the-art LLMs compare to expert chemists on a broad, education-aligned chemistry corpus.
• Analyze model performance across chemistry subfields and question types to identify strengths and gaps.
• Provide open, extensible evaluation infrastructure and leaderboards to track future progress and foster safer, more useful chemistry AI systems.

Proposed method

• Curate 2,788 question-answer pairs from manual and semi-automatic sources covering undergraduate and graduate chemistry topics.
• Encode chemical-specific modalities (e.g., SMILES) using annotated tokens to support tool-augmented systems.
• Evaluate a wide range of models, including open- and closed-source LLMs, with strict correct/incorrect scoring and tool-assisted settings.
• Survey human experts on a subset of questions to establish a comparative baseline and explore expert-model agreement.
• Implement a parsing and prompting pipeline to extract final text completions, including handling of domain-specific formats like SMILES and equations.

Experimental results

Research questions

• RQ1 How do state-of-the-art LLMs perform on a broad chemistry benchmark relative to expert chemists?
• RQ2 What are the strengths and limitations of LLMs across different chemistry topics and question types?
• RQ3 To what extent do model size, tool augmentation, and domain-specific encoding influence performance on chemical reasoning tasks?

Key findings

• Best models outperform the average human chemist on ChemBench overall (best model about twice human performance).
• Open-source models (e.g., Llama-3.1-405B-Instruct) approach the performance of leading proprietary models in several tasks.
• Performance gaps persist in knowledge-intensive questions and specific subfields like toxicity/safety and analytical chemistry (e.g., NMR signal counting remains difficult).
• Model performance does not consistently scale with molecular complexity, suggesting reliance on training data proximity rather than structural reasoning.
• Models often provide overconfident or miscalibrated uncertainty estimates, raising safety and reliability concerns for real-world use.