[Paper Review] Clever Materials: When Models Identify Good Materials for the Wrong Reasons

Kevin Maik Jablonka|arXiv (Cornell University)|Feb 18, 2026

Machine Learning in Materials Science0 citations

TL;DR

The paper shows that models predicting materials properties can rely on bibliographic metadata learned from descriptors, sometimes matching chemistry-based predictors, revealing dataset vulnerabilities to proxy learning and the need for falsification tests.

ABSTRACT

Machine learning can accelerate materials discovery. Models perform impressively on many benchmarks. However, strong benchmark performance does not imply that a model learned chemistry. I test a concrete alternative hypothesis: that property prediction can be driven by bibliographic confounding. Across five tasks spanning MOFs (thermal and solvent stability), perovskite solar cells (efficiency), batteries (capacity), and TADF emitters (emission wavelength), models trained on standard chemical descriptors predict author, journal, and publication year well above chance. When these predicted metadata ("bibliographic fingerprints") are used as the sole input to a second model, performance is sometimes competitive with conventional descriptor-based predictors. These results show that many datasets do not rule out non-chemical explanations of success. Progress requires routine falsification tests (e.g., group/time splits and metadata ablations), datasets designed to resist spurious correlations, and explicit separation of two goals: predictive utility versus evidence of chemical understanding.

Motivation & Objective

• Investigate whether standard materials-property prediction models rely on non-chemical signals such as author, journal, and year rather than true chemical structure–property relationships.
• Assess the prevalence and strength of such proxy learning ('Clever Hans' effects) across diverse materials domains.
• Propose evaluation strategies and data infrastructure changes to routinely falsify alternative hypotheses about model performance.
• Examine differences in robustness of predictions across tasks to guide better dataset design and validation practices.

Proposed method

• Train three model classes on identical cross-validation folds: (i) conventional descriptor-to-property models, (ii) metadata-prediction models mapping descriptors to bibliographic variables, and (iii) proxy models predicting properties from predicted bibliographic data.
• Use gradient boosting (LightGBM) with standardized preprocessing and feature generation for chemical descriptors.
• Enrich datasets with bibliographic metadata via Crossref and create meta-features for top-N authors/journals.
• Evaluate with multiple metrics and cross-validation to compare direct, metadata, and proxy models under realistic testing conditions.
• Implement a systematic Clever Hans analysis framework to quantify whether predicted bibliographic variables can replace chemical descriptors in property prediction.
• Apply time/split strategies and baseline comparisons to assess robustness of results.

Experimental results

Research questions

• RQ1 Can models predict material properties using only predicted bibliographic information derived from descriptors?
• RQ2 To what extent do bibliographic signals (authors, journals, years) enable competitive property predictions across MOFs, perovskites, batteries, and TADF emitters?
• RQ3 How do evaluation metrics and baselines affect the detection of Clever Hans effects in materials datasets?
• RQ4 What dataset-design and data-infrastructure changes are needed to resist spurious correlations and improve validation rigor?

Key findings

• Proxy models using predicted bibliographic metadata can achieve performance close to conventional descriptor-based predictors in several tasks.
• MOF thermal stability: bibliographic signals enable near-top performance in classification depending on metric used, with partial Clever Hans susceptibility.
• MOF solvent stability shows moderate proxy learning, with authorship and publication venue predictability from descriptors and non-trivial proxy performance.
• Perovskite solar-cell efficiency: proxy models can match top-10% efficiency classification using predicted bibliographic data, suggesting possible reliance on meta-patterns rather than pure composition–performance relationships.
• TADF emitter emission wavelength shows detectable but limited Clever Hans effects; battery capacity prediction shows negligible proxy learning, with proxy performance not exceeding naive baselines.
• Overall, bibliographic shortcuts vary by domain and metric, and standard validation can miss such shortcuts without targeted tests.

Figure 2 : For the classification task of membership in the top-10% of thermally stable MOFs, one can be fooled (by Clever Hans effects). a The model predicts the authors of the associated paper with high accuracy, much better than a random baseline. b This also holds for predicting in which journal