[Paper Review] From Word2Vec to Transformers: Text-Derived Composition Embeddings for Filtering Combinatorial Electrocatalysts

Lei Zhang, Markus Stricker|arXiv (Cornell University)|Mar 9, 2026

Machine Learning in Materials Science0 citations

TL;DR

This paper evaluates a label-free screening approach that embeds each composition in a text-derived latent space and uses dual Pareto-front filtering against conductivity and dielectric concept directions to reduce combinatorial electrocatalyst libraries while retaining near-best performers.

ABSTRACT

Compositionally complex solid solution electrocatalysts span vast composition spaces, and even one materials system can contain more candidate compositions than can be measured exhaustively. Here we evaluate a label-free screening strategy that represents each composition using embeddings derived from scientific texts and prioritizes candidates based on similarity to two property concepts. We compare a corpus-trained Word2Vec baseline with transformer-based embeddings, where compositions are encoded either by linear element-wise mixing or by short composition prompts. Similarities to `concept directions', the terms conductivity and dielectric, define a 2-dimensional descriptor space, and a symmetric Pareto-front selection is used to filter candidate subsets without using electrochemical labels. Performance is assessed on 15 materials libraries including noble metal alloys and multicomponent oxides. In this setting, the lightweight Word2Vec baseline, which uses a simple linear combination of element embeddings, often achieves the highest number of reductions of possible candidate compositions while staying close to the best measured performance.

Motivation & Objective

• Motivate efficient screening of compositionally complex electrocatalysts by leveraging text-derived embeddings without explicit electrochemical labels.
• Compare Word2Vec and transformer-based embeddings for representing multi-element compositions.
• Assess two embedding forms (element-wise linear mixtures vs full composition prompts) and their impact on candidate reduction and retention of top performers.
• Evaluate a dual Pareto-front filtering approach in a two-dimensional conductivity–dielectric descriptor space.
• Test cross-library transferability across 15 HER/ORR/OER materials libraries.

Proposed method

• Train a Word2Vec baseline on a corpus of electrocatalysis abstracts to create 200-dim element embeddings and form composition vectors via concentration-weighted linear mixing.
• Generate transformer-based element embeddings (MatSciBERT, Qwen) by encoding short element prompts and linearly aggregating by composition fractions.
• Create composition-prompt transformer embeddings (MatSciBERT_Full, Qwen_Full) by encoding full composition prompts and pooling to fixed-length vectors.
• Project each composition vector onto two concept directions: conductivity and dielectric, using cosine similarity to obtain a 2D descriptor (S_dielectric, S_conductivity).
• Apply a dual Pareto-front filter (maximize conductivity and minimize dielectric; and vice versa) to select non-dominated compositions from each embedding model.
• Evaluate performance on 15 libraries spanning HER, ORR, and OER by comparing retained fraction and deviation of best current density vs the original library.

Experimental results

Research questions

• RQ1 Can text-derived embeddings (Word2Vec and transformers) effectively screen combinatorial electrocatalyst spaces without electrochemical labels?
• RQ2 How do element-wise vs full-composition prompt representations affect filtering strength and retention of high-performing compositions?
• RQ3 Do two simple concept directions (conductivity, dielectric) suffice for near-optimal candidate selection across different reaction types (HER, ORR, OER)?
• RQ4 How transferable are text-derived filters across diverse material libraries with varying processing histories?

Key findings

• A lightweight Word2Vec baseline often achieves the highest number of reductions while staying close to the best measured performance.
• Transformer-based (MatSciBERT, Qwen) element-wise embeddings can retain more candidates but with varying trade-offs in error vs retention.
• Composition-prompt models (MatSciBERT_Full, Qwen_Full) typically retain 10–20% of candidates with modest deviations; in some systems they match or exceed element-wise results.
• Across most libraries, Pareto-filtered subsets remain diverse and spread through the library rather than collapsing to a small cluster, enabling exploratory screening with reduced experiments.
• In several oxide OER libraries, MatSciBERT_Full and Qwen_Full show advantages over their element-wise counterparts, but the simple Word2Vec baseline remains competitive across many systems.
• W2V often provides the strongest filtering with smallest retained fractions and low deviation, suggesting corpus statistics and two-concept space may capture key structure for this task.