[Paper Review] Improving Performance Prediction of Electrolyte Formulations with Transformer-based Molecular Representation Model

Indra Priyadarsini, Vidushi Sharma+4 more|arXiv (Cornell University)|Jun 28, 2024

Machine Learning in Materials ScienceMaterials Science3 citations

TL;DR

This paper proposes a transformer-based molecular representation model, BART-SA, that improves electrolyte formulation performance prediction by learning robust molecular embeddings via SELFIES-encoding and scaling them by concentration before summing into a unified feature vector. The method achieves state-of-the-art results on two battery property prediction tasks, with RMSE of 0.148 for Coulombic Efficiency and 20.001 mAh/g for Specific Capacity.

ABSTRACT

Development of efficient and high-performing electrolytes is crucial for advancing energy storage technologies, particularly in batteries. Predicting the performance of battery electrolytes rely on complex interactions between the individual constituents. Consequently, a strategy that adeptly captures these relationships and forms a robust representation of the formulation is essential for integrating with machine learning models to predict properties accurately. In this paper, we introduce a novel approach leveraging a transformer-based molecular representation model to effectively and efficiently capture the representation of electrolyte formulations. The performance of the proposed approach is evaluated on two battery property prediction tasks and the results show superior performance compared to the state-of-the-art methods.

Motivation & Objective

• To address the challenge of accurately predicting battery electrolyte performance despite complex multi-component interactions and variable formulation compositions.
• To develop a scalable, generalizable representation method for multi-constituent electrolyte formulations that captures both molecular identity and relative concentration.
• To overcome limitations of prior models that rely on dummy featurization or fail to model concentration-dependent effects effectively.
• To improve generalization and predictive accuracy in battery property prediction by leveraging self-supervised pretraining on molecular sequences with enhanced representational fidelity.

Proposed method

• Pretrain a bidirectional auto-regressive transformer (BART) on 500M and 118M molecules from ZINC and PubChem using SELFIES representation to ensure syntactic and semantic validity.
• Use word-level tokenization with a 3160-vocabulary on SELFIES-encoded molecules to enable robust sequence modeling.
• Apply masked language modeling during pretraining by randomly masking 15% of tokens and reconstructing them to learn contextual molecular representations.
• Construct a unified feature vector for each electrolyte formulation by scaling individual molecular representations (from the pretrained BART) by their molar concentration and summing them.
• Fine-tune the resulting feature vector on downstream battery property prediction tasks such as Coulombic Efficiency and Specific Capacity prediction.
• Avoid dummy featurization by using concentration-weighted vector summation, enabling flexibility across formulations with variable numbers of components.

Figure 1: Pre-training model architecture

Experimental results

Research questions

• RQ1 Can a self-supervised transformer model trained on SELFIES-encoded molecular sequences improve molecular representation learning for complex electrolyte formulations?
• RQ2 Does concentration-aware scaling of molecular embeddings enhance the predictive performance of machine learning models for battery electrolyte properties?
• RQ3 How does the proposed method compare to state-of-the-art models like F-GCN, MolFormer, and MM-MolFormer in predicting electrolyte performance?
• RQ4 To what extent does the use of SELFIES instead of SMILES improve model generalization and reduce invalid molecular representation learning?
• RQ5 Can the model generalize across diverse electrolyte formulations with variable numbers of components without requiring dummy padding or fixed-size feature engineering?

Key findings

• The proposed BART-SA model achieved an RMSE of 0.148 on the Coulombic Efficiency (LCE) prediction task, outperforming all prior methods including F-GCN TL, MolFormer, and MM-MolFormer.
• On the Specific Capacity prediction task, BART-SA achieved an RMSE of 20.001 mAh/g, surpassing F-GCN TL (20.495 mAh/g) and F-GCN no-TL (39.823 mAh/g).
• The model demonstrated superior performance on high-performing electrolytes, with predicted values closely aligning with actual values in parity plots for both tasks.
• The model showed particular strength in predicting high-performance electrolytes, while higher errors were observed for low-capacity cells due to experimental noise and instability.
• The use of SELFIES during pretraining significantly enhanced the quality of molecular representations, reducing the risk of learning invalid or non-chemical structures.
• The concentration-weighted vector summation strategy enabled a consistent, scalable, and generalizable feature representation without dummy padding, supporting variable-component formulations.

Figure 2: Illustration of the general schematic of the proposed method. (a) shows the general format of the electrolyte formulation dataset. (b) describes the procedure to construct the feature vector for an electrolyte formulation. (c) shows the fine-tuning model trained using the feature vector fo