[Paper Review] Multimodal Transformer-based Model for Buchwald-Hartwig and Suzuki-Miyaura Reaction Yield Prediction

Shimaa Baraka, Ahmed M. El Kerdawy|arXiv (Cornell University)|Apr 27, 2022

Machine Learning in Materials Science4 citations

TL;DR

This paper proposes a multimodal Transformer-based model that combines SMILES representations and chemical descriptors to predict yields in Buchwald–Hartwig and Suzuki–Miyaura cross-coupling reactions. By fusing a fine-tuned BERT encoder for SMILES with a multi-layer perceptron for descriptors, the model achieves state-of-the-art performance with RMSE of 5.5 and R² of 0.959 on Buchwald–Hartwig data, and successfully suggests near-optimal reaction conditions, achieving 94% of the reported maximum yield.

ABSTRACT

Predicting the yield percentage of a chemical reaction is useful in many aspects such as reducing wet-lab experimentation by giving the priority to the reactions with a high predicted yield. In this work we investigated the use of multiple type inputs to predict chemical reaction yield. We used simplified molecular-input line-entry system (SMILES) as well as calculated chemical descriptors as model inputs. The model consists of a pre-trained bidirectional transformer-based encoder (BERT) and a multi-layer perceptron (MLP) with a regression head to predict the yield. We experimented on two high throughput experimentation (HTE) datasets for Buchwald-Hartwig and Suzuki-Miyaura reactions. The experiments show improvements in the prediction on both datasets compared to systems using only SMILES or chemical descriptors as input. We also tested the model's performance on out-of-sample dataset splits of Buchwald-Hartwig and achieved comparable results with the state-of-the-art. In addition to predicting the yield, we demonstrated the model's ability to suggest the optimum (highest yield) reaction conditions. The model was able to suggest conditions that achieves 94% of the optimum reported yields. This proves the model to be useful in achieving the best results in the wet lab without expensive experimentation.

Motivation & Objective

• To improve reaction yield prediction by integrating multiple data modalities—SMILES and chemical descriptors—beyond single-modality approaches.
• To develop a deep learning model capable of generalizing to out-of-distribution reaction conditions, especially under distribution shifts.
• To enable the model to suggest optimal reaction conditions that achieve high yields without costly experimental trials.
• To evaluate performance on high-throughput experimentation (HTE) datasets for two key cross-coupling reactions: Buchwald–Hartwig and Suzuki–Miyaura.

Proposed method

• The model employs a pre-trained bidirectional Transformer encoder (BERT) fine-tuned on SMILES strings to learn structural representations of reactants.
• Chemical descriptors are processed through a multi-layer perceptron (MLP) to encode molecular properties such as electronic and steric features.
• The outputs from the SMILES and descriptor branches are concatenated and passed through a regression head to predict reaction yield.
• The model is trained end-to-end on two HTE datasets: 3,955 data points for Buchwald–Hartwig and 5,760 for Suzuki–Miyaura, using 70/30 train-test splits.
• Hyperparameter optimization is performed on 1/7 of the first fold’s training set, following prior work for fair comparison.
• The model’s ability to suggest optimal conditions is evaluated by comparing top-k predicted conditions against experimentally reported best yields.

Experimental results

Research questions

• RQ1 Can combining SMILES and chemical descriptors improve yield prediction accuracy compared to models using only one modality?
• RQ2 How well does the multimodal model generalize to out-of-distribution test sets, particularly under distribution shifts in additives or ligands?
• RQ3 To what extent can the model suggest reaction conditions that achieve yields close to the experimentally reported optimum?
• RQ4 Does the model’s performance on out-of-sample data remain competitive with state-of-the-art models despite distribution shifts?

Key findings

• The multimodal model achieved an RMSE of 5.5 ± 0.3 and R² of 0.959 ± 0.005 on the Buchwald–Hartwig dataset, outperforming models using only SMILES or descriptors.
• On the Suzuki–Miyaura dataset, the model achieved an RMSE of 11.5 ± 0.3 and R² of 0.833 ± 0.01, showing consistent improvement over single-modality baselines.
• On out-of-sample Buchwald–Hartwig test sets with unseen isoxazole additives, the model achieved an average R² of 0.72, comparable to state-of-the-art.
• The model suggested reaction conditions that achieved an average yield of 0.82, representing 94% of the reported optimal yield (0.87) across 15 reactant pairs.
• The accuracy of including the true optimal condition in the top-k% of predictions reached 86.6% at 30% of the top suggestions, indicating strong practical utility.
• Performance dropped significantly (R² = 0.2 ± 0.25) on Suzuki–Miyaura when ligands in test sets were disjoint from training sets, indicating greater sensitivity to distribution shifts.