[Paper Review] Augmented Transformer Achieves 97% and 85% for Top5 Prediction of Direct and Classical Retro-Synthesis.

Igor V. Tetko, Pavel Karpov+2 more|arXiv (Cornell University)|Mar 5, 2020

Computational Drug Discovery Methods33 references9 citations

TL;DR

This paper proposes an augmented Transformer model that improves retro-synthesis prediction by applying SMILES sequence augmentation to both input and target data, reducing data memorization and enhancing generalization. The method achieves 97% Top-5 accuracy on the USPTO-MIT separated set and 85.4% on the USPTO-50k test set for predicting the largest fragment in classical retro-synthesis.

ABSTRACT

We investigated the effect of different augmentation scenarios on predicting (retro)synthesis of chemical compounds using SMILES representation. We showed that augmentation of not only input sequences but also, importantly, of the target data eliminated the effect of data memorization by neural networks and improved their generalization performance for prediction of new sequences. The Top-5 accuracy was 85.4% for the prediction of the largest fragment (thus identifying principal transformation for classical retro-synthesis) for USPTO-50k test dataset and was achieved by a combination of SMILES augmentation and beam search. The same approach also outperformed best published results for prediction of direct reactions from the USPTO-MIT test set. Our model achieved 90.4% Top-1 and 96.5% Top-5 accuracy for its most challenging mixed set and 97% Top-5 accuracy for the USPTO-MIT separated set. The appearance frequency of the most abundantly generated SMILES was well correlated with the prediction outcome and can be used as a measure of the quality of reaction prediction.

Motivation & Objective

• To address data memorization in neural networks for chemical reaction prediction.
• To improve generalization performance in retro-synthesis prediction using sequence augmentation.
• To enhance Top-5 accuracy for identifying the principal transformation in classical retro-synthesis.
• To evaluate the impact of input and target data augmentation on model generalization.
• To correlate SMILES generation frequency with prediction quality for model interpretability.

Proposed method

• Applying SMILES sequence augmentation to both input and target sequences to reduce memorization effects.
• Using beam search during inference to improve prediction diversity and accuracy.
• Training a Transformer-based model on augmented data to enhance generalization.
• Measuring prediction quality via Top-1 and Top-5 accuracy on benchmark datasets.
• Analyzing the frequency of the most commonly generated SMILES as a proxy for prediction reliability.
• Evaluating performance on USPTO-50k and USPTO-MIT test sets, including mixed and separated reaction sets.

Experimental results

Research questions

• RQ1 Does augmenting both input and target sequences improve generalization in retro-synthesis prediction?
• RQ2 Can data augmentation reduce neural network memorization in chemical reaction prediction?
• RQ3 What is the impact of SMILES augmentation on Top-5 accuracy for identifying the largest fragment in classical retro-synthesis?
• RQ4 How does beam search interact with data augmentation to improve prediction performance?
• RQ5 Can the frequency of generated SMILES be used as a reliable indicator of prediction quality?

Key findings

• The model achieved 97% Top-5 accuracy on the USPTO-MIT separated test set for direct reaction prediction.
• For the USPTO-50k test set, the model reached 85.4% Top-5 accuracy in predicting the largest fragment, indicating the principal transformation.
• The model outperformed previous published results on the USPTO-MIT test set, achieving 90.4% Top-1 and 96.5% Top-5 accuracy on the most challenging mixed set.
• The appearance frequency of the most frequently generated SMILES was strongly correlated with correct prediction outcomes.
• Joint augmentation of input and target sequences significantly reduced memorization and improved model generalization.
• The combination of SMILES augmentation and beam search was key to achieving state-of-the-art performance across multiple benchmarks.