[Paper Review] SMILES Transformer: Pre-trained Molecular Fingerprint for Low Data Drug Discovery

Shion Honda, Shoi Shi+1 more|arXiv (Cornell University)|Nov 12, 2019

Computational Drug Discovery Methods32 references163 citations

TL;DR

Introduces SMILES Transformer, a pre-trained Transformer-based fingerprint for molecules that yields data-efficient predictions on small datasets and competitive results on MoleculeNet benchmarks.

ABSTRACT

In drug-discovery-related tasks such as virtual screening, machine learning is emerging as a promising way to predict molecular properties. Conventionally, molecular fingerprints (numerical representations of molecules) are calculated through rule-based algorithms that map molecules to a sparse discrete space. However, these algorithms perform poorly for shallow prediction models or small datasets. To address this issue, we present SMILES Transformer. Inspired by Transformer and pre-trained language models from natural language processing, SMILES Transformer learns molecular fingerprints through unsupervised pre-training of the sequence-to-sequence language model using a huge corpus of SMILES, a text representation system for molecules. We performed benchmarks on 10 datasets against existing fingerprints and graph-based methods and demonstrated the superiority of the proposed algorithms in small-data settings where pre-training facilitated good generalization. Moreover, we define a novel metric to concurrently measure model accuracy and data efficiency.

Motivation & Objective

• Motivate the need for data-efficient molecular representations in drug discovery, especially with limited labeled data.
• Propose a Transformer-based, text-derived fingerprint learned from large unlabeled SMILES corpora.
• Demonstrate that the SMILES Transformer (ST) fingerprint supports simple predictors and yields strong data efficiency on MoleculeNet tasks.
• Introduce a data-efficiency metric (DEM) to evaluate performance with varying training data sizes.

Proposed method

• Build an encoder-decoder Transformer with four blocks and four-head attention to produce continuous molecular fingerprints from SMILES.
• Pre-train on 861,000 unlabeled SMILES from ChEMBL24 with a SMILES enumeration strategy and cross-entropy objective.
• Extract molecule-level fingerprints by pooling symbol-level outputs (mean, max, first-last-layer) to yield a 1024-dimensional vector.
• Compare ST fingerprints against ECFP4, RNNS2S, and GraphConv on 10 MoleculeNet datasets using MLP-style predictors.
• Define and compute a Data Efficiency Metric (DEM) that averages performance across exponentially increasing training set sizes.
• Visualize latent space with t-SNE to explore why ST fingerprints perform well on certain datasets.

Experimental results

Research questions

• RQ1 Does the ST fingerprint outperform traditional fingerprints and graph-based methods in small data regimes?
• RQ2 How data-efficient is ST compared to baselines when training data is scarce?
• RQ3 What properties of the ST latent space correlate with predictive performance across datasets?

Key findings

• ST achieves best DEM performance on 5 out of 10 MoleculeNet datasets, particularly in small-data regimes (ESOL, FreeSolv, BBBP, ClinTox).
• ST fingerprints, when combined with simple predictors (MLP, ridge/logistic regression), yield competitive or superior results to baselines in several tasks.
• ST is competitive with GraphConv and ECFP4 overall, and can match or exceed baselines in data-limited settings.
• Longer SMILES tend to enhance ST performance, indicating richer information content with longer sequences.
• A novel data-efficiency metric (DEM) effectively captures performance as training data size varies.