[Paper Review] SMILES-Mamba: Chemical Mamba Foundation Models for Drug ADMET Prediction

Bohao Xu, Yingzhou Lu+6 more|arXiv (Cornell University)|Aug 11, 2024

Computational Drug Discovery Methods4 citations

TL;DR

SMILES-Mamba is a two-stage molecular foundation model that leverages self-supervised pretraining on unlabeled SMILES strings followed by fine-tuning on labeled ADMET datasets, achieving state-of-the-art performance across 22 ADMET prediction tasks—ranking first in 14 tasks and top-2 in 17, demonstrating the power of self-supervised learning in reducing reliance on large labeled datasets for drug property prediction.

ABSTRACT

In drug discovery, predicting the absorption, distribution, metabolism, excretion, and toxicity (ADMET) properties of small-molecule drugs is critical for ensuring safety and efficacy. However, the process of accurately predicting these properties is often resource-intensive and requires extensive experimental data. To address this challenge, we propose SMILES-Mamba, a two-stage model that leverages both unlabeled and labeled data through a combination of self-supervised pretraining and fine-tuning strategies. The model first pre-trains on a large corpus of unlabeled SMILES strings to capture the underlying chemical structure and relationships, before being fine-tuned on smaller, labeled datasets specific to ADMET tasks. Our results demonstrate that SMILES-Mamba exhibits competitive performance across 22 ADMET datasets, achieving the highest score in 14 tasks, highlighting the potential of self-supervised learning in improving molecular property prediction. This approach not only enhances prediction accuracy but also reduces the dependence on large, labeled datasets, offering a promising direction for future research in drug discovery.

Motivation & Objective

• To address the high cost and data scarcity in ADMET property prediction by leveraging both unlabeled and labeled molecular data.
• To improve prediction accuracy for drug absorption, distribution, metabolism, excretion, and toxicity using self-supervised representation learning.
• To reduce dependency on large, expensive labeled datasets by pretraining on vast corpora of unlabeled SMILES strings.
• To develop a foundation model architecture that captures complex chemical structure relationships for better generalization across diverse ADMET tasks.

Proposed method

• Employs a two-stage training paradigm: self-supervised pretraining on a large corpus of unlabeled SMILES strings, followed by task-specific fine-tuning on labeled ADMET datasets.
• Uses the Mamba architecture—a state space model (SSM) variant—optimized for long-range sequence modeling to capture long-range dependencies in SMILES sequences.
• Applies a masked language modeling (MLM) objective during pretraining to predict masked tokens in SMILES strings, enabling the model to learn chemical grammar and structural patterns.
• Fine-tunes the pretrained Mamba encoder on 22 diverse ADMET datasets using standard machine learning heads for both regression (e.g., LD50) and classification (e.g., hERG, AMES) tasks.
• Leverages SMILES as input representation, enabling efficient processing of molecular sequences while preserving chemical semantics.
• Integrates residual connections and layer normalization to stabilize training and improve convergence across diverse ADMET prediction tasks.

Experimental results

Research questions

• RQ1 Can a foundation model based on the Mamba architecture outperform existing SOTA models in predicting diverse ADMET properties using only SMILES input?
• RQ2 To what extent does self-supervised pretraining on large-scale unlabeled SMILES data improve downstream ADMET prediction performance?
• RQ3 How does the Mamba-based architecture compare to traditional models like GCN, CNN, and MLP in capturing long-range chemical dependencies in SMILES sequences?
• RQ4 Does the proposed two-stage training strategy (pretraining + fine-tuning) lead to consistent performance gains across multiple ADMET tasks with varying data distributions?
• RQ5 Can the model generalize effectively across different ADMET endpoints despite their distinct chemical and biological underpinnings?

Key findings

• SMILES-Mamba achieved the highest performance in 14 out of 22 ADMET prediction tasks, demonstrating strong generalization across diverse drug property endpoints.
• On the hERG dataset, SMILES-Mamba achieved an ROC-AUC of 0.708 ± 0.045, ranking second among all methods, with a notable improvement in stability over baseline models.
• On the DILI dataset, SMILES-Mamba achieved an ROC-AUC of 0.928 ± 0.022, significantly outperforming the second-best method (NeuralFP at 0.851 ± 0.026).
• The model achieved top-2 performance in 17 out of 22 tasks, indicating consistent and robust performance across diverse ADMET prediction challenges.
• Self-supervised pretraining was shown to be highly effective, with models like SMILES-Mamba and NeuralFP significantly outperforming traditional models without pretraining.
• No single model dominated all tasks, highlighting the importance of architectural design and representation learning in capturing task-specific chemical features.