[Paper Review] BioT5+: Towards Generalized Biological Understanding with IUPAC Integration and Multi-task Tuning

Qizhi Pei, Lijun Wu+7 more|arXiv (Cornell University)|Feb 27, 2024

Chemical Reactions and Isotopes4 citations

TL;DR

BioT5+ enhances biological language modeling by integrating IUPAC names, multi-task instruction tuning, and numerical tokenization, significantly improving generalization across 15 diverse tasks on 21 benchmarks. It achieves state-of-the-art performance in classification, regression, and generation tasks, particularly excelling in description-guided protein design with a 0.972 normalized Smith-Waterman score.

ABSTRACT

Recent research trends in computational biology have increasingly focused on integrating text and bio-entity modeling, especially in the context of molecules and proteins. However, previous efforts like BioT5 faced challenges in generalizing across diverse tasks and lacked a nuanced understanding of molecular structures, particularly in their textual representations (e.g., IUPAC). This paper introduces BioT5+, an extension of the BioT5 framework, tailored to enhance biological research and drug discovery. BioT5+ incorporates several novel features: integration of IUPAC names for molecular understanding, inclusion of extensive bio-text and molecule data from sources like bioRxiv and PubChem, the multi-task instruction tuning for generality across tasks, and a numerical tokenization technique for improved processing of numerical data. These enhancements allow BioT5+ to bridge the gap between molecular representations and their textual descriptions, providing a more holistic understanding of biological entities, and largely improving the grounded reasoning of bio-text and bio-sequences. The model is pre-trained and fine-tuned with a large number of experiments, including \emph{3 types of problems (classification, regression, generation), 15 kinds of tasks, and 21 total benchmark datasets}, demonstrating the remarkable performance and state-of-the-art results in most cases. BioT5+ stands out for its ability to capture intricate relationships in biological data, thereby contributing significantly to bioinformatics and computational biology. Our code is available at \url{https://github.com/QizhiPei/BioT5}.

Motivation & Objective

• To address the lack of IUPAC name modeling in existing biological language models, which limits textual understanding of molecular structures.
• To improve model generalization across diverse biological tasks by replacing task-specific fine-tuning with a unified multi-task instruction tuning framework.
• To enhance regression performance by introducing a novel numerical tokenization method for better handling of molecular property prediction tasks.
• To expand the training data scope by incorporating extensive bio-text from bioRxiv and PubMed, and high-quality molecules from PubChem.
• To enable grounded, instruction-based generation of biological sequences, such as proteins matching specific functional descriptions.

Proposed method

• Integrates IUPAC names into the BioT5 framework to align formal molecular representations (e.g., SELFIES) with natural language descriptions found in scientific literature.
• Expands pre-training data with 1.2M bio-texts from bioRxiv and PubMed, and 1.5M molecules from PubChem to enrich contextual and structural knowledge.
• Employs multi-task instruction tuning across 15 diverse biological tasks, including classification, regression, and text-to-sequence generation, to improve zero-shot and few-shot generalization.
• Introduces a novel numerical tokenization technique that separates and encodes numerical values (e.g., pIC50, logP) to improve model performance on regression tasks.
• Fine-tunes the model on 21 benchmark datasets spanning drug discovery, protein function prediction, and sequence generation, using a unified T5-based architecture.
• Uses normalized Smith-Waterman alignment scores to evaluate protein sequence generation quality, ensuring robust comparison with ground truth sequences.

Experimental results

Research questions

• RQ1 Can integrating IUPAC names into a biological language model improve its understanding of molecular structures in natural language contexts?
• RQ2 Does multi-task instruction tuning outperform task-specific fine-tuning in terms of generalization across diverse biological NLP tasks?
• RQ3 Can a novel numerical tokenization strategy significantly improve performance on regression tasks in bio-sequences and bio-texts?
• RQ4 To what extent does expanding pre-training data with scientific literature and molecular databases enhance model performance and reasoning capabilities?
• RQ5 How well can the model generate biologically plausible protein sequences based on functional descriptions compared to existing models?

Key findings

• BioT5+ achieves state-of-the-art performance on 18 out of 21 benchmark datasets across classification, regression, and generation tasks.
• In description-guided protein design, BioT5+ achieves a normalized Smith-Waterman score of 0.972, significantly outperforming Galactica (0.109) and Mol-Instructions (0.428).
• The model demonstrates strong zero-shot generalization, particularly in regression tasks, due to its specialized numerical tokenization method.
• Multi-task instruction tuning enables the model to perform well across 15 different tasks without task-specific fine-tuning, reducing training cost and improving adaptability.
• The integration of IUPAC names enhances the model’s ability to link textual descriptions with molecular structures, improving grounded reasoning in biological text.
• BioT5+ achieves a normalized SW score of 0.916 on a second protein design benchmark, outperforming Mol-Instructions (0.328) and Galactica (0.119).