[Paper Review] GIT-Mol: A Multi-modal Large Language Model for Molecular Science with Graph, Image, and Text

Pengfei Liu, Yiming Ren+2 more|arXiv (Cornell University)|Aug 14, 2023

Computational Drug Discovery Methods47 references8 citations

TL;DR

GIT-Mol is a 700M multi-modal LLM that fuses graph, image, and text to improve molecule captioning, text-based molecule generation, image recognition, and property prediction via the GIT-Former modality mixer and Xmodal pre-training strategies.

ABSTRACT

Large language models have made significant strides in natural language processing, enabling innovative applications in molecular science by processing textual representations of molecules. However, most existing language models cannot capture the rich information with complex molecular structures or images. In this paper, we introduce GIT-Mol, a multi-modal large language model that integrates the Graph, Image, and Text information. To facilitate the integration of multi-modal molecular data, we propose GIT-Former, a novel architecture that is capable of aligning all modalities into a unified latent space. We achieve a 5%-10% accuracy increase in properties prediction and a 20.2% boost in molecule generation validity compared to the baselines. With the any-to-language molecular translation strategy, our model has the potential to perform more downstream tasks, such as compound name recognition and chemical reaction prediction.

Motivation & Objective

• Motivate and address the limitations of text-only LLMs in fully leveraging molecular graphs and images.
• Develop GIT-Mol to integrate graph, image, and text modalities into a unified latent space.
• Propose GIT-Former with cross-attention to fuse modalities and enable any-to-language translation.
• Demonstrate improvements in molecule captioning, de novo generation, image recognition, and property prediction.
• Provide ablations and analyses to validate the contribution of each modality and training strategy.

Proposed method

• Introduce GIT-Former, a cross-attention based modality mixer that maps graph, image, and text into a unified latent space.
• Use modality-specific encoders (MolT5 for text, Swin Transformer for images, GIN for graphs) and a MolT5 decoder for generation tasks.
• Pre-train with Xmodal-Text Matching (XTM) and Xmodal-Text Contrastive Learning (XTC) to align modalities.
• Apply any-to-language prompting during fine-tuning for modality translation tasks.
• Fine-tune on MoleculeNet-property tasks and use prompt tuning for language-based outputs.

Figure 1: An overview of GIT-Mol . (a) Internal Information , including sequence and graph structure representations, emphasizes inherent chemical properties and simple topology; (b) External Information , e.g., images and text descriptions, provide richer details and help the human understanding; (

Experimental results

Research questions

• RQ1 Can GIT-Former effectively align graph, image, and text modalities into a shared latent space for molecular tasks?
• RQ2 Do multi-modal inputs improve molecule captioning, image-based recognition, and SMILES generation compared to single modalities?
• RQ3 What is the impact of XTM and XTC training strategies on cross-modal alignment and downstream performance?
• RQ4 How does prompt learning influence any-to-language modality translation and property prediction?
• RQ5 What are the gains in molecular property prediction accuracy and molecule generation validity with GIT-Mol?

Key findings

• GIT-Mol achieves higher captioning performance than single-modality baselines across metrics.
• The graph-based variant typically outperforms SMILES in captioning metrics, with multi-modal outpacing both.
• Ablation shows 10–15% improvement from multi-modality over single modalities.
• In de novo generation, GIT-Mol-captions+MolT5 yields higher validity (0.928) and competitive similarity metrics.
• Cross-modal pre-training (XTM first, then XTC) and prompt learning significantly affect results.
• GIT-Mol outperforms baselines on several metrics in cross-modal molecule generation and property prediction tasks.

Figure 2: Study case of Molecule Caption . The GIT-Mol model exhibits precise chemical characterization, aligning closely with ground truth information.