[Paper Review] DeepMoLM: Leveraging Visual and Geometric Structural Information for Molecule-Text Modeling

Jing Lan, Hexiao Ding+9 more|arXiv (Cornell University)|Jan 21, 2026

Machine Learning in Materials Science0 citations

TL;DR

DeepMoLM fuses high-resolution molecular images with 3D conformer-derived fingerprints via cross-attention to ground vision-language representations in geometric invariants, enabling chemically plausible captioning and property prediction without atom coordinates.

ABSTRACT

AI models for drug discovery and chemical literature mining must interpret molecular images and generate outputs consistent with 3D geometry and stereochemistry. Most molecular language models rely on strings or graphs, while vision-language models often miss stereochemical details and struggle to map continuous 3D structures into discrete tokens. We propose DeepMoLM: Deep Molecular Language M odeling, a dual-view framework that grounds high-resolution molecular images in geometric invariants derived from molecular conformations. DeepMoLM preserves high-frequency evidence from 1024 $ imes$ 1024 inputs, encodes conformer neighborhoods as discrete Extended 3-Dimensional Fingerprints, and fuses visual and geometric streams with cross-attention, enabling physically grounded generation without atom coordinates. DeepMoLM improves PubChem captioning with a 12.3% relative METEOR gain over the strongest generalist baseline while staying competitive with specialist methods. It produces valid numeric outputs for all property queries and attains MAE 13.64 g/mol on Molecular Weight and 37.89 on Complexity in the specialist setting. On ChEBI-20 description generation from images, it exceeds generalist baselines and matches state-of-the-art vision-language models. Code is available at https://github.com/1anj/DeepMoLM.

Motivation & Objective

• Motivate grounding molecular understanding in both visual and geometric information to preserve stereochemistry and 3D structure in language outputs.
• Develop a dual-view DeepEncoder that preserves high-frequency visual cues and incorporates discrete 3D fingerprints.
• Introduce a cross-attention Fusion Projector to align visual tokens with Extended 3D Fingerprints (E3FP) for geometrically grounded generation.
• Demonstrate improvements on molecule captioning, description, and property prediction compared to specialist and generalist baselines.

Proposed method

• Use a dual-pathway DeepEncoder to process 1024×1024 molecular images via a SAM-Base local encoder and a CLIP-Large global encoder with a convolutional token compressor.
• Represent conformer geometry with Extended 3-Dimensional Fingerprints (E3FP) and align them to canonical SELFIES tokens through a discrete token embedding fusion.
• Apply a cross-attention Fusion Projector that queries geometric descriptors from visual tokens to ground image features before decoding.
• Decode outputs with a Vision-Language Model (Qwen2-VL) in an autoregressive framework conditioned on fused multimodal representations.
• Train in two stages: Stage-1 vision-language alignment with a frozen decoder, Stage-2 multimodal instruction tuning with the decoder.

Experimental results

Research questions

• RQ1 Can grounding molecular images with discrete 3D geometric fingerprints improve alignment between visual and textual modalities in molecule understanding?
• RQ2 Does the fusion of high-resolution visual cues with E3FP-based geometry yield better captioning and property prediction than image-only or geometry-only baselines?
• RQ3 How does preserving stereochemical information affect generation quality and validity of molecular outputs?
• RQ4 Are 3D fingerprints and pre-training essential for robust multimodal grounding in molecular tasks?

Key findings

• DeepMoLM achieves a METEOR gain of 12.3% relative to the strongest generalist baseline on PubChem captioning.
• In the specialist setting, DeepMoLM attains MAE 13.64 g/mol on Molecular Weight and 37.89 on Complexity for property prediction, and provides valid numeric outputs for all queries.
• On ChEBI-20 description generation, it exceeds generalist baselines and matches state-of-the-art vision-language models.
• Across molecule captioning and description, DeepMoLM outperforms several specialist models and most generalist baselines in key metrics such as ROUGE and METEOR.
• Ablation studies show pre-training, the 3D-E3FP branch, and the fusion projector are all essential for peak performance.