[Paper Review] CrysMMNet: Multimodal Representation for Crystal Property Prediction

Kishalay Das, Pawan Goyal+3 more|arXiv (Cornell University)|Jun 9, 2023

Machine Learning in Materials Science9 citations

TL;DR

CrysMMNet combines crystal graph structure with textual material descriptions to produce a joint multimodal representation for predicting crystal properties, outperforming state-of-the-art baselines on two benchmarks.

ABSTRACT

Machine Learning models have emerged as a powerful tool for fast and accurate prediction of different crystalline properties. Exiting state-of-the-art models rely on a single modality of crystal data i.e. crystal graph structure, where they construct multi-graph by establishing edges between nearby atoms in 3D space and apply GNN to learn materials representation. Thereby, they encode local chemical semantics around the atoms successfully but fail to capture important global periodic structural information like space group number, crystal symmetry, rotational information, etc, which influence different crystal properties. In this work, we leverage textual descriptions of materials to model global structural information into graph structure and learn a more robust and enriched representation of crystalline materials. To this effect, we first curate a textual dataset for crystalline material databases containing descriptions of each material. Further, we propose CrysMMNet, a simple multi-modal framework, which fuses both structural and textual representation together to generate a joint multimodal representation of crystalline materials. We conduct extensive experiments on two benchmark datasets across ten different properties to show that CrysMMNet outperforms existing state-of-the-art baseline methods with a good margin. We also observe that fusing the textual representation with crystal graph structure provides consistent improvement for all the SOTA GNN models compared to their own vanilla versions. We have shared the textual dataset, that we have curated for both the benchmark material databases, with the community for future use.

Motivation & Objective

• Motivate improving crystal property prediction by incorporating global periodic information often missed by graph-only models.
• Propose a multimodal framework that fuses graph-based local structure with text-based global structural knowledge.
• Curate a textual dataset for crystalline materials using Robocrystallographer to capture space group, symmetry, and dimensionality.
• Evaluate CrysMMNet on large crystal datasets across multiple properties and analyze ablations to understand modality contributions.

Proposed method

• Represent crystals as a graph encoding local chemical/structural information via a GNN (graph encoder).
• Encode global periodic knowledge from material descriptions with a transformer-based text encoder (MatSciBERT) and a projection layer.
• Fuse graph and text embeddings through concatenation to form a joint multimodal representation.
• Train an end-to-end MLP on the multimodal embedding to predict crystal properties using MSE loss.
• Curate textual descriptions for MP and JARVIS datasets using Robocrystallographer to capture local and global information.

Experimental results

Research questions

• RQ1 Can textual descriptions of crystalline materials provide global periodic structural information that improves property prediction when combined with graph-based representations?
• RQ2 Does multimodal fusion consistently outperform single-modality models across diverse crystal properties and datasets?
• RQ3 How does the textual modality contribute relative to local compositional information, and how robust is the approach to different GNN architectures?
• RQ4 What is the impact of limited training data on multimodal crystal property prediction?

Key findings

• CrysMMNet outperforms state-of-the-art baseline models across properties on the JARVIS-DFT and Materials Project datasets.
• Multimodal fusion provides consistent improvements for various GNN backbones compared to their vanilla versions.
• Textual representations alone are not sufficient to replace graph-based models, but they contribute global structural knowledge that enhances performance when fused.
• In low-data regimes, CrysMMNet maintains strong performance, showing robustness with limited labeled examples.
• Ablations show that both local (composition) and global (space group, symmetry) knowledge from text contribute to improved predictions.