[论文解读] FlowLLM: Flow Matching for Material Generation with Large Language Models as Base Distributions

Anuroop Sriram, Benjamin Kurt Miller+2 more|arXiv (Cornell University)|Oct 30, 2024

Natural Language Processing Techniques被引用 7

一句话总结

FlowLLM 将微调后的大型语言模型作为基础分布与 Riemannian Flow Matching 相结合，以生成元稳定的晶体材料，在此前方法基础上实现超过 300% 的稳定性生成提升，SUN（稳定、唯一、新颖）率大约提高 50%，同时产生更接近其放松基态的结构。

ABSTRACT

Material discovery is a critical area of research with the potential to revolutionize various fields, including carbon capture, renewable energy, and electronics. However, the immense scale of the chemical space makes it challenging to explore all possible materials experimentally. In this paper, we introduce FlowLLM, a novel generative model that combines large language models (LLMs) and Riemannian flow matching (RFM) to design novel crystalline materials. FlowLLM first fine-tunes an LLM to learn an effective base distribution of meta-stable crystals in a text representation. After converting to a graph representation, the RFM model takes samples from the LLM and iteratively refines the coordinates and lattice parameters. Our approach significantly outperforms state-of-the-art methods, increasing the generation rate of stable materials by over three times and increasing the rate for stable, unique, and novel crystals by $\sim50\%$ - a huge improvement on a difficult problem. Additionally, the crystals generated by FlowLLM are much closer to their relaxed state when compared with another leading model, significantly reducing post-hoc computational cost.

研究动机与目标

• Motivate accelerated discovery of meta-stable crystalline materials by leveraging the strengths of large language models (LLMs) and flow-based refinement.
• Propose FlowLLM, a hybrid model where an LLM generates an initial crystal representation that is iteratively refined by a Riemannian Flow Matching (RFM) model.
• Demonstrate substantial improvements over state-of-the-art baselines in stability and SUN rates on the MP-20 dataset.
• Highlight the impact of using a learned base distribution from the LLM for RFM refinement and discuss implications for synthesis feasibility and computational efficiency.

提出的方法

• Fine-tune a pre-trained LLaMA-2 LLM on crystal-material strings to learn a base distribution of meta-stable materials.
• Sample from the LLM to obtain initial crystal representations (atom types, fractional coordinates, lattice parameters) and reject invalid crystals.
• Convert text outputs to a crystal representation and iteratively refine atom positions and lattice parameters via a Riemannian Flow Matching (RFM) model on a crystal manifold with periodic boundary conditions.
• Train the RFM velocity field v_t using a Conditional Flow Matching objective adapted to crystals, with geodesic-based supervision on the crystal manifold.
• Represent fractional coordinates on a flat torus and lattice parameters in Euclidean space; enforce symmetry through graph neural networks to achieve permutation, translation, and rotation equivariances.
• Use LLM as a learned base distribution and RFM as a denoising/refinement step to bridge discrete (atom types) and continuous (positions, lattice) variables while maintaining LLM prompting capability.

实验结果

研究问题

• RQ1 Can FlowLLM increase the generation rate of thermodynamically stable materials compared to prior generative models?
• RQ2 Does using an LLM as a learned base distribution for RFM improve stability and novelty (SUN) rates, and how does this compare to solely diffusion/flow-based approaches?
• RQ3 How close are FlowLLM-generated structures to their relaxed ground states, and does this reduce post-hoc relaxation cost?
• RQ4 What is the impact of conditioning (e.g., chemical formula) and sampling parameters (temperature, nucleus sampling) on quality metrics like stability and SUN rates?

主要发现

• FlowLLM generates stable materials at over 300% higher rate than the best prior method on MP-20.
• FlowLLM achieves SUN (stable, unique, novel) rates approximately 50% higher than prior methods.
• FlowLLM-generated structures are closer to their CHGNet-relaxed ground states than FlowMM, with higher Match Rate (94.9% vs 74.3%), lower RMSD (0.023 Å vs 0.096 Å), and lower ΔE per atom (0.0898 eV/atom vs 0.3031 eV/atom).
• FlowLLM converges with as few as ~50 RFM integration steps, faster than many diffusion/flow-based baselines.
• The FlowLLM-Types variant shows that leveraging an LLM mainly for accurate atom-type prediction still improves stability rates, underscoring the benefit of learned base distributions.
• FlowLLM provides a strong compromise between structural validity and coverage, contributing to superior stability and SUN metrics across multiple prompting and sampling setups.