[Paper Review] Benchmarking Graphormer on Large-Scale Molecular Modeling Datasets

Yu Shi, Shuxin Zheng+8 more|arXiv (Cornell University)|Mar 9, 2022

Machine Learning in Materials Science24 citations

TL;DR

This technical note updates Graphormer with architecture tweaks and 3D adaptations, achieving stronger results on large-scale molecular datasets and proving global attention can surpass classic GNNs in expressiveness.

ABSTRACT

This technical note describes the recent updates of Graphormer, including architecture design modifications, and the adaption to 3D molecular dynamics simulation. With these simple modifications, Graphormer could attain better results on large-scale molecular modeling datasets than the vanilla one, and the performance gain could be consistently obtained on 2D and 3D molecular graph modeling tasks. In addition, we show that with a global receptive field and an adaptive aggregation strategy, Graphormer is more powerful than classic message-passing-based GNNs. Empirically, Graphormer could achieve much less MAE than the originally reported results on the PCQM4M quantum chemistry dataset used in KDD Cup 2021. In the meanwhile, it greatly outperforms the competitors in the recent Open Catalyst Challenge, which is a competition track on NeurIPS 2021 workshop, and aims to model the catalyst-adsorbate reaction system with advanced AI models. All codes could be found at https://github.com/Microsoft/Graphormer.

Motivation & Objective

• Investigate how Graphormer variants perform on large-scale molecular property prediction tasks.
• Assess the impact of architecture choices such as layer normalization placement on model generalization.
• Extend Graphormer to 3D molecular graphs with spatial and centrality encodings.
• Evaluate Graphormer on 3D molecular datasets like OC20 for energy prediction accuracy.
• Provide theoretical insight into Graphormer expressiveness via distributed computing concepts.

Proposed method

• Compare Pre-LN vs Post-LN Transformer variants within Graphormer on PCQM4M v1 and v2 datasets.
• Scale Graphormer to 24 layers and 1024 hidden units to assess depth impact.
• Adapt Graphormer with 3D spatial and centrality encodings and a 3D attention layer for rotational equivariance.
• Apply Gaussian basis functions to encode Euclidean distances in spatial encoding.
• Evaluate on the OC20 electrocatalyst dataset with repeated outputs to boost accuracy, plus auxiliary node-level objectives.
• Discuss expressiveness using CONGEST vs CONGESTED CLIQUE models to illustrate global receptive field benefits.

Experimental results

Research questions

• RQ1 How do different layer normalization placements (Pre-LN vs Post-LN) affect Graphormer performance on large-scale molecular property prediction?
• RQ2 Does extending Graphormer to 3D molecular graphs improve predictive accuracy on 3D datasets like OC20?
• RQ3 Can Graphormer with a global receptive field outperform traditional MPGNNs in expressiveness on molecular graphs?
• RQ4 What is the impact of adaptive attention and 3D encodings on energy prediction for electrocatalyst datasets?
• RQ5 Do 2D and 3D Graphormer variants generalize consistently across large-scale molecular modeling benchmarks (PCQM4M v1/v2 and OC20)?

Key findings

• Post-LN Graphormer variants generally yield better generalization on PCQM4M v1/v2 than Pre-LN variants for large models.
• Graphormer with 24 layers shows different generalization behavior, indicating potential for improved generalization with further tuning.
• 3D adaptations using spatial distances with Gaussian bases and rotationally-aware 3D attention improve 3D molecular modeling performance.
• On OC20 energy prediction, Graphormer achieves strong results, with ensemble variants providing notable gains.
• Theoretical framing suggests Graphormer’s global receptive field enhances expressiveness beyond local MP GNNs, aligning with distributed computing insights.
• Graphormer can significantly reduce MAE on PCQM4M datasets compared to earlier reports, and outperforms competitors on OC20 benchmarks.