[Paper Review] Does GNN Pretraining Help Molecular Representation?

Ruoxi Sun, Dai, Hanjun+1 more|arXiv (Cornell University)|Jul 13, 2022

Machine Learning in Materials Science18 citations

TL;DR

This paper investigates whether self-supervised graph neural network (GNN) pretraining improves molecular representation learning on small molecular datasets. Through systematic ablation studies across pretraining objectives, data splits, features, and hyperparameters, the authors find that pretraining often provides no statistically significant gains over training from scratch, and that hyperparameter tuning and feature engineering can outweigh pretraining benefits, challenging the assumed universality of GNN pretraining in molecular AI.

ABSTRACT

Extracting informative representations of molecules using Graph neural networks (GNNs) is crucial in AI-driven drug discovery. Recently, the graph research community has been trying to replicate the success of self-supervised pretraining in natural language processing, with several successes claimed. However, we find the benefit brought by self-supervised pretraining on small molecular data can be negligible in many cases. We conduct thorough ablation studies on the key components of GNN pretraining, including pretraining objectives, data splitting methods, input features, pretraining dataset scales, and GNN architectures, to see how they affect the accuracy of the downstream tasks. Our first important finding is, self-supervised graph pretraining do not always have statistically significant advantages over non-pretraining methods in many settings. Secondly, although noticeable improvement can be observed with additional supervised pretraining, the improvement may diminish with richer features or more balanced data splits. Thirdly, hyper-parameters could have larger impacts on accuracy of downstream tasks than the choice of pretraining tasks, especially when the scales of downstream tasks are small. Finally, we provide our conjectures where the complexity of some pretraining methods on small molecules might be insufficient, followed by empirical evidences on different pretraining datasets.

Motivation & Objective

• To evaluate whether self-supervised GNN pretraining provides consistent performance gains for molecular property prediction tasks.
• To isolate and analyze the impact of key experimental factors—such as data splitting, input features, pretraining objectives, and hyperparameters—on downstream performance.
• To challenge the assumption that pretraining universally improves molecular representation learning, especially on small-scale datasets.
• To investigate why some pretraining methods may fail to transfer knowledge effectively to downstream molecular tasks.
• To guide future research by identifying confounding factors that may mask or exaggerate pretraining benefits.

Proposed method

• The authors conduct systematic ablation studies across 13 experimental configurations, varying pretraining objectives (self-supervised, supervised, or both), input features (basic vs. rich), data splits (scaffold vs. balanced), and GNN architectures (GIN, GraphSage).
• They evaluate performance on five benchmark molecular datasets: BBBP, BACE, TOX21, TOXCAST, and SIDER, using standard binary classification metrics with repeated runs for statistical reliability.
• The study compares models trained from random initialization against those fine-tuned after pretraining on large-scale unlabeled datasets such as ZINC15 and SAVI.
• Hyperparameter search is performed for each setup, with learning rates and training repetitions varied to assess sensitivity.
• Empirical results are compared across configurations to isolate the contribution of pretraining from other factors like feature engineering and data split strategy.
• The authors reproduce results from prior works (e.g., GROVER, Hu et al.) to validate their experimental pipeline and assess reproducibility.

Experimental results

Research questions

• RQ1 Does self-supervised pretraining on large-scale molecular graphs lead to statistically significant improvements in downstream molecular property prediction tasks?
• RQ2 How do data splitting strategies (e.g., scaffold vs. random) affect the observed benefits of pretraining?
• RQ3 To what extent do hand-crafted features or hyperparameter tuning outweigh the gains from pretraining?
• RQ4 Can supervised pretraining after self-supervised pretraining yield consistent improvements, and under what conditions does this improvement diminish?
• RQ5 Why might current pretraining objectives fail to transfer knowledge effectively to small molecular datasets?

Key findings

• Self-supervised pretraining alone does not yield statistically significant performance gains over training from random initialization on most molecular benchmark tasks.
• When supervised pretraining is added after self-supervised pretraining, improvements are observed, but these gains are marginal or diminish when richer input features are used.
• The choice of hyperparameters—especially learning rates and number of training repeats—has a larger impact on downstream accuracy than the selection of pretraining objective, particularly on small datasets.
• Data splitting methods significantly influence results: scaffold splits often mask pretraining benefits, while balanced splits reveal more consistent trends.
• Reproducing prior claims of strong pretraining gains requires extensive hyperparameter tuning, and in some cases, the reported improvements may stem more from experimental setup than from pretraining itself.
• The authors hypothesize that the complexity of current pretraining objectives may be insufficient for small molecules, leading to limited transferable knowledge.