[論文レビュー] ProGCL: Rethinking Hard Negative Mining in Graph Contrastive Learning

Jun Xia, Lirong Wu+3 more|arXiv (Cornell University)|Oct 5, 2021

Advanced Graph Neural Networks被引用数 43

ひとこと要約

ProGCL は、グラフ対比学習における真のネガティブと偽のネガティブを区別する Beta Mixture Model ベースの測度を導入し、GCL におけるハードネガティブ採掘を改善する 2 つのスキーム（ProGCL-weight と ProGCL-mix）を提供します。

ABSTRACT

Contrastive Learning (CL) has emerged as a dominant technique for unsupervised representation learning which embeds augmented versions of the anchor close to each other (positive samples) and pushes the embeddings of other samples (negatives) apart. As revealed in recent studies, CL can benefit from hard negatives (negatives that are most similar to the anchor). However, we observe limited benefits when we adopt existing hard negative mining techniques of other domains in Graph Contrastive Learning (GCL). We perform both experimental and theoretical analysis on this phenomenon and find it can be attributed to the message passing of Graph Neural Networks (GNNs). Unlike CL in other domains, most hard negatives are potentially false negatives (negatives that share the same class with the anchor) if they are selected merely according to the similarities between anchor and themselves, which will undesirably push away the samples of the same class. To remedy this deficiency, we propose an effective method, dubbed \textbf{ProGCL}, to estimate the probability of a negative being true one, which constitutes a more suitable measure for negatives' hardness together with similarity. Additionally, we devise two schemes (i.e., \textbf{ProGCL-weight} and \textbf{ProGCL-mix}) to boost the performance of GCL. Extensive experiments demonstrate that ProGCL brings notable and consistent improvements over base GCL methods and yields multiple state-of-the-art results on several unsupervised benchmarks or even exceeds the performance of supervised ones. Also, ProGCL is readily pluggable into various negatives-based GCL methods for performance improvement. We release the code at \textcolor{magenta}{\url{https://github.com/junxia97/ProGCL}}.

研究の動機と目的

• Explain why existing hard negative mining methods from other domains underperform in Graph Contrastive Learning (GCL).
• Develop a probabilistic measure to distinguish true vs false negatives in GCL using a Beta Mixture Model.
• Propose two practical schemes (ProGCL-weight and ProGCL-mix) to leverage the new negative hardness measure.
• Demonstrate that ProGCL improves base GCL methods and achieves state-of-the-art results on unsupervised benchmarks.
• Show that ProGCL is pluggable into various negatives-based GCL approaches and scales to large graphs.

提案手法

• Model the distribution of negatives’ similarity with a two-component Beta Mixture Model (BMM) to estimate the probability that a negative is a true negative.
• Compute posterior probabilities p(c|s) for true/false negatives given similarity s, and use these probabilities to redefine negativity strength.
• Introduce ProGCL-weight: weight negative pairs by the joint measure of similarity and true-negative probability in the contrastive loss.
• Introduce ProGCL-mix: synthesize hard negatives by convex combinations of top-scoring inter-view negatives, weighted by their true-negative probabilities, and incorporate them into the loss.
• Provide two training schemes compatible with common GCL frameworks (transductive and inductive) with acceptable computational overhead (BMM fitting on M samples, EM iterations).

実験結果

リサーチクエスチョン

• RQ1 Why do existing hard negative mining methods from other domains underperform in Graph Contrastive Learning?
• RQ2 Can a mixture-model-based posterior probability distinguish true negatives from false negatives in GCL, and how can this be integrated into training?
• RQ3 Do ProGCL-weight and ProGCL-mix consistently improve performance across various GCL baselines and datasets?
• RQ4 Is ProGCL pluggable to different negatives-based GCL methods and scalable to large graphs?

主な発見

手法	Amazon-Photo	Amazon-Computers	Coauthor-CS	Wiki-CS
GCA ∗	92.55	87.82	92.40	78.26
ProGCL-weight	93.30	89.28	93.51	78.68
ProGCL-mix	93.64	89.55	93.67	78.45
Supervised GCN	92.42	86.51	93.03	77.19
Supervised GAT	92.56	86.93	92.31	77.65

• ProGCL consistently improves base GCL methods and achieves state-of-the-art results on several unsupervised benchmarks.
• A Beta Mixture Model better fits the distribution of negatives (true vs false) in GCL than Gaussian mixtures, enabling reliable posterior probabilities.
• ProGCL-weight and ProGCL-mix outperform baseline GCL on transductive and inductive node classification tasks across multiple datasets.
• ProGCL-mix often yields slightly better performance than ProGCL-weight, particularly on certain datasets.
• Across large-scale graphs (e.g., ogbn-arXiv), ProGCL maintains superior unsupervised performance compared to other baselines.
• ProGCL improves other negatives-based GCL methods (e.g., MERIT) when applied, demonstrating its plug-and-play nature.