[論文レビュー] Contrastive Laplacian Eigenmaps

Hao Zhu, Ke Sun+1 more|arXiv (Cornell University)|Jan 14, 2022

Advanced Graph Neural Networks被引用数 23

ひとこと要約

COLESはLaplacian EigenmapsをGANにインスパイアされたWassersteinベースの対照的損失で拡張し、無監督グラフ埋め込みにおいて複数のベースラインに対して精度とスケーラビリティの点で有利な結果を達成する。

ABSTRACT

Graph contrastive learning attracts/disperses node representations for similar/dissimilar node pairs under some notion of similarity. It may be combined with a low-dimensional embedding of nodes to preserve intrinsic and structural properties of a graph. In this paper, we extend the celebrated Laplacian Eigenmaps with contrastive learning, and call them COntrastive Laplacian EigenmapS (COLES). Starting from a GAN-inspired contrastive formulation, we show that the Jensen-Shannon divergence underlying many contrastive graph embedding models fails under disjoint positive and negative distributions, which may naturally emerge during sampling in the contrastive setting. In contrast, we demonstrate analytically that COLES essentially minimizes a surrogate of Wasserstein distance, which is known to cope well under disjoint distributions. Moreover, we show that the loss of COLES belongs to the family of so-called block-contrastive losses, previously shown to be superior compared to pair-wise losses typically used by contrastive methods. We show on popular benchmarks/backbones that COLES offers favourable accuracy/scalability compared to DeepWalk, GCN, Graph2Gauss, DGI and GRACE baselines.

研究の動機と目的

• Motivate and extend Laplacian Eigenmaps with contrastive learning to better separate dissimilar nodes.
• Derive COLES from a SampledNCE framework and a GAN-inspired formulation to address issues with overlapping distributions.
• Show that COLES minimizes a surrogate of Wasserstein distance, improving robustness when positive/negative distributions are disjoint.
• Demonstrate COLES effectiveness and scalability across standard graph benchmarks and backbones.

提案手法

• Replace the traditional sigmoid-based contrastive terms with a linear/exp formulation to align with a GAN-inspired objective.
• Formulate the learning as optimizing Tr(Y^T ΔW Y) with orthogonality constraints, where ΔW encodes positive/negative graph structure (Equation 5).
• Specialize COLES to Linear Graph Networks, reducing to a small SVD-based eigenproblem (Equation 6).
• Explore a Stiefel-manifold variant (COLES-GCN with orthogonality constraints) for richer embeddings.
• Show COLES corresponds to a block-contrastive loss, offering advantages over pairwise losses.

実験結果

リサーチクエスチョン

• RQ1 How can Laplacian Eigenmaps be reformulated in a contrastive learning framework?
• RQ2 Does a Wasserstein-distance surrogate provide robustness when positive and negative samples have disjoint distributions?
• RQ3 What are the implications of a block-contrastive loss for graph embeddings, and how does COLES perform across backbones like GCN, SGC, and S2GC?

主な発見

• COLES reformulates Laplacian Eigenmaps as a contrastive objective, derived from SampledNCE but aligned with Wasserstein distance principles.
• The method avoids JS-divergence issues by using a Lipschitz-continuous, GAN-inspired setup with a negative sampling scheme.
• COLES belongs to the block-contrastive loss family, which is theoretically favored over pairwise losses and improves generalization.
• Empirical results on standard benchmarks show COLES-based backbones (COLES-GCN, COLES-S2GC) outperform several unsupervised and contrastive baselines (e.g., DeepWalk, DGI, GRACE) in terms of accuracy and scalability.