[Paper Review] From Canonical Correlation Analysis to Self-supervised Graph Neural Networks

Hengrui Zhang, Qitian Wu+3 more|arXiv (Cornell University)|Jun 23, 2021

Advanced Graph Neural NetworksComputer Science96 citations

TL;DR

CCA-SSG introduces a non-contrastive, feature-level objective for self-supervised graph representation learning, aligning two augmented views via canonical correlation while decorrelating feature dimensions, and matching or exceeding state-of-the-art on several node classification benchmarks without negative samples or MI estimators.

ABSTRACT

We introduce a conceptually simple yet effective model for self-supervised representation learning with graph data. It follows the previous methods that generate two views of an input graph through data augmentation. However, unlike contrastive methods that focus on instance-level discrimination, we optimize an innovative feature-level objective inspired by classical Canonical Correlation Analysis. Compared with other works, our approach requires none of the parameterized mutual information estimator, additional projector, asymmetric structures, and most importantly, negative samples which can be costly. We show that the new objective essentially 1) aims at discarding augmentation-variant information by learning invariant representations, and 2) can prevent degenerated solutions by decorrelating features in different dimensions. Our theoretical analysis further provides an understanding for the new objective which can be equivalently seen as an instantiation of the Information Bottleneck Principle under the self-supervised setting. Despite its simplicity, our method performs competitively on seven public graph datasets. The code is available at: https://github.com/hengruizhang98/CCA-SSG.

Motivation & Objective

• Motivate a simple SSL approach for graphs that avoids complex contrastive designs and negative samples.
• Propose a Canonical Correlation Analysis inspired self-supervised objective for graph representations.
• Show that decorrelating feature dimensions prevents dimensional collapse and links to Information Bottleneck principles.
• Provide theoretical insights and empirical evidence across multiple graph datasets.

Proposed method

• Generate two augmented views of the same input graph with random graph augmentations.
• Use a shared GNN encoder to obtain embeddings for both views.
• Normalize embeddings and compute an invariant loss between views.
• Add a decorrelation loss that pushes the cross-view feature covariances toward an identity matrix.
• Optimize the sum of an invariance term and a decorrelation term with a trading parameter lambda.
• Optionally relate the approach to Information Bottleneck under SSL settings.

Experimental results

Research questions

• RQ1 Can a non-contrastive, feature-level objective outperform contrastive SSL methods for graphs?
• RQ2 Does decorrelating feature dimensions prevent collapse and improve representation quality in SSL for graphs?
• RQ3 How does CCA-SSG relate to Information Bottleneck principles in a self-supervised graph setting?
• RQ4 What is the empirical performance of CCA-SSG on standard node classification benchmarks compared to existing SSL methods?

Key findings

• CCA-SSG achieves competitive or superior accuracy on seven node classification benchmarks, outperforming several baselines on multiple datasets.
• The method requires no negative samples, no parameterized mutual information estimators, no projector/predictor, and no asymmetric architectures.
• The invariance term preserves augmentation-invariant information, while the decorrelation term prevents dimensional collapse by decorrelating feature dimensions.
• Theoretical analysis shows the objective corresponds to an instantiation of the Information Bottleneck principle under self-supervised learning.
• Ablation studies show both invariance and decorrelation are essential, with decorrelation preventing collapse and invariance driving performance.
• Embedding dimension and hyperparameter lambda affect performance, with larger D helping up to a point and proper lambda balancing the two terms.