[Paper Review] Graph Clustering with Graph Neural Networks

Anton Tsitsulin, John Palowitch+2 more|arXiv (Cornell University)|Jun 30, 2020

Advanced Graph Neural NetworksComputer Science67 references50 citations

TL;DR

The paper introduces Deep Modularity Networks (DMoN), an unsupervised GNN-based clustering method that optimizes a differentiable modularity-inspired objective with a collapse regularization to reliably recover graph clusters, achieving state-of-the-art results among pooling methods.

ABSTRACT

Graph Neural Networks (GNNs) have achieved state-of-the-art results on many graph analysis tasks such as node classification and link prediction. However, important unsupervised problems on graphs, such as graph clustering, have proved more resistant to advances in GNNs. Graph clustering has the same overall goal as node pooling in GNNs - does this mean that GNN pooling methods do a good job at clustering graphs? Surprisingly, the answer is no - current GNN pooling methods often fail to recover the cluster structure in cases where simple baselines, such as k-means applied on learned representations, work well. We investigate further by carefully designing a set of experiments to study different signal-to-noise scenarios both in graph structure and attribute data. To address these methods' poor performance in clustering, we introduce Deep Modularity Networks (DMoN), an unsupervised pooling method inspired by the modularity measure of clustering quality, and show how it tackles recovery of the challenging clustering structure of real-world graphs. Similarly, on real-world data, we show that DMoN produces high quality clusters which correlate strongly with ground truth labels, achieving state-of-the-art results with over 40% improvement over other pooling methods across different metrics.

Motivation & Objective

• Motivate unsupervised graph clustering within GNNs and bridge clustering objectives with deep learning.
• Propose a fully differentiable clustering objective inspired by modularity and spectral methods.
• Address trivial solutions via collapse regularization and dropout to stabilize training.
• Demonstrate improved clustering quality on synthetic and real-world graphs compared to baselines.

Proposed method

• Obtain soft cluster assignments C via softmax(GCN(Ã, X)).
• Optimize LDMoN(C; A) = −(1/2m) Tr(C⊤BC) + (√k/n) X_i C⊤_i F^−1, where B = A − dd⊤/2m.
• Decompose Tr(C⊤BC) into sparse A-term and degree normalization for O(d^2 n) per update.
• Introduce collapse regularization as Frobenius norm of soft cluster counts normalized to [0, √k], to avoid trivial solutions.
• Apply dropout to GNN representations before softmax to prevent poor local minima.
• Provide theoretical results showing collapse regularization avoids trivial clustering and preserves asymptotic consistency under DC-SBM.

Experimental results

Research questions

• RQ1 Can end-to-end differentiable clustering be achieved for attributed graphs within a GNN framework?
• RQ2 Does a modularity-inspired objective aid recovery of real-world graph clustering structures in an unsupervised setting?
• RQ3 Is a soft-clustering approach with a suitable regularizer robust to noise and feature-structure misalignment?
• RQ4 Do GNN-based clustering methods scale to large graphs without sacrificing clustering quality?

Key findings

• DMoN provides high-quality clusters on real-world data and correlates strongly with ground-truth labels.
• DMoN achieves state-of-the-art results with over 40% improvement over other pooling methods across different metrics.
• The collapse regularization effectively prevents trivial all-in-one-cluster solutions and improves training stability.
• Synthetic experiments show DMoN outperforms baselines under varying signal-to-noise scenarios in both graph structure and attributes.
• Theoretical results establish that the collapse regularization does not harm asymptotic consistency under DC-SBM.