[Paper Review] CaEGCN: Cross-Attention Fusion based Enhanced Graph Convolutional Network for Clustering

Guangyu Huo, Yong Zhang+4 more|arXiv (Cornell University)|Jan 18, 2021

Advanced Graph Neural Networks50 references51 citations

TL;DR

This paper proposes CaEGCN, a cross-attention fusion-based deep clustering framework that integrates content-aware autoencoders (CAE) and graph convolutional autoencoders (GAE) through a novel cross-attention mechanism to mitigate GCN over-smoothing. By jointly reconstructing data content and graph relationships while enforcing self-supervised consistency, CaEGCN achieves state-of-the-art clustering performance across diverse datasets, outperforming baselines by up to 28.48% in accuracy on the ACM dataset.

ABSTRACT

With the powerful learning ability of deep convolutional networks, deep clustering methods can extract the most discriminative information from individual data and produce more satisfactory clustering results. However, existing deep clustering methods usually ignore the relationship between the data. Fortunately, the graph convolutional network can handle such relationship, opening up a new research direction for deep clustering. In this paper, we propose a cross-attention based deep clustering framework, named Cross-Attention Fusion based Enhanced Graph Convolutional Network (CaEGCN), which contains four main modules: the cross-attention fusion module which innovatively concatenates the Content Auto-encoder module (CAE) relating to the individual data and Graph Convolutional Auto-encoder module (GAE) relating to the relationship between the data in a layer-by-layer manner, and the self-supervised model that highlights the discriminative information for clustering tasks. While the cross-attention fusion module fuses two kinds of heterogeneous representation, the CAE module supplements the content information for the GAE module, which avoids the over-smoothing problem of GCN. In the GAE module, two novel loss functions are proposed that reconstruct the content and relationship between the data, respectively. Finally, the self-supervised module constrains the distributions of the middle layer representations of CAE and GAE to be consistent. Experimental results on different types of datasets prove the superiority and robustness of the proposed CaEGCN.

Motivation & Objective

• To address the limitation of existing deep clustering methods that ignore structural relationships between data.
• To mitigate the over-smoothing problem in Graph Convolutional Networks (GCNs) during deep clustering.
• To enhance clustering performance by fusing heterogeneous representations—data content and graph structure—through an end-to-end framework.
• To develop a self-supervised module that guides the model toward optimal clustering by aligning intermediate representations from CAE and GAE modules.

Proposed method

• Proposes a cross-attention fusion module that layer-by-layer combines representations from a Content Auto-encoder (CAE) and a Graph Convolutional Auto-encoder (GAE), assigning attention weights to highlight discriminative features.
• Introduces a novel GAE module with two distinct loss functions: one for reconstructing node content and another for reconstructing graph structure, enhancing representation quality.
• Employs a self-supervised module that constrains the distributions of middle-layer representations from CAE and GAE to be consistent, promoting clustering-oriented feature learning.
• Uses a two-stage training process: first pre-training the CAE and GAE modules separately, then jointly fine-tuning the entire CaEGCN model end-to-end.

Experimental results

Research questions

• RQ1 Can cross-attention fusion effectively combine content and structural representations to improve deep clustering?
• RQ2 How does the proposed fusion mechanism mitigate the over-smoothing problem in GCNs during deep clustering?
• RQ3 To what extent do separate content and graph reconstruction losses enhance clustering performance compared to single-loss baselines?
• RQ4 How robust is CaEGCN across diverse data types, including text, human behavior, and image datasets?

Key findings

• CaEGCN achieves a 28.48% accuracy improvement over VGAE on the ACM dataset, demonstrating its superiority in handling high-order structural relationships.
• On the DBLP dataset, CaEGCN outperforms ARGA by 19.47% and DAEGC by 12.48%, confirming its effectiveness in complex graph-structured data.
• The ablation study shows that removing the cross-attention module leads to the sharpest performance drop, proving its critical role in fusing heterogeneous representations.
• The self-supervised module significantly improves clustering by aligning intermediate representations, with CaEGCN w/o self-supervision showing reduced performance across all datasets.
• On the USPS dataset, CaEGCN maintains strong performance despite limited graph connectivity, indicating robustness to sparse or low-quality initial graphs.
• The model achieves state-of-the-art results on all five benchmark datasets, with consistent improvements over SDCN, the previous SOTA method, by 3.4% in accuracy.