[Paper Review] Simple Truncated SVD based Model for Node Classification on Heterophilic Graphs

Vijay Lingam, Rahul Ragesh+2 more|arXiv (Cornell University)|Jun 24, 2021

Advanced Graph Neural Networks17 references6 citations

TL;DR

This paper proposes a simple Truncated SVD (TSVD)-based model that outperforms state-of-the-art methods by up to ~30% on heterophilic graph node classification tasks. By applying TSVD to both node features and graph topology, the method captures high-frequency components through induced negative edges, effectively acting as a hard low-pass filter without relying on message-passing aggregation.

ABSTRACT

Graph Neural Networks (GNNs) have shown excellent performance on graphs that exhibit strong homophily with respect to the node labels i.e. connected nodes have same labels. However, they perform poorly on heterophilic graphs. Recent approaches have typically modified aggregation schemes, designed adaptive graph filters, etc. to address this limitation. In spite of this, the performance on heterophilic graphs can still be poor. We propose a simple alternative method that exploits Truncated Singular Value Decomposition (TSVD) of topological structure and node features. Our approach achieves up to ~30% improvement in performance over state-of-the-art methods on heterophilic graphs. This work is an early investigation into methods that differ from aggregation based approaches. Our experimental results suggest that it might be important to explore other alternatives to aggregation methods for heterophilic setting.

Motivation & Objective

• To address the poor performance of Graph Neural Networks (GNNs) on heterophilic graphs, where connected nodes often have different labels.
• To explore alternatives to message-passing aggregation mechanisms that dominate current GNN architectures.
• To improve node classification performance on heterophilic graphs using a simple, efficient, and interpretable method based on Truncated SVD.
• To investigate whether spectral decomposition of graph structure and features can yield better representations than learning-based aggregation.
• To demonstrate that TSVD-based regularization can act as an effective inductive bias for heterophilic settings.

Proposed method

• The method applies Truncated SVD (TSVD) independently to the node feature matrix and the graph adjacency matrix to extract low-rank approximations.
• It constructs a new graph by learning a weighted adjacency matrix from the left singular vectors of the TSVD decomposition, effectively inducing negative edges that capture high-frequency components.
• The model uses a concatenation-based architecture that combines the TSVD-processed features and the modified graph structure, avoiding standard neighborhood aggregation.
• The approach treats the number of singular values (k) as a hyperparameter, allowing flexible dimensionality control for both features and graph structure.
• Graph normalization is varied during training, with symmetric, row, and no normalization tested to assess impact on performance.
• The method is inspired by GPR-GNN, interpreting TSVD as a way to scale and select singular vectors, effectively acting as a hard low-pass filter.

Experimental results

Research questions

• RQ1 Can a simple TSVD-based method outperform message-passing GNNs on heterophilic graph node classification?
• RQ2 Does decomposing both node features and graph structure via TSVD lead to better representation learning in heterophilic settings?
• RQ3 Can induced negative edges from TSVD capture high-frequency patterns that improve classification on heterophilic graphs?
• RQ4 Is the performance gain from TSVD robust across diverse heterophilic benchmark datasets?
• RQ5 Can a non-aggregation-based approach achieve competitive or superior results compared to state-of-the-art GNNs on heterophilic graphs?

Key findings

• The HLP Concat model achieved a 74.17% accuracy on the Squirrel dataset, a 17.58% improvement over the baseline GPR-GNN (56.59%) and a 30% relative gain over state-of-the-art methods.
• Varying the number of truncated singular values (k) significantly improved performance, with Squirrel's accuracy rising from 56.59% to 74.17% when k was optimized.
• The model outperformed all baselines across all benchmark datasets, including Texas, Wisconsin, Actor, Chameleon, and Cornell, with consistent gains.
• Using variable k for both features and graph structure yielded better results than fixed k, indicating that optimal dimensionality varies per dataset.
• Graph normalization had a strong impact: treating normalization as a hyperparameter (symmetric, row, no) improved performance over fixed symmetric normalization, suggesting that standard normalization may constrain performance.
• TSNE visualization confirmed that the HLP Concat model produced more discernible, class-separated clusters than GPR-GNN, indicating superior embedding quality.