[논문 리뷰] GResNet: Graph Residual Network for Reviving Deep GNNs from Suspended Animation

Jiawei Zhang, Lin Meng|arXiv (Cornell University)|2019. 09. 12.

Advanced Graph Neural Networks참고 문헌 41인용 수 46

한 줄 요약

이 논문은 GResNet을 소개하는 그래프 잔차 학습 프레임워크로, 깊은 GNN에서 노드 표현을 보존하기 위한 광범위한 하이웨이 연결을 생성하고, suspended animation 문제를 해결하며 표준 그래프 벤치마크에서 성능을 향상시킨다.

ABSTRACT

The existing graph neural networks (GNNs) based on the spectral graph convolutional operator have been criticized for its performance degradation, which is especially common for the models with deep architectures. In this paper, we further identify the suspended animation problem with the existing GNNs. Such a problem happens when the model depth reaches the suspended animation limit, and the model will not respond to the training data any more and become not learnable. Analysis about the causes of the suspended animation problem with existing GNNs will be provided in this paper, whereas several other peripheral factors that will impact the problem will be reported as well. To resolve the problem, we introduce the GResNet (Graph Residual Network) framework in this paper, which creates extensively connected highways to involve nodes' raw features or intermediate representations throughout the graph for all the model layers. Different from the other learning settings, the extensive connections in the graph data will render the existing simple residual learning methods fail to work. We prove the effectiveness of the introduced new graph residual terms from the norm preservation perspective, which will help avoid dramatic changes to the node's representations between sequential layers. Detailed studies about the GResNet framework for many existing GNNs, including GCN, GAT and LoopyNet, will be reported in the paper with extensive empirical experiments on real-world benchmark datasets.

연구 동기 및 목표

• Identify and analyze the suspended animation problem in deep GCN/GNN models when increasing depth.
• Propose graph residual terms and a Graph Residual Network (GResNet) architecture to maintain information flow across layers.
• Theoretically analyze norm preservation to explain why graph residuals help avoid representation collapse.
• Empirically validate GResNet with deep architectures on standard graph datasets (e.g., Cora, Citeseer, PubMed) and compare with baseline GNNs (GCN, GAT, LoopyNet).

제안 방법

• Define and analyze the suspended animation limit for spectral GCNs in terms of the graph structure and eigenvalues of the normalized adjacency matrix.
• Propose graph residual terms (naive, graph-naive, raw, graph-raw) to replace or augment the standard layer update, enabling extensive inter-layer feature propagation.
• Formulate the GResNet layer update as H^(k) = ReLU( Ã H^(k-1) W^(k-1) + R(H^(k-1), X; G) ), with various R terms that encode node- and graph-level residuals.
• Provide theoretical justifications from a norm-preservation perspective, showing how residual terms stabilize gradient flow and preserve representations across layers.
• Demonstrate through experiments that deeper GResNet variants avoid suspended animation and outperform vanilla GCN/GAT/LoopyNet baselines on benchmark datasets.

실험 결과

연구 질문

• RQ1 What causes the suspended animation phenomenon in deep GNNs with spectral graph convolutions?
• RQ2 Can graph-aware residual terms help preserve node representations across many layers and extend the effective depth of GNNs?
• RQ3 How do different graph residual formulations compare in terms of accuracy and training stability on standard benchmarks?
• RQ4 Do theoretical insights about norm preservation explain the empirical benefits of GResNet?
• RQ5 How does GResNet perform across vanilla GCN, GAT, and LoopyNet backbones on real-world datasets?

주요 결과

• Deep GCN/GNNs exhibit a suspended animation limit where increasing depth degrades learning and eventually stops responding to data.
• Graph residual terms that are correlated with the graph structure (graph-naive, graph-raw) improve deep architectures, enabling deeper networks with better accuracy.
• GResNet variants with graph residuals achieve higher accuracy than vanilla baselines on benchmark datasets such as Cora, Citeseer, and PubMed, across multiple backbones (GCN, GAT, LoopyNet).
• The paper provides a norm-preservation analysis showing that residual terms help stabilize representations across layers, supporting effective training of deep GNNs.
• Empirical results indicate that deeper GResNet models can outperform shallow counterparts, suggesting the proposed residual design mitigates deterioration due to depth.