[Paper Review] How to Find Your Friendly Neighborhood: Graph Attention Design with Self-Supervision

Dongkwan Kim, Alice Oh|arXiv (Cornell University)|Apr 11, 2022

Advanced Graph Neural Networks66 references116 citations

TL;DR

This paper introduces SuperGAT, a self-supervised graph attention model that uses edge presence as supervision to shape attention, with a recipe that selects attention design based on graph homophily and average degree; it generalizes across 17 real-world datasets.

ABSTRACT

Attention mechanism in graph neural networks is designed to assign larger weights to important neighbor nodes for better representation. However, what graph attention learns is not understood well, particularly when graphs are noisy. In this paper, we propose a self-supervised graph attention network (SuperGAT), an improved graph attention model for noisy graphs. Specifically, we exploit two attention forms compatible with a self-supervised task to predict edges, whose presence and absence contain the inherent information about the importance of the relationships between nodes. By encoding edges, SuperGAT learns more expressive attention in distinguishing mislinked neighbors. We find two graph characteristics influence the effectiveness of attention forms and self-supervision: homophily and average degree. Thus, our recipe provides guidance on which attention design to use when those two graph characteristics are known. Our experiment on 17 real-world datasets demonstrates that our recipe generalizes across 15 datasets of them, and our models designed by recipe show improved performance over baselines.

Motivation & Objective

• Assess what standard graph attention learns without supervision
• Introduce edge-based self-supervision to sharpen attention
• Compare GO and DP attention and derive two enhanced variants (SD and MX)
• Analyze how homophily and average degree influence attention effectiveness and provide a design recipe
• Validate the approach on multiple real-world and synthetic datasets

Proposed method

• Define SuperGAT variants based on GO and DP attention (GO, DP, and the proposed SD and MX)
• Introduce a self-supervised edge-prediction task with a_phi and phi_ij to supervise attention
• Use binary cross-entropy loss on edges (L_E) and combine with node-label loss (L_V) and L2 regularization to form the final objective
• Apply negative sampling to balance edge and non-edge pairs during training
• Maintain equivalent time/space complexity to GAT while adding edge-prediction supervision
• Share multi-head attention structure and compute edge probabilities via sigmoid over learned representations

Experimental results

Research questions

• RQ1 RQ1: Does graph attention learn label-agreement without edge supervision?
• RQ2 RQ2: Is graph attention predictive of edge presence when trained with edge supervision?
• RQ3 RQ3: Which graph attention design is most effective given a graph's characteristics (homophily and average degree)?
• RQ4 RQ4: Do design guidelines derived from synthetic graphs generalize to real-world datasets?

Key findings

• GO tends to align attention with label agreement better than DP in many settings
• DP attention more effectively encodes edge presence for link prediction across datasets
• A mixed design (MX) or a scaled-dot-product design (SD) can outperform standard GO or DP depending on graph properties, especially average degree and homophily
• A design recipe based on synthetic experiments generalizes to most real-world datasets (15 of 17), with MX often providing top performance in many regimes
• SuperGAT variants generally outperform baselines (GCN, GraphSAGE, GAT, and several edge-aware baselines) across a broad set of real-world datasets