[Paper Review] Learning Discrete Structures for Graph Neural Networks

Luca Franceschi, Mathias Niepert+2 more|arXiv (Cornell University)|Mar 28, 2019

Advanced Graph Neural Networks57 references146 citations

TL;DR

The paper proposes LDS, a bilevel framework that jointly learns a sparse, probabilistic graph generator and GCN parameters, enabling graph-based learning when the graph is missing or noisy. It uses hypergradient-based training with a straight-through estimator to optimize discrete edge variables.

ABSTRACT

Graph neural networks (GNNs) are a popular class of machine learning models whose major advantage is their ability to incorporate a sparse and discrete dependency structure between data points. Unfortunately, GNNs can only be used when such a graph-structure is available. In practice, however, real-world graphs are often noisy and incomplete or might not be available at all. With this work, we propose to jointly learn the graph structure and the parameters of graph convolutional networks (GCNs) by approximately solving a bilevel program that learns a discrete probability distribution on the edges of the graph. This allows one to apply GCNs not only in scenarios where the given graph is incomplete or corrupted but also in those where a graph is not available. We conduct a series of experiments that analyze the behavior of the proposed method and demonstrate that it outperforms related methods by a significant margin.

Motivation & Objective

• Motivate learning graph structure alongside GCN parameters when a graph is missing, incomplete, or noisy.
• Develop a bilevel optimization framework where the outer problem learns edge probabilities and the inner problem learns GCN weights.
• Introduce a practical algorithm using hypergradient descent with a straight-through estimator to handle discrete graph variables.
• Demonstrate that learned sparse graphs can outperform traditional graph construction approaches and that the model yields meaningful edge distributions.

Proposed method

• Model edges as independent Bernoulli random variables with parameters theta and sample graphs A ~ Ber(theta) to form a distribution over graphs.
• Formulate a bilevel objective: outer objective minimizes validation loss F(w_theta, A) with w_theta = argmin_w E_A~Ber(theta)[L(w, A)], inner objective minimizes training loss over sampled graphs.
• Approximate the inner and outer objectives via SGD on w with A_t ~ Ber(theta) and use truncated backpropagation through time to compute STE-based hypergradients for theta.
• Use a straight-through estimator to approximate gradients wrt discrete edges, yielding a biased but practically effective hypergradient.
• Estimate the model’s final predictions by Monte Carlo averaging over S sampled graphs: f_w^{exp}(X) ≈ E_A[f_w(X, A)], with an unbiased estimator hat{f}_w(X) = (1/S) sum_i f_w(X, A_i).
• Initialize theta with a kNN graph (theta initialized to 0/1 to reflect known edges) and optimize theta within the convex hull of adjacency matrices.

Experimental results

Research questions

• RQ1 Can a learned probabilistic graph generator jointly optimize with GCN parameters to improve semi-supervised node classification when the graph is missing or corrupted?
• RQ2 Do learned edge distributions tend to place higher probabilities on edges between nodes with the same class, and are the resulting graphs sparse yet informative?
• RQ3 Is a bilevel, gradient-based optimization with STE hypergradients effective for discrete graph structures in practice?
• RQ4 How does LDS perform compared to classical GCNs and other baselines under varying degrees of graph incompleteness?

Key findings

• LDS achieves competitive or superior accuracy compared to vanilla GCNs, especially as edges are removed, with accuracy gains up to several percentage points in some settings.
• The learned graphs remain sparse (often much less than the full graph) yet more informative, increasing edge probability for same-class node pairs.
• LDS outperforms kNN-based graph methods and even dense adjacency learning in several datasets, particularly when a genuine graph structure is available.
• Stepping beyond alternation, using truncated STE hypergradients (tau > 0) yields better performance than single-step or pure alternating optimization.
• The learned edge probabilities reveal meaningful structure, with higher probabilities for edges connecting nodes of the same class and partial recovery of true adjacency links.