[Paper Review] Graph Neural Networks Go Forward-Forward

Daniele Paliotta, Mathieu Alain+2 more|arXiv (Cornell University)|Feb 10, 2023

Advanced Graph Neural Networks4 citations

TL;DR

This paper introduces Graph Forward-Forward (GFF), a novel training method for graph neural networks that replaces backpropagation with a forward-pass-only procedure using positive and negative graph samples. GFF trains GNNs greedily layer-by-layer by maximizing goodness on positive graphs and minimizing it on negative ones, achieving performance comparable to backpropagation while offering improved biological plausibility and reduced memory usage.

ABSTRACT

We present the Graph Forward-Forward (GFF) algorithm, an extension of the Forward-Forward procedure to graphs, able to handle features distributed over a graph's nodes. This allows training graph neural networks with forward passes only, without backpropagation. Our method is agnostic to the message-passing scheme, and provides a more biologically plausible learning scheme than backpropagation, while also carrying computational advantages. With GFF, graph neural networks are trained greedily layer by layer, using both positive and negative samples. We run experiments on 11 standard graph property prediction tasks, showing how GFF provides an effective alternative to backpropagation for training graph neural networks. This shows in particular that this procedure is remarkably efficient in spite of combining the per-layer training with the locality of the processing in a GNN.

Motivation & Objective

• To develop a biologically plausible, backpropagation-free training method for graph neural networks.
• To extend the Forward-Forward algorithm to graph-structured data with node features.
• To enable training GNNs using only forward passes, avoiding gradient computation and memory-intensive backward passes.
• To evaluate GFF on standard graph property prediction benchmarks and compare its performance and efficiency to standard backpropagation.
• To explore the impact of different goodness functions and negative sample generation strategies on GNN training.

Proposed method

• GFF extends the Forward-Forward algorithm to graphs by defining a goodness function based on aggregated node activations, typically the sum of squared activations across layers.
• Positive and negative graph samples are created by appending correct or incorrect label encodings to node features, respectively.
• The model is trained layer-by-layer in a greedy fashion: each layer's parameters are updated to maximize goodness on positive graphs and minimize it on negative graphs.
• The method is compatible with any message-passing GNN architecture and does not require differentiable components between layers.
• Negative samples are generated by assigning incorrect labels to existing graphs, and alternative strategies such as graph augmentation (e.g., virtual nodes) are explored.
• The training process avoids storing all activations, significantly reducing memory footprint compared to backpropagation.

Experimental results

Research questions

• RQ1 Can the Forward-Forward algorithm be successfully extended to graph-structured data with node features?
• RQ2 How does GFF perform in comparison to standard backpropagation on graph property prediction tasks?
• RQ3 What are the computational and memory advantages of GFF over backpropagation in GNN training?
• RQ4 How do different choices of goodness functions and negative sample generation methods affect GFF performance?
• RQ5 Can GFF effectively capture long-range dependencies in graphs despite the absence of top-down gradient flow?

Key findings

• GFF achieves performance on par with backpropagation across 11 standard graph property prediction datasets, demonstrating its viability as a training alternative.
• The memory consumption of GFF is significantly lower than backpropagation, especially in deep networks, due to on-the-fly activation storage rather than full activation caching.
• The method remains effective even when combining per-layer training with the local inductive bias of message-passing GNNs.
• Different goodness functions, such as sum of squared activations, yield strong results, and alternative aggregation methods may further improve performance.
• The lack of top-down information flow limits GFF’s ability to propagate long-range dependencies to earlier layers, suggesting a need for architectural enhancements.
• Negative samples generated via label corruption are effective, but graph augmentation techniques like virtual nodes may improve performance on long-range tasks.