[Paper Review] Graph2Seq: Graph to Sequence Learning with Attention-based Neural Networks

Kun Xu, Lingfei Wu+4 more|arXiv (Cornell University)|Apr 3, 2018

Topic ModelingComputer Science53 references162 citations

TL;DR

Graph2Seq presents an end-to-end graph-to-sequence model with a bi-directional graph encoder and attention-based decoder, achieving state-of-the-art results on graph-structured to sequence tasks including bAbI, Shortest Path, and WikiSQL. The encoder aggregates directed neighborhood information and produces graph embeddings used by an attentive RNN decoder.

ABSTRACT

The celebrated Sequence to Sequence learning (Seq2Seq) technique and its numerous variants achieve excellent performance on many tasks. However, many machine learning tasks have inputs naturally represented as graphs; existing Seq2Seq models face a significant challenge in achieving accurate conversion from graph form to the appropriate sequence. To address this challenge, we introduce a novel general end-to-end graph-to-sequence neural encoder-decoder model that maps an input graph to a sequence of vectors and uses an attention-based LSTM method to decode the target sequence from these vectors. Our method first generates the node and graph embeddings using an improved graph-based neural network with a novel aggregation strategy to incorporate edge direction information in the node embeddings. We further introduce an attention mechanism that aligns node embeddings and the decoding sequence to better cope with large graphs. Experimental results on bAbI, Shortest Path, and Natural Language Generation tasks demonstrate that our model achieves state-of-the-art performance and significantly outperforms existing graph neural networks, Seq2Seq, and Tree2Seq models; using the proposed bi-directional node embedding aggregation strategy, the model can converge rapidly to the optimal performance.

Motivation & Objective

• Develop a general end-to-end graph-to-sequence model that maps graph inputs to sequence outputs.
• Learn expressive node and graph embeddings that incorporate edge directions.
• Enable attention-based decoding to align graph representations with output sequences.

Proposed method

• Graph encoder learns bi-directional node embeddings by aggregating forward and backward neighborhoods up to K hops with separate aggregators (mean, LSTM, pooling).
• Two graph embedding schemes (pooling-based and node-based with a supernode) construct graph-level representations from node embeddings.
• Attention-based RNN decoder uses a context vector computed over node embeddings to generate the target sequence.
• Alignment model a(j) scores between decoder state and node embeddings to produce attention weights for decoding.
• Training maximizes conditional log-probability of the correct description given the input graph; beam search with width 5 used at inference.
• The model is compatible with alternative graph encoders and decoders (e.g., GCNs, Tree/Set variants) for extensibility.

Experimental results

Research questions

• RQ1 Can a general encoder-decoder framework effectively translate graph-structured inputs to sequences across diverse tasks?
• RQ2 Does a bi-directional, edge-aware graph encoder improve graph-to-sequence performance, especially on larger graphs?
• RQ3 How does attention over node embeddings influence decoding quality for graph-to-sequence tasks?
• RQ4 How do different graph embedding strategies (pooling vs. node-based) impact performance on varied graph structures?
• RQ5 What gains arise when applying Graph2Seq to real-world graph-to-sequence tasks like SQL-to-English generation?

Key findings

• On WikiSQL, Graph2Seq variants outperform Seq2Seq, Tree2Seq, Graph2Seq baselines, with Graph2Seq-PGE achieving 38.97 BLEU-4.
• In synthetic and SP tasks, Graph2Seq demonstrates superior accuracy over LSTM and competitive or superior performance to GGS-NN and GCN, particularly on larger graphs.
• Ablation studies show attention significantly boosts performance (>=14.9% improvement) across DAG, DCG, and SEQ graphs.
• The mean aggregator with pooling-based graph embeddings generally yields strong results, and bidirectional aggregation improves performance over single-direction variants on larger graphs.
• Increasing hop size improves model performance and convergence, with bidirectional aggregation needing fewer hops to reach optimal performance.