[Paper Review] Structured Neural Summarization

Patrick Fernandes, Miltiadis Allamanis+1 more|arXiv (Cornell University)|Nov 5, 2018

Topic ModelingComputer Science38 references116 citations

TL;DR

The paper augments standard sequence encoders with a graph neural network (GNN) component to better capture long-range relationships in structured data, improving summarization across code and natural language tasks.

ABSTRACT

Summarization of long sequences into a concise statement is a core problem in natural language processing, requiring non-trivial understanding of the input. Based on the promising results of graph neural networks on highly structured data, we develop a framework to extend existing sequence encoders with a graph component that can reason about long-distance relationships in weakly structured data such as text. In an extensive evaluation, we show that the resulting hybrid sequence-graph models outperform both pure sequence models as well as pure graph models on a range of summarization tasks.

Motivation & Objective

• Motivate the use of explicit relational structure to improve long-sequence summarization.
• Propose a hybrid sequence encoder that integrates a GNN with a traditional sequence encoder.
• Demonstrate that the hybrid model outperforms pure sequence or pure graph baselines across multiple tasks.

Proposed method

• Encode input with a standard sequence model to obtain per-token representations.
• Initialize a gated graph neural network (GGNN) with these token representations as node features.
• Propagate information over a constructed graph that encodes relationships (e.g., NEXTTOKEN, IN, REF edges).
• Compute a graph-level representation via weighted readout and combine with the sequence representation for decoding.
• Train with standard sequence decoders (LSTM/Pointer) and evaluate on multiple summarization tasks.
• Provide reproducible code and data augmentations for replicability.

Experimental results

Research questions

• RQ1 Can integrating a graph component with a standard sequence encoder improve summarization on long or structured inputs?
• RQ2 What types of relational edges (e.g., token-level, syntax tree, coreference) most improve performance across tasks?
• RQ3 Do hybrid sequence-graph models outperform pure sequence and pure graph baselines across code and natural language summarization tasks?
• RQ4 How does the hybrid model perform with different decoder architectures (attention, copying) on these tasks?

Key findings

• Hybrid sequence-graph encoders outperform pure sequence and pure graph baselines across all tasks.
• For METHODNAMING and METHODDOC on code, GNN-augmented models surpass the state of the art with relatively simple graph structures.
• On NLSUMMARIZATION, the hybrid model is competitive with task-specific natural language models, though some metrics lag behind the strongest decoders in See et al. (2017).
• Ablation studies show that additional linguistic structure helps but the GNN still provides gains even with fewer linguistic cues.
• Long-range and cross-token relationships via graph edges aid in maintaining a global view for abstractive summaries.
• Key qualitative examples indicate improved handling of references and long-distance dependencies, with some repetition remaining as a challenge.