[Paper Review] Toward Abstractive Summarization Using Semantic Representations

Fei Liu, Jeffrey Flanigan+3 more|arXiv (Cornell University)|May 25, 2018

Topic ModelingComputer Science41 references56 citations

TL;DR

This paper proposes a novel abstractive summarization framework that uses Abstract Meaning Representation (AMR) graphs to generate summaries by transforming multiple input sentence AMRs into a single, condensed summary AMR graph through concept merging, sentence conjunction, and optional graph expansion. The method achieves a ROUGE-1 F1 score of 57.8% on test data using JAMR parses, demonstrating the feasibility of graph-based abstractive summarization with semantic representations.

ABSTRACT

We present a novel abstractive summarization framework that draws on the recent development of a treebank for the Abstract Meaning Representation (AMR). In this framework, the source text is parsed to a set of AMR graphs, the graphs are transformed into a summary graph, and then text is generated from the summary graph. We focus on the graph-to-graph transformation that reduces the source semantic graph into a summary graph, making use of an existing AMR parser and assuming the eventual availability of an AMR-to-text generator. The framework is data-driven, trainable, and not specifically designed for a particular domain. Experiments on gold-standard AMR annotations and system parses show promising results. Code is available at: https://github.com/summarization

Motivation & Objective

• To develop a framework for abstractive summarization that leverages semantic representations like AMR to generate new, non-extractive summaries.
• To address the challenge of creating high-quality abstractive summaries for long or complex texts where extractive methods fall short.
• To explore the feasibility of using graph-to-graph transformation of AMR representations as a core mechanism for summarization.
• To isolate and evaluate the graph summarization component (step 2) independently from AMR parsing and text generation.
• To assess the impact of AMR parsing errors and graph expansion on summarization performance.

Proposed method

• Parse input sentences into AMR graphs using the JAMR parser, with gold-standard AMR annotations used for training and evaluation.
• Construct a unified source graph by merging coreferent nodes across sentences and connecting all sentence roots to a dummy 'ROOT' node.
• Apply optional graph expansion to create a fully dense graph on the sentence level, increasing the number of edges for richer semantic connectivity.
• Train a structured prediction model to select salient nodes and edges from the source graph to form a summary graph, using hinge, perceptron, or ramp loss functions.
• Generate summary text from the summary graph using word spans aligned by JAMR, evaluating via ROUGE-1 on a bag-of-words basis.
• Use oracle summaries (based on gold-standard AMR parses of reference summaries) to establish an upper bound on performance.

Experimental results

Research questions

• RQ1 Can a graph-to-graph transformation of AMR representations effectively condense multiple sentence meanings into a single summary graph?
• RQ2 How does the performance of the graph summarizer vary when using gold-standard AMR parses versus system-generated JAMR parses?
• RQ3 What is the impact of graph expansion on the quality of subgraph prediction and final summary generation?
• RQ4 How do different loss functions (hinge, perceptron, ramp) affect the performance of the structured prediction model?
• RQ5 To what extent do parsing errors in JAMR limit the overall summarization performance?

Key findings

• The framework achieves a ROUGE-1 F1 score of 57.8% on the test set when using JAMR parses, indicating strong potential for abstractive summarization using semantic graphs.
• Oracle summaries based on gold-standard AMR parses achieve a ROUGE-1 F1 score of 65.8%, suggesting room for improvement through better AMR parsing.
• Graph expansion marginally improves oracle performance (to 71.2% F1 with ramp loss) but slightly degrades system performance, indicating a trade-off in model complexity.
• The use of ramp loss consistently outperforms hinge and perceptron loss, especially in edge prediction, showing its effectiveness for structured prediction in this task.
• AMR parsing errors are a major source of performance degradation, particularly in edge prediction, though they have a smaller impact on concept prediction.
• Despite parsing errors, the system still achieves a ROUGE-1 F1 of 51.2% on concept prediction and 19.0% on edge prediction with graph expansion, showing the model’s robustness to noise.