[Paper Review] Abstractive Summarization of Reddit Posts with Multi-level Memory Networks

Byeongchang Kim, Hyunwoo Kim+1 more|arXiv (Cornell University)|Nov 2, 2018

Topic Modeling50 references60 citations

TL;DR

The paper introduces Reddit TIFU, a large-scale abstractive summarization dataset from Reddit, and a novel multi-level memory networks (MMN) model that uses multi-level memory with dilated convolutions and normalized gated tanh units to improve abstractive summary generation.

ABSTRACT

We address the problem of abstractive summarization in two directions: proposing a novel dataset and a new model. First, we collect Reddit TIFU dataset, consisting of 120K posts from the online discussion forum Reddit. We use such informal crowd-generated posts as text source, in contrast with existing datasets that mostly use formal documents as source such as news articles. Thus, our dataset could less suffer from some biases that key sentences usually locate at the beginning of the text and favorable summary candidates are already inside the text in similar forms. Second, we propose a novel abstractive summarization model named multi-level memory networks (MMN), equipped with multi-level memory to store the information of text from different levels of abstraction. With quantitative evaluation and user studies via Amazon Mechanical Turk, we show the Reddit TIFU dataset is highly abstractive and the MMN outperforms the state-of-the-art summarization models.

Motivation & Objective

• Introduce a large-scale abstractive summarization dataset from informal online text (Reddit TIFU).
• Develop a novel model (MMN) that stores information at multiple abstraction levels to improve abstractive summarization.
• Evaluate MMN against state-of-the-art abstractive models on Reddit TIFU, Newsroom-Abs, and XSum datasets.
• Assess the dataset's abstractive characteristics and the model's ability to generate highly abstractive summaries.

Proposed method

• Create Reddit TIFU dataset with 122,933 post–summary pairs from Reddit (TIFU-short and TIFU-long).
• Develop multi-level memory networks (MMN) with memory at word-, sentence-, paragraph-, and document-level using dilated convolutions for writing memory.
• Introduce normalized gated tanh units (NGTU) with layer and weight normalization in a convolutional memory framework.
• Use a WaveNet-like decoder with global conditioning on a whole-document embedding and attention to multi-level memory slots.
• Train with label-smoothed cross-entropy loss to regularize predictions.
• Evaluate with perplexity and ROUGE metrics, plus Amazon Mechanical Turk (AMT) user studies.

Experimental results

Research questions

• RQ1 Is Reddit TIFU a highly abstractive dataset compared to traditional news-based datasets?
• RQ2 Does the MMN model outperform state-of-the-art abstractive summarization methods on Reddit TIFU, Newsroom-Abs, and XSum?
• RQ3 Do multi-level memory representations and dilated convolutional writing improve abstractive summary quality over standard seq2seq models?
• RQ4 How does MMN perform in human evaluations relative to baselines on abstractive summarization tasks.

Key findings

• Reddit TIFU is highly abstractive, with extensiveness of abstraction indicated by low Ext-Oracle performance and distribution of summary content across the source text.
• MMN outperforms state-of-the-art abstractive models on Reddit TIFU in ROUGE and perplexity across both TIFU-short and TIFU-long datasets.
• MMN also yields competitive or superior results on Newsroom-Abs and XSum compared to prior abstractive methods.
• Ablation studies show contributions from dilated convolutions, multi-level memory, and normalized gated tanh units to MMN's performance.
• AMT user studies indicate that human evaluators prefer MMN-generated summaries over several baselines.