[Paper Review] WikiHow: A Large Scale Text Summarization Dataset

Mahnaz Koupaee, William Yang Wang|arXiv (Cornell University)|Oct 18, 2018

Topic Modeling14 references178 citations

TL;DR

WikiHow introduces a large, diverse text summarization dataset with over 200k article-summary pairs from WikiHow, enabling long-sequence abstractive tasks and new baselines.

ABSTRACT

Sequence-to-sequence models have recently gained the state of the art performance in summarization. However, not too many large-scale high-quality datasets are available and almost all the available ones are mainly news articles with specific writing style. Moreover, abstractive human-style systems involving description of the content at a deeper level require data with higher levels of abstraction. In this paper, we present WikiHow, a dataset of more than 230,000 article and summary pairs extracted and constructed from an online knowledge base written by different human authors. The articles span a wide range of topics and therefore represent high diversity styles. We evaluate the performance of the existing methods on WikiHow to present its challenges and set some baselines to further improve it.

Motivation & Objective

• Motivate the need for large-scale, diverse summarization data beyond news articles.
• Introduce WikiHow as a large, multi-topic dataset suitable for long-sequence abstractive summarization.
• Define metrics for abstraction level and compression to characterize the dataset.
• Provide baselines (extractive and abstractive) on WikiHow to establish benchmarking references.

Proposed method

• Crawl WikiHow using Python Scrapy to collect articles across 20+ categories.
• Construct article-summary pairs by taking bold step-summaries as references and concatenating remaining step descriptions as articles.
• Filter out unusable pairs where summaries are not shorter than the article content, yielding a final set of 204,004 pairs.
• Define abstraction and compression metrics by comparing reference summaries to articles (n-gram uniqueness) and average sentence lengths.
• Evaluate existing extractive and abstractive baselines (TextRank, seq-to-seq with attention, pointer-generator, pointer-generator with coverage, Lead-3) on WikiHow and CNN/Daily Mail.
• Use ROUGE and METEOR as evaluation metrics for the baselines.

Experimental results

Research questions

• RQ1 How large and diverse is the WikiHow dataset compared to existing summarization corpora?
• RQ2 What is the level of abstractive content in WikiHow relative to CNN/Daily Mail?
• RQ3 How do standard summarization models perform on WikiHow as a benchmark for long-sequence abstractive tasks?
• RQ4 What are the compression ratios and their implications for abstractive difficulty?
• RQ5 Do existing baselines generalize from news-focused datasets to WikiHow’s knowledge-base articles?

Key findings

• WikiHow contains 204,004 article-summary pairs with average article length ~579.8 words and average summary length ~62.1 words.
• WikiHow exhibits a higher compression ratio (2.38) than CNN/Daily Mail, indicating greater abstractive challenge.
• There is a higher level of abstractedness in WikiHow, with fewer overlapping n-grams between articles and summaries except for common unigrams, bigrams, and trigrams.
• Baselines (TextRank, seq-to-seq with attention, pointer-generator, pointer-generator with coverage, Lead-3) perform worse on WikiHow than on CNN/Daily Mail, highlighting WikiHow’s challenging, diverse writing style.
• Lead-3 underperforms on WikiHow due to non-news writing style, contrasting with its strong performance on news datasets.
• The paper provides baseline results and emphasizes the need for improved models to handle long, diverse, and abstractive WikiHow content.