[Paper Review] Exploring Domain Shift in Extractive Text Summarization

Danqing Wang, Pengfei Liu+4 more|arXiv (Cornell University)|Aug 30, 2019

Topic Modeling41 references38 citations

TL;DR

This paper defines domain as article publications, builds a multi-domain SUM dataset (MULTI-SUM) to study domain shift in extractive summarization, and analyzes four learning strategies (including meta-learning) for cross-domain generalization.

ABSTRACT

Although domain shift has been well explored in many NLP applications, it still has received little attention in the domain of extractive text summarization. As a result, the model is under-utilizing the nature of the training data due to ignoring the difference in the distribution of training sets and shows poor generalization on the unseen domain. With the above limitation in mind, in this paper, we first extend the conventional definition of the domain from categories into data sources for the text summarization task. Then we re-purpose a multi-domain summarization dataset and verify how the gap between different domains influences the performance of neural summarization models. Furthermore, we investigate four learning strategies and examine their abilities to deal with the domain shift problem. Experimental results on three different settings show their different characteristics in our new testbed. Our source code including extit{BERT-based}, extit{meta-learning} methods for multi-domain summarization learning and the re-purposed dataset extsc{Multi-SUM} will be available on our project: \url{http://pfliu.com/TransferSum/}.

Motivation & Objective

• Extend domain concept from categories to data sources (publication sources) for summarization to study distribution gaps across domains.
• Re-purpose the MULTI-SUM dataset to create a multi-domain testbed with in-domain and out-of-domain settings.
• Evaluate how different learning strategies handle domain shift in extractive summarization and provide practical guidance for multi-domain learning.

Proposed method

• Model extractive summarization as sentence-labeling using a CNN sentence encoder and Transformer document encoder (CNN-Transformer).
• Define four learning strategies for multi-domain summarization: (I) Basic multi-domain training, (II) BERT-enhanced pre-training for multi-domain learning, (III) Domain tag embedding to make models domain-aware, (IV) Meta-learning to align gradient updates across domains.
• Formalize domain-shift mitigation with equations: L^(k)_I = L(Basic(S^(k), θ^(s)), Y^(k)); L^(k)_III = L(Basic(S^(k), C^(k), θ^(s)), Y^(k)); L^(k)_IV = γ L^(k) + (1-γ) ∑_{j≠k} L^{k←j}, with γ ∈ [0,1].
• Create MULTI-SUM by selecting top ten publications from Newsroom and splitting into training/testing domains; evaluate in-domain, out-of-domain, and cross-dataset transfer (CNN/DM).
• Compare against baselines and prior models using ROUGE metrics to assess domain transfer performance.

Experimental results

Research questions

• RQ1 How does shifting to unseen publications (domains) affect extractive summarization performance?
• RQ2 Can domain-aware or meta-learning approaches improve generalization across publications and datasets compared to a monolithic model?
• RQ3 What is the influence of pre-trained models (e.g., BERT) on multi-domain summarization and cross-domain transfer?
• RQ4 How do domain shifts manifest in in-domain vs out-of-domain vs cross-dataset settings for extractive summarization?

Key findings

• Domain shift in extractive summarization is significant: models trained on one publication underperform on unseen publications.
• Domain-aware modeling (using domain tags) improves in-domain and out-of-domain ROUGE performance compared to a vanilla multi-domain model.
• Meta-learning (Model IV) yields the best cross-domain generalization, with smaller gains in in-domain performance but stronger gains in unseen domains.
• Pre-trained BERT provides strong feature extraction and helps within MULTI-SUM but may underperform domain-aware or meta-learning strategies in cross-domain transfer.
• On CNN/DailyMail, publication tags plus BERT achieve the best performance among the tested configurations, indicating dataset-specific domain signals matter.