[Paper Review] Subjective Bias in Abstractive Summarization

Lei Li, Wei Liu+5 more|arXiv (Cornell University)|Jun 18, 2021

Topic Modeling22 references4 citations

TL;DR

This paper introduces a method to identify and cluster subjective writing styles in abstractive summarization datasets using graph-based style embeddings via Graph Convolutional Networks (GCN). Experiments on CNN-DM show that training on single-style clusters improves model convergence, abstraction, and generalization—particularly for certain styles—while mixing all styles harms performance, challenging the assumption that diverse gold summaries always benefit training.

ABSTRACT

Due to the subjectivity of the summarization, it is a good practice to have more than one gold summary for each training document. However, many modern large-scale abstractive summarization datasets have only one-to-one samples written by different human with different styles. The impact of this phenomenon is understudied. We formulate the differences among possible multiple expressions summarizing the same content as subjective bias and examine the role of this bias in the context of abstractive summarization. In this paper a lightweight and effective method to extract the feature embeddings of subjective styles is proposed. Results of summarization models trained on style-clustered datasets show that there are certain types of styles that lead to better convergence, abstraction and generalization. The reproducible code and generated summaries are available online.

Motivation & Objective

• To investigate the impact of subjective writing style variation in gold summaries on abstractive summarization model performance.
• To propose a method for extracting and clustering subjective writing styles from human-annotated summaries.
• To evaluate whether training on style-clustered datasets yields better convergence, abstraction, and generalization than training on mixed-style datasets.
• To challenge the common assumption that diverse gold summaries always improve model performance.

Proposed method

• Represent syntactic structures in summaries using a graph structure to capture syntactic transformations from source articles (oracles) to summaries.
• Use a self-supervised Graph Convolutional Network (GCN) to learn subjective style embeddings from these syntactic graphs.
• Cluster the dataset based on the learned style embeddings to form style-specific training sets.
• Fine-tune T5-small on each style-clustered dataset and compare performance against baseline models trained on the full, mixed-style dataset.
• Classify test samples into clusters based on nearest centroid to evaluate style-specific model generalization.
• Report results using ROUGE, BERTScore, and other metrics to assess abstraction, fluency, and factual consistency.

Experimental results

Research questions

• RQ1 How does subjective writing style variation across gold summaries affect the performance of abstractive summarization models?
• RQ2 Can subjective styles be effectively extracted and embedded using a graph-based, self-supervised GCN approach?
• RQ3 Does training on a single, homogeneous style lead to better model convergence and generalization than training on a mixed-style dataset?
• RQ4 Are there specific subjective styles that consistently outperform others in terms of ROUGE scores and abstraction quality?
• RQ5 Does combining all styles in a dataset truly lead to optimal model performance, or does it introduce noise that degrades results?

Key findings

• Training on a single, homogeneous style (e.g., Cluster_1) leads to better convergence and higher ROUGE scores than training on the full, mixed-style dataset.
• The model trained on Cluster_1 achieved the highest ROUGE-1 (40.2) and ROUGE-2 (17.3) scores, outperforming both baseline and full-dataset finetuning.
• The full-dataset finetuned model showed the slowest convergence and worst performance, with ROUGE-2 F1 of only 16.1, indicating that mixed styles can hinder learning.
• Models without finetuning outperformed the finetuned full-dataset model on summary-related metrics, suggesting that fine-tuning may degrade abstraction due to copy-paste tendencies.
• The ensemble of best summaries per sample (Cluster_Best) achieved ROUGE-2 F1 of 20.0, nearly matching a 4-model ensemble on the full dataset, implying that mixing all styles is suboptimal.
• Style 1 was the most favorable for model learning, as models trained on it achieved the best performance on test sets of the same style, indicating a preference for specific summarization patterns.