[Paper Review] Features in Extractive Supervised Single-document Summarization: Case of Persian News

Hosein Rezaei, Seyed Amid Moeinzadeh+2 more|arXiv (Cornell University)|Sep 6, 2019

Advanced Text Analysis Techniques38 references4 citations

TL;DR

This paper proposes a document-aware feature engineering approach for supervised extractive single-document summarization in Persian, enhancing sentence ranking by integrating document-level features (e.g., length, topic) into sentence vectors. Experiments show that this method improves model precision (R² increased from 0.122 to 0.176) and summary quality (ROUGE-F1 improved over random baseline), demonstrating that context-aware features significantly boost performance in extractive summarization.

ABSTRACT

Text summarization has been one of the most challenging areas of research in NLP. Much effort has been made to overcome this challenge by using either the abstractive or extractive methods. Extractive methods are more popular, due to their simplicity compared with the more elaborate abstractive methods. In extractive approaches, the system will not generate sentences. Instead, it learns how to score sentences within the text by using some textual features and subsequently selecting those with the highest-rank. Therefore, the core objective is ranking and it highly depends on the document. This dependency has been unnoticed by many state-of-the-art solutions. In this work, the features of the document are integrated into vectors of every sentence. In this way, the system becomes informed about the context, increases the precision of the learned model and consequently produces comprehensive and brief summaries.

Motivation & Objective

• To address the limitation in supervised extractive summarization where sentence ranking is context-dependent on the document, yet most models treat sentences as independent samples.
• To improve model generalization and precision by incorporating document-level features such as length, topical category, and structural properties into sentence feature vectors.
• To demonstrate that document-aware features lead to better performance in both regression model accuracy and summary quality.
• To establish a new baseline using random regression for more accurate evaluation of extractive summarizers.

Proposed method

• The method integrates document-level features—such as document length, topic category, and sentence position—into the feature vector of each sentence, creating document-aware sentence representations.
• A supervised regression framework is used, with three regression models tested: Linear Regression, Decision Tree Regression, and Epsilon-Support Vector Regression (SVR), with SVR yielding the best results.
• The final model uses SVR with RBF kernel, epsilon=0.01, and default parameters, trained on merged sentence vectors from the entire dataset.
• Sentence ranking is performed using the trained regressor on the test set, and summaries are generated by selecting the top-n ranked sentences.
• Evaluation is conducted using ROUGE metrics (precision, recall, F1) and model performance is compared against a random regressor baseline.
• The system is implemented with a web interface and Telegram bot for public access and reproducibility.

Experimental results

Research questions

• RQ1 Does incorporating document-level features into sentence representations improve the performance of supervised extractive summarization models?
• RQ2 How does the inclusion of document-aware features affect the regression model's ability to predict sentence importance accurately?
• RQ3 To what extent does the proposed method outperform a random baseline in terms of ROUGE scores and model R²?
• RQ4 Can document-aware features reduce the model's reliance on global dataset statistics and improve local document-level summarization quality?
• RQ5 How do document-level features influence the decision-making process in tree-based models, especially in terms of information gain and entropy?

Key findings

• The inclusion of document-aware features reduced the mean squared error (MSE) from 0.03448 (experiment 1) to 0.03068 (experiment 2), indicating improved regression model accuracy.
• The R² score increased from 0.12238 to 0.17576 when document-aware features were used, showing a significant improvement in model fit and predictive power.
• ROUGE-F1 scores for the proposed method were substantially higher than the random baseline, with F1 values exceeding 50% in the random case but showing marked improvement with the proposed model.
• The random regressor baseline achieved a ROUGE-F1 above 50%, but the proposed method still outperformed it, indicating that the model learns meaningful patterns beyond random chance.
• The results suggest that document-aware features help the model make more context-sensitive decisions, especially in tree-based models, by reducing over-reliance on global dataset statistics.
• The study confirms that sentence ranking is inherently document-relative, and ignoring document boundaries during training leads to suboptimal performance.