[Paper Review] VieSum: How Robust Are Transformer-based Models on Vietnamese Summarization?

Hieu Nguyen, Long Phan+3 more|arXiv (Cornell University)|Oct 8, 2021

Topic Modeling23 references4 citations

TL;DR

This paper evaluates the robustness of Transformer-based encoder-decoder models for Vietnamese abstractive text summarization using two datasets: Wikilingua and Vietnews. It finds that multilingual models like mT5 and mBART outperform monolingual models such as PhoBERT and ViBERT, especially on Wikipedia-based data, highlighting the importance of pre-training data quality and the potential for improvement through stronger monolingual models.

ABSTRACT

Text summarization is a challenging task within natural language processing that involves text generation from lengthy input sequences. While this task has been widely studied in English, there is very limited research on summarization for Vietnamese text. In this paper, we investigate the robustness of transformer-based encoder-decoder architectures for Vietnamese abstractive summarization. Leveraging transfer learning and self-supervised learning, we validate the performance of the methods on two Vietnamese datasets.

Motivation & Objective

• To investigate the robustness of pre-trained Transformer-based models on Vietnamese abstractive summarization tasks.
• To compare the performance of monolingual Vietnamese models (e.g., PhoBERT, ViBERT) with multilingual models (e.g., mT5, mBART) on Vietnamese summarization.
• To analyze the impact of pre-training data distribution and scale on model performance in low-resource settings like Vietnamese NLP.
• To evaluate whether transfer learning and self-supervised pre-training improve abstractive summarization in Vietnamese.
• To provide a benchmark for future research by finetuning models on two publicly available Vietnamese summarization datasets.

Proposed method

• Fine-tuned encoder-decoder Transformer architectures based on Vaswani et al. (2017), using pre-trained encoders and decoders.
• Applied the BERT2BERT architecture (Rothe et al., 2019) to adapt pre-trained models for sequence-to-sequence summarization.
• Used two datasets: Wikilingua (Wikipedia-based) and Vietnews (news articles from Vietnamese media), both with human-annotated summaries.
• Evaluated models using ROUGE-1, ROUGE-2, and ROUGE-L metrics to measure n-gram and longest common subsequence overlap with reference summaries.
• Trained and compared monolingual models (PhoBERT, ViBERT) and multilingual models (mBERT, mBART, mT5) on the same fine-tuning setup.
• Leveraged self-supervised pre-training objectives such as Masked Language Modeling (MLM) during initial pre-training of base models.

Experimental results

Research questions

• RQ1 How do monolingual Vietnamese pre-trained models compare to multilingual models in abstractive summarization on Vietnamese text?
• RQ2 Does the quality and domain of pre-training data affect model performance on constrained-domain summarization tasks in Vietnamese?
• RQ3 To what extent does incorporating a pre-trained decoder improve summarization performance compared to random initialization?
• RQ4 Can multilingual models like mT5 and mBART generalize effectively to Vietnamese summarization despite not being specifically fine-tuned on Vietnamese data?
• RQ5 What is the relative performance of PhoBERT and ViBERT as monolingual models compared to multilingual models on Vietnamese news and Wikipedia-style documents?

Key findings

• mT5 achieved the highest ROUGE scores on Wikilingua, with ROUGE-1 of 55.27, ROUGE-2 of 27.63, and ROUGE-L of 38.30, outperforming all other models.
• mBART achieved second-best performance on Wikilingua with ROUGE-1 of 55.21, ROUGE-2 of 25.69, and ROUGE-L of 37.33.
• On the Vietnews dataset, PhoBERT2PhoBERT achieved the best results (ROUGE-1: 60.37, ROUGE-2: 29.12, ROUGE-L: 39.44), outperforming multilingual models.
• The monolingual PhoBERT2PhoBERT model significantly outperformed its random-weighted baseline (RND2RND), showing a 2–4% improvement across all ROUGE metrics on Wikilingua.
• Multilingual models like mT5 and mBART showed only marginal gains over the RND2RND baseline on the Vietnews dataset, suggesting limited transferability to news-domain text.
• Pre-training on large-scale, diverse corpora like Common Crawl (used by mT5 and mBART) improved performance on Wikipedia-style, constrained-domain data, but not on domain-specific news text.