[Paper Review] Automatic Text Summarization of COVID-19 Medical Research Articles using BERT and GPT-2

Virapat Kieuvongngam, Bowen Tan+1 more|arXiv (Cornell University)|Jun 3, 2020

Topic Modeling8 references82 citations

TL;DR

The paper fine-tunes DistilGPT-2 for abstractive summarization of COVID-19 papers by mapping keyword sets to summaries, using a multi-task loss and ROUGE evaluation. It compares abstractive summaries to extractive baselines on CORD-19 data.

ABSTRACT

With the COVID-19 pandemic, there is a growing urgency for medical community to keep up with the accelerating growth in the new coronavirus-related literature. As a result, the COVID-19 Open Research Dataset Challenge has released a corpus of scholarly articles and is calling for machine learning approaches to help bridging the gap between the researchers and the rapidly growing publications. Here, we take advantage of the recent advances in pre-trained NLP models, BERT and OpenAI GPT-2, to solve this challenge by performing text summarization on this dataset. We evaluate the results using ROUGE scores and visual inspection. Our model provides abstractive and comprehensive information based on keywords extracted from the original articles. Our work can help the the medical community, by providing succinct summaries of articles for which the abstract are not already available.

Motivation & Objective

• Address the rapid growth of COVID-19 literature and provide concise summaries for articles lacking abstracts.
• Leverage pre-trained NLP models (BERT for extraction and GPT-2 for generation) to produce abstractive summaries.
• Develop a keyword-to-summary fine-tuning approach with a multi-task loss.
• Evaluate abstractive summaries against gold abstracts using ROUGE and qualitative inspection.

Proposed method

• Use DistilBERT for unsupervised extractive sentence embeddings and K-medoid clustering as a baseline for extractive summaries.
• Fine-tune DistilGPT-2 to generate abstractive summaries from keyword inputs derived from nouns and verbs.
• Train with a multi-task loss combining language modeling loss and a multiple-choice loss, using a special <|summarize|> token to separate keywords from the summary.
• Create training samples by pairing keywords with gold summaries and distractor summaries for a 4-option multiple-choice setup.
• Integrate top-p sampling with temperature scaling for generation and evaluate using ROUGE scores.

Experimental results

Research questions

• RQ1 Can a keyword-to-summary fine-tuning of a pre-trained GPT-2 model produce coherent abstractive summaries for COVID-19 literature?
• RQ2 What is the impact of keyword type (nouns, verbs, or both) on abstractive summary quality?
• RQ3 How do abstractive summaries generated by DistilGPT-2 compare to extractive baselines in ROUGE scores?
• RQ4 Does longer fine-tuning improve alignment between input keywords and generated summaries?

Key findings

• Extractive summaries with 60% compression yield higher ROUGE scores than 40% compression and outperform abstractive results in ROUGE.
• Abstractive GPT-2 summaries improve with more epochs and more keywords, but ROUGE gains may be limited compared to extractive baselines.
• Using nouns (with or without verbs) as keywords generally improves ROUGE scores over using only verbs.
• Top-p sampling outperforms greedy decoding in readability and abstractive quality, with similar ROUGE scores.
• Attention visualizations show fine-tuning strengthens keyword–summary relations in later layers.