[Paper Review] Clustering of Deep Contextualized Representations for Summarization of Biomedical Texts

Milad Moradi, Matthias Samwald|arXiv (Cornell University)|Aug 6, 2019

Topic Modeling5 citations

TL;DR

This paper proposes a BERT-based summarizer that leverages deep contextualized representations to cluster and rank sentences for biomedical text summarization without requiring external domain knowledge. By measuring sentence similarity and informative content through BERT embeddings, the method achieves improved summarization performance over baseline approaches, demonstrating that contextualized representations alone can effectively capture biomedical context.

ABSTRACT

In recent years, summarizers that incorporate domain knowledge into the process of text summarization have outperformed generic methods, especially for summarization of biomedical texts. However, construction and maintenance of domain knowledge bases are resource-intense tasks requiring significant manual annotation. In this paper, we demonstrate that contextualized representations extracted from the pre-trained deep language model BERT, can be effectively used to measure the similarity between sentences and to quantify the informative content. The results show that our BERT-based summarizer can improve the performance of biomedical summarization. Although the summarizer does not use any sources of domain knowledge, it can capture the context of sentences more accurately than the comparison methods. The source code and data are available at https://github.com/BioTextSumm/BERT-based-Summ.

Motivation & Objective

• To develop a low-resource, domain-agnostic summarization method for biomedical texts that avoids costly manual knowledge base construction.
• To investigate whether pre-trained BERT representations can effectively capture sentence similarity and informativeness in biomedical contexts.
• To evaluate the performance of a BERT-based clustering approach against existing generic and domain-specific summarization methods.
• To demonstrate that contextualized embeddings alone can achieve competitive summarization results without explicit domain knowledge.

Proposed method

• Utilizes contextualized embeddings from the pre-trained BERT model to represent sentences in the biomedical domain.
• Employs clustering techniques on BERT embeddings to group semantically similar sentences and identify representative content.
• Ranks sentences based on their centrality within clusters and informativeness inferred from embedding representations.
• Applies a sentence selection strategy that prioritizes high-impact, diverse sentences from clustered representations.
• Trains and evaluates the model end-to-end on standard biomedical summarization benchmarks without fine-tuning on domain-specific data.
• Relies solely on the semantic richness of BERT embeddings to infer sentence relevance and redundancy.

Experimental results

Research questions

• RQ1 Can BERT-based contextualized representations effectively measure sentence similarity in biomedical texts without additional domain knowledge?
• RQ2 To what extent can clustering of BERT embeddings improve the quality of automatic summarization in biomedical documents?
• RQ3 How does a BERT-only summarizer compare to methods that rely on external domain knowledge bases in terms of summarization performance?
• RQ4 Can informative content be quantified through embedding-based clustering without explicit feature engineering?

Key findings

• The BERT-based summarizer outperforms generic baseline methods in automatic evaluation metrics for biomedical text summarization.
• The model achieves competitive performance despite not using any manually constructed domain knowledge or external resources.
• Clustering of contextualized representations effectively identifies salient and non-reduundant sentences, improving summary coherence and coverage.
• The method captures contextual nuances in biomedical language more accurately than comparison methods, as evidenced by improved ROUGE scores.