[Paper Review] SuMe: A Dataset Towards Summarizing Biomedical Mechanisms

Mohaddeseh Bastan, N. Udaya Shankar+2 more|arXiv (Cornell University)|May 10, 2022

Biomedical Text Mining and Ontologies4 citations

TL;DR

This paper introduces SuMe, a large-scale dataset of 22,000 biomedical mechanism summarization instances derived from scientific abstracts, using a semi-automated bootstrapping method with a small set of expert-labeled mechanism sentences and a fine-tuned classifier. Despite pretraining on 611,000 conclusion sentences, the best model generates acceptable mechanism summaries in only 32% of cases, highlighting significant challenges in biomedical language understanding and explanation generation.

ABSTRACT

Can language models read biomedical texts and explain the biomedical mechanisms discussed? In this work we introduce a biomedical mechanism summarization task. Biomedical studies often investigate the mechanisms behind how one entity (e.g., a protein or a chemical) affects another in a biological context. The abstracts of these publications often include a focused set of sentences that present relevant supporting statements regarding such relationships, associated experimental evidence, and a concluding sentence that summarizes the mechanism underlying the relationship. We leverage this structure and create a summarization task, where the input is a collection of sentences and the main entities in an abstract, and the output includes the relationship and a sentence that summarizes the mechanism. Using a small amount of manually labeled mechanism sentences, we train a mechanism sentence classifier to filter a large biomedical abstract collection and create a summarization dataset with 22k instances. We also introduce conclusion sentence generation as a pretraining task with 611k instances. We benchmark the performance of large bio-domain language models. We find that while the pretraining task help improves performance, the best model produces acceptable mechanism outputs in only 32% of the instances, which shows the task presents significant challenges in biomedical language understanding and summarization.

Motivation & Objective

• To address the growing challenge of information overload in biomedical literature by enabling automated summarization of complex biological mechanisms.
• To develop a scalable method for creating high-quality, large-scale datasets for mechanism summarization without relying solely on manual curation.
• To investigate whether large language models can accurately generate mechanistic explanations for relationships between biochemical entities in scientific texts.
• To benchmark the performance of state-of-the-art bio-language models on a novel, complex task combining relation extraction and explanation generation.
• To identify and analyze failure modes in model-generated mechanism summaries to guide future improvements in biomedical NLP.

Proposed method

• Leverage the structural pattern in biomedical abstracts—supporting sentences, a concluding mechanism sentence, and clearly defined entities (regulator and regulated)—to define a summarization task.
• Use a small set of expert-annotated mechanism sentences to train a mechanism sentence classifier based on a biomedical language model.
• Apply the trained classifier to a large collection (~611K abstracts) to automatically identify and extract mechanism summarization instances, yielding 22,000 training instances.
• Introduce a pretraining task focused on conclusion sentence generation using the same abstracts to improve downstream mechanism generation performance.
• Create a manually curated subset of 125 instances for evaluation, achieving 84% expert-annotated quality, which serves as a benchmark for model evaluation.
• Fine-tune state-of-the-art bio-domain language models on the SuMe dataset and evaluate performance using automatic metrics and human evaluation.

Experimental results

Research questions

• RQ1 Can large language models generate accurate and faithful summaries of the underlying biological mechanisms described in biomedical abstracts?
• RQ2 How effective is weak supervision via a small set of expert-labeled examples in bootstrapping a large-scale mechanism summarization dataset?
• RQ3 To what extent does pretraining on conclusion sentence generation improve the quality of mechanism summarization?
• RQ4 What are the dominant failure modes in model-generated mechanism summaries, and how do they relate to entity tracking and relation understanding?
• RQ5 How does the performance of current SOTA models compare to human-level understanding on this complex biomedical explanation generation task?

Key findings

• The best-performing model generates acceptable mechanism summaries in only 32% of test instances, indicating that the task remains highly challenging for current language models.
• The most frequent error type is missing one of the main entities in the generated mechanism sentence, occurring in 35% of failures, suggesting difficulties in coreference and entity tracking.
• Incorrect mechanism generation accounts for 24% of failures, where models produce plausible-sounding but factually unsupported or unrelated explanations.
• Flipped relations occur in 19% of failures, indicating that relation prediction errors significantly degrade mechanism generation quality.
• Non-mechanism outputs (11%) and incomplete mechanisms (11%) further demonstrate that models struggle to generate semantically complete and accurate explanations.
• The presence of causal and reasoning indicators—such as 'binding', 'caused', 'dose dependent', and 'demonstrated'—in supporting sentences correlates with higher model confidence and better generation quality.