[Paper Review] LitSumm: Large language models for literature summarisation of non-coding RNAs

Andrew R. Green, Carlos Eduardo Ribas+5 more|arXiv (Cornell University)|Nov 6, 2023

RNA modifications and cancer4 citations

TL;DR

This paper introduces LitSumm, a system that uses fine-tuned large language models (LLMs) with structured prompting and automated validation to generate high-quality, factually accurate summaries of scientific literature on non-coding RNAs. The method achieves high manual evaluation scores and was applied to over 4,600 ncRNAs, making summaries publicly available via RNAcentral.

ABSTRACT

Curation of literature in life sciences is a growing challenge. The continued increase in the rate of publication, coupled with the relatively fixed number of curators worldwide presents a major challenge to developers of biomedical knowledgebases. Very few knowledgebases have resources to scale to the whole relevant literature and all have to prioritise their efforts. In this work, we take a first step to alleviating the lack of curator time in RNA science by generating summaries of literature for non-coding RNAs using large language models (LLMs). We demonstrate that high-quality, factually accurate summaries with accurate references can be automatically generated from the literature using a commercial LLM and a chain of prompts and checks. Manual assessment was carried out for a subset of summaries, with the majority being rated extremely high quality. We apply our tool to a selection of over 4,600 ncRNAs and make the generated summaries available via the RNAcentral resource. We conclude that automated literature summarization is feasible with the current generation of LLMs, provided careful prompting and automated checking are applied.

Motivation & Objective

• To address the growing challenge of curating rapidly expanding literature in non-coding RNA (ncRNA) research.
• To reduce reliance on manual curation by automating literature summarization using large language models.
• To develop a scalable, reliable method for generating factually accurate summaries with correct references.
• To evaluate the quality of LLM-generated summaries through manual assessment.
• To deploy the system at scale, producing summaries for over 4,600 ncRNAs and integrating them into the RNAcentral knowledgebase.

Proposed method

• Employing a commercial large language model (LLM) with a chain-of-thought prompting strategy to generate structured summaries.
• Designing a sequence of prompts to guide the LLM in extracting key facts, including functional roles, expression patterns, and molecular interactions.
• Implementing automated checks to validate factual consistency and reference accuracy in generated summaries.
• Using reference grounding to ensure citations in summaries correspond to actual literature entries.
• Applying iterative refinement and filtering to improve summary quality before final deployment.
• Integrating the final summaries into the RNAcentral database for public access.

Experimental results

Research questions

• RQ1 Can large language models generate factually accurate summaries of ncRNA literature with minimal human intervention?
• RQ2 How does the quality of LLM-generated summaries compare to human-curated standards when assessed manually?
• RQ3 Can structured prompting and automated validation significantly improve factual consistency in LLM-generated summaries?
• RQ4 What is the scalability of this approach across a large set of non-coding RNAs?
• RQ5 Can such summaries be reliably integrated into existing biological knowledgebases like RNAcentral?

Key findings

• The majority of manually assessed summaries were rated as extremely high quality, indicating strong alignment with expert curation standards.
• The system successfully generated summaries for over 4,600 non-coding RNAs, demonstrating scalability.
• Automated checks significantly improved factual accuracy and reference correctness in the summaries.
• The chain-of-prompting approach enabled consistent extraction of key biological facts, including functional roles and regulatory mechanisms.
• The final summaries were integrated into RNAcentral, making them publicly accessible for researchers.
• The results confirm that current LLMs, when combined with careful prompting and validation, can produce reliable literature summaries in genomics.