[Paper Review] Investigating Large Language Models and Control Mechanisms to Improve Text Readability of Biomedical Abstracts

Zihao Li, Samuel Belkadi+4 more|arXiv (Cornell University)|Sep 22, 2023

Text Readability and SimplificationComputer Science3 citations

TL;DR

This study investigates large language models (LLMs) and control mechanisms for simplifying biomedical abstracts to improve public health literacy. Using the PLABA dataset, it applies domain fine-tuning and prompt-based learning to models like BART-Large with control tokens, achieving a SARI score of 46.54 and outperforming others in simplification quality while balancing meaning preservation.

ABSTRACT

Biomedical literature often uses complex language and inaccessible professional terminologies. That is why simplification plays an important role in improving public health literacy. Applying Natural Language Processing (NLP) models to automate such tasks allows for quick and direct accessibility for lay readers. In this work, we investigate the ability of state-of-the-art large language models (LLMs) on the task of biomedical abstract simplification, using the publicly available dataset for plain language adaptation of biomedical abstracts ( extbf{PLABA}). The methods applied include domain fine-tuning and prompt-based learning (PBL) on: 1) Encoder-decoder models (T5, SciFive, and BART), 2) Decoder-only GPT models (GPT-3.5 and GPT-4) from OpenAI and BioGPT, and 3) Control-token mechanisms on BART-based models. We used a range of automatic evaluation metrics, including BLEU, ROUGE, SARI, and BERTscore, and also conducted human evaluations. BART-Large with Control Token (BART-L-w-CT) mechanisms reported the highest SARI score of 46.54 and T5-base reported the highest BERTscore 72.62. In human evaluation, BART-L-w-CTs achieved a better simplicity score over T5-Base (2.9 vs. 2.2), while T5-Base achieved a better meaning preservation score over BART-L-w-CTs (3.1 vs. 2.6). We also categorised the system outputs with examples, hoping this will shed some light for future research on this task. Our code, fine-tuned models, and data splits are available at \url{https://github.com/HECTA-UoM/PLABA-MU} \begin{IEEEkeywords} Large Language Models, Text Simplification, Biomedical NLP, Control Mechanisms, Health Informatics \end{IEEEkeywords}

Motivation & Objective

• To improve public health literacy by simplifying complex biomedical abstracts using NLP techniques.
• To evaluate the effectiveness of state-of-the-art LLMs in biomedical text simplification using automatic and human evaluation metrics.
• To investigate the impact of control mechanisms and prompt-based learning on simplification quality and meaning preservation.
• To provide a benchmark for future research using the PLABA dataset with fine-tuned models and code.
• To explore the trade-offs between simplification and semantic fidelity in LLM-generated plain language abstracts.

Proposed method

• Fine-tuned encoder-decoder models (T5, SciFive, BART) and decoder-only models (GPT-3.5, GPT-4, BioGPT) on the PLABA dataset.
• Applied prompt-based learning (PBL) to generate simplified outputs from GPT-based models with zero-shot and one-shot prompting.
• Implemented control-token mechanisms on BART-based models to guide simplification while preserving semantic content.
• Used LoRA for efficient fine-tuning of BART models with control tokens to reduce computational cost.
• Evaluated outputs using automatic metrics: BLEU, ROUGE, SARI, and BERTScore on a 2-reference test set.
• Conducted human evaluation with simplicity and meaning preservation scores on 814 sentence pairs to assess qualitative performance.

Experimental results

Research questions

• RQ1 How do different LLM architectures perform in simplifying biomedical abstracts from the PLABA dataset?
• RQ2 To what extent do control-token mechanisms improve simplification quality and meaning preservation in BART-based models?
• RQ3 How do automatic evaluation metrics correlate with human judgments in biomedical text simplification?
• RQ4 What trade-offs exist between simplification depth and semantic fidelity in LLM-generated plain language abstracts?
• RQ5 Can one-shot prompting with GPT models achieve competitive performance compared to fine-tuned models?

Key findings

• BART-Large with control tokens (BART-L-w-CT) achieved the highest SARI score of 46.54, indicating superior generation quality in simplification.
• T5-Base achieved the highest BERTScore of 72.62, indicating stronger meaning preservation compared to other models.
• In human evaluation, BART-L-w-CT scored 2.9 on simplicity (vs. 2.2 for T5-Base), showing better simplification, while T5-Base scored 3.1 on meaning preservation (vs. 2.6 for BART-L-w-CT).
• The automatic metrics showed inconsistent rankings: BART excelled in SARI but lagged in BERTScore, highlighting a trade-off between simplification and fidelity.
• GPT-3.5 and GPT-4 with one-shot prompting achieved competitive performance, suggesting strong zero-shot generalization in this task.
• The study provides fine-tuned models, code, and data splits at https://github.com/HECTA-UoM/PLABA-MU for reproducibility and future benchmarking.