[Paper Review] PharmaGPT: Domain-Specific Large Language Models for Bio-Pharmaceutical and Chemistry

Linqing Chen, Wei‐Lei Wang+34 more|arXiv (Cornell University)|Jun 26, 2024

Biomedical Text Mining and Ontologies4 citations

TL;DR

PharmaGPT introduces domain-specific large language models (13B and 70B parameters) fine-tuned on a vast biomedical and chemical corpus, achieving state-of-the-art performance on NAPLEX and biomedical translation benchmarks—surpassing general-purpose models like GPT-3.5 and Claude 3 despite using significantly fewer parameters, demonstrating the efficacy of specialized pre-training in bio-pharmaceutical NLP.

ABSTRACT

Large language models (LLMs) have revolutionized Natural Language Processing (NLP) by minimizing the need for complex feature engineering. However, the application of LLMs in specialized domains like biopharmaceuticals and chemistry remains largely unexplored. These fields are characterized by intricate terminologies, specialized knowledge, and a high demand for precision areas where general purpose LLMs often fall short. In this study, we introduce PharmaGPT, a suite of domain specilized LLMs with 13 billion and 70 billion parameters, specifically trained on a comprehensive corpus tailored to the Bio-Pharmaceutical and Chemical domains. Our evaluation shows that PharmaGPT surpasses existing general models on specific-domain benchmarks such as NAPLEX, demonstrating its exceptional capability in domain-specific tasks. Remarkably, this performance is achieved with a model that has only a fraction, sometimes just one-tenth-of the parameters of general-purpose large models. This advancement establishes a new benchmark for LLMs in the bio-pharmaceutical and chemical fields, addressing the existing gap in specialized language modeling. It also suggests a promising path for enhanced research and development, paving the way for more precise and effective NLP applications in these areas.

Motivation & Objective

• Address the lack of precision and domain-specific knowledge in general-purpose LLMs for bio-pharmaceutical and chemical domains.
• Develop a multilingual, domain-specialized LLM capable of handling complex terminology and nuanced scientific language.
• Demonstrate that smaller, specialized models can outperform larger, general-purpose models in domain-specific NLP tasks.
• Establish a scalable, efficient training framework for domain-specific LLMs using curated, high-quality scientific corpora.
• Provide a reproducible, open-architecture roadmap for future development of vertical-domain LLMs in science and medicine.

Proposed method

• Curated a comprehensive, multilingual corpus of billions of tokens from bio-pharmaceutical and chemical literature for pre-training.
• Fine-tuned 13B and 70B parameter LLMs on domain-specific tasks using a combination of masked language modeling and instruction tuning.
• Employed advanced distributed training techniques to optimize efficiency and scalability during pre-training.
• Applied a multi-granularity evaluation strategy across paragraph, sentence, and word-level translation tasks using BLEU scores.
• Benchmarked performance against leading general-purpose models (GPT-3.5, Claude 3, Google’s model) on U.S. NAPLEX and Chinese pharmaceutical exams.
• Analyzed scaling laws to correlate model size with performance gains in domain-specific benchmarks.

Experimental results

Research questions

• RQ1 Can a smaller, domain-specific LLM outperform larger, general-purpose LLMs in bio-pharmaceutical and chemical NLP tasks?
• RQ2 To what extent does pre-training on a curated biomedical and chemical corpus improve performance on domain-specific benchmarks like NAPLEX?
• RQ3 How does the performance of PharmaGPT compare to state-of-the-art general-purpose LLMs in multilingual biomedical text translation?
• RQ4 What is the impact of model size on performance in specialized scientific domains, and does scaling follow predictable trends?
• RQ5 Can a domain-specialized LLM achieve high-quality results with significantly fewer parameters than general-purpose models?

Key findings

• PharmaGPT achieved a BLEU score of 30 on paragraph-level biomedical translation, outperforming GPT-3.5 (27), Claude 3 (26), and Google’s model (27).
• On word-level translation, PharmaGPT scored 10, compared to 8 for GPT-3.5 and 9 for both Claude 3 and Google’s model.
• At the sentence level, PharmaGPT achieved a BLEU score of 18, surpassing GPT-3.5 (15) and Claude 3 (16).
• PharmaGPT’s performance on the U.S. NAPLEX and Chinese pharmaceutical exams improved with increasing model size, ultimately surpassing GPT.
• The model demonstrated consistent superiority across all translation granularities, indicating robustness in handling complex scientific language.
• Despite having only one-tenth the parameters of some general-purpose models, PharmaGPT achieved state-of-the-art results in domain-specific evaluations.