[Paper Review] PLLaMa: An Open-source Large Language Model for Plant Science

Xianjun Yang, Junfeng Gao+2 more|arXiv (Cornell University)|Jan 3, 2024

Biomedical Text Mining and Ontologies10 citations

TL;DR

PLLaMa extends LLaMa-2 with a plant science–focused training corpus of over 1.5 million scholarly articles, followed by instruction tuning, to improve plant science QA and dialogue capabilities; checkpoints are released for community use.

ABSTRACT

Large Language Models (LLMs) have exhibited remarkable capabilities in understanding and interacting with natural language across various sectors. However, their effectiveness is limited in specialized areas requiring high accuracy, such as plant science, due to a lack of specific expertise in these fields. This paper introduces PLLaMa, an open-source language model that evolved from LLaMa-2. It's enhanced with a comprehensive database, comprising more than 1.5 million scholarly articles in plant science. This development significantly enriches PLLaMa with extensive knowledge and proficiency in plant and agricultural sciences. Our initial tests, involving specific datasets related to plants and agriculture, show that PLLaMa substantially improves its understanding of plant science-related topics. Moreover, we have formed an international panel of professionals, including plant scientists, agricultural engineers, and plant breeders. This team plays a crucial role in verifying the accuracy of PLLaMa's responses to various academic inquiries, ensuring its effective and reliable application in the field. To support further research and development, we have made the model's checkpoints and source codes accessible to the scientific community. These resources are available for download at \url{https://github.com/Xianjun-Yang/PLLaMa}.

Motivation & Objective

• Motivation to improve LLM accuracy in domain-specific plant science beyond general-domain models.
• Develop an open-source plant-science–oriented LLM by extended pretraining on plant literature.
• Enhance conversational ability through instruction tuning to support academic inquiries in plant science.
• Provide public access to training checkpoints and source code for reproducibility and further research.

Proposed method

• Extend pretraining of LLaMa-2-7B and LLaMa-2-13B using 1.5 million+ plant science articles.
• Construct a plant science corpus by filtering S2ORC to journal names (750 plant science journals).
• Mix plant-science corpus with 10% general RedPajama-Data-1T-Sample to mitigate catastrophic forgetting.
• Apply bf16, FlashAttention, zero-stage-3 DeepSpeed, and Fully Sharded Data Parallel (FSDP) for efficient training.
• Perform instruction tuning using 1030 instructions from the LIMA set plus plant-science–specific prompts; train with bf16 and FSDP.
• Evaluate via held-out plant science quiz and zero-shot cases; report accuracy and qualitative assessments.

Experimental results

Research questions

• RQ1 Can an open-source LLM be significantly improved for plant science tasks via domain-specific pretraining?
• RQ2 Does instruction tuning further enhance plant-science dialogue and question-answer capabilities beyond extended pretraining?
• RQ3 What is the measured performance of PLLaMa on a plant science quiz and zero-shot inquiries?
• RQ4 Are public checkpoints and code sufficient to enable reproducibility and further domain-specific work?

Key findings

• PLLaMa-13B-Chat achieves around 60% accuracy on a 10-question held-out plant science quiz.
• Pretraining and instruction tuning were conducted on eight A100 GPUs and four A100 GPUs respectively, with detailed resource usage and timelines reported (e.g., 7B ~26 hours; 13B ~57 hours for pretraining; 7B ~1.3 hours; 13B ~2.7 hours for instruction tuning).
• The model outputs domain-relevant Q&A and exhibits usefulness according to an international panel of plant scientists and engineers.
• Model checkpoints and source code are released to the community for download and replication.
• PLLaMa builds on LLaMa-2 by adding a large plant-science corpus and domain-focused instruction tuning to close the gap with domain specialists.