[Paper Review] Galactica: A Large Language Model for Science

Ross Taylor, Marcin Kardas+7 more|arXiv (Cornell University)|Nov 16, 2022

Mathematics, Computing, and Information Processing257 citations

TL;DR

Galactica is a large, curated scientific language model designed to store, combine, and reason about scientific knowledge, achieving state-of-the-art results on several science-focused tasks and enabling multi-modal interactions with scientific data.

ABSTRACT

Information overload is a major obstacle to scientific progress. The explosive growth in scientific literature and data has made it ever harder to discover useful insights in a large mass of information. Today scientific knowledge is accessed through search engines, but they are unable to organize scientific knowledge alone. In this paper we introduce Galactica: a large language model that can store, combine and reason about scientific knowledge. We train on a large scientific corpus of papers, reference material, knowledge bases and many other sources. We outperform existing models on a range of scientific tasks. On technical knowledge probes such as LaTeX equations, Galactica outperforms the latest GPT-3 by 68.2% versus 49.0%. Galactica also performs well on reasoning, outperforming Chinchilla on mathematical MMLU by 41.3% to 35.7%, and PaLM 540B on MATH with a score of 20.4% versus 8.8%. It also sets a new state-of-the-art on downstream tasks such as PubMedQA and MedMCQA dev of 77.6% and 52.9%. And despite not being trained on a general corpus, Galactica outperforms BLOOM and OPT-175B on BIG-bench. We believe these results demonstrate the potential for language models as a new interface for science. We open source the model for the benefit of the scientific community.

Motivation & Objective

• Motivate and address information overload in science by creating an LLM that can store, combine, and reason about scientific knowledge.
• Develop a curated, normative scientific corpus and specialized interfaces to interact with diverse modalities (LaTeX, SMILES, amino acids, DNA).
• Demonstrate state-of-the-art performance on knowledge-intensive scientific tasks and reasoning benchmarks.
• Explore whether repeated tokens in a curated corpus can improve both upstream and downstream performance.

Proposed method

• Train decoder-only Transformers (GeLU activations, 2048 context window, no biases, learned positional embeddings) on a 106B-token curated scientific corpus.
• Use a 50k token BPE vocabulary and specialized tokens for citations, reasoning work memory, SMILES, amino acids, and DNA sequences.
• Pre-train with a mixture of prompts alongside general data to boost task-specific signals (e.g., chemical properties, QA, summarization, reasoning, etc.).
• Evaluate via knowledge probes (LaTeX equations, domain probes) and downstream tasks (PubMedQA, MedMCQA, MMLU, MATH) to benchmark knowledge, reasoning, and multi-modal capabilities.

Experimental results

Research questions

• RQ1 Can a curated, normative scientific corpus enable a decoder-only LLM to store and reason about scientific knowledge effectively?
• RQ2 How do repeated tokens in a curated dataset affect both upstream training loss and downstream scientific performance?
• RQ3 Do task-context prompts during pre-training improve performance on scientific and non-scientific tasks without harming generality?
• RQ4 To what extent can an LLM learn and utilize modalities like LaTeX, SMILES, amino acid sequences, and DNA within a single model?
• RQ5 How does Galactica perform on knowledge-intensive benchmarks and specialized downstream tasks compared to general LLMs?

Key findings

• Repeated tokens in a 120B model continue to improve validation loss and downstream performance up to four epochs, with the largest model showing initial overfitting only near epoch five.
• LaTeX equation knowledge improves with scale, with 120B achieving 68.2% vs 49.0% for GPT-3 on the LaTeX equations probe.
• Galactica achieves state-of-the-art on PubMedQA (77.6%) and MedMCQA dev (52.9%).
• On reasoning benchmarks, Galactica outperforms Chinchilla on MMLU (41.3% vs 35.7%) and PaLM 540B on MATH (20.4% vs 8.8%).
• Galactica outperforms BLOOM and OPT-175B on BIG-bench despite not being trained on a general corpus, and demonstrates multi-modal capabilities with SMILES and amino acid sequences.