[Paper Review] BloombergGPT: A Large Language Model for Finance

Shijie Wu, Ozan İrsoy+7 more|arXiv (Cornell University)|Mar 30, 2023

Topic Modeling299 citations

TL;DR

BloombergGPT is a 50B-parameter decoder-only LLM trained on a large, curated mix of financial and public data (FinPile + public corpora) that significantly improves financial task performance while remaining competitive on general NLP benchmarks.

ABSTRACT

The use of NLP in the realm of financial technology is broad and complex, with applications ranging from sentiment analysis and named entity recognition to question answering. Large Language Models (LLMs) have been shown to be effective on a variety of tasks; however, no LLM specialized for the financial domain has been reported in literature. In this work, we present BloombergGPT, a 50 billion parameter language model that is trained on a wide range of financial data. We construct a 363 billion token dataset based on Bloomberg's extensive data sources, perhaps the largest domain-specific dataset yet, augmented with 345 billion tokens from general purpose datasets. We validate BloombergGPT on standard LLM benchmarks, open financial benchmarks, and a suite of internal benchmarks that most accurately reflect our intended usage. Our mixed dataset training leads to a model that outperforms existing models on financial tasks by significant margins without sacrificing performance on general LLM benchmarks. Additionally, we explain our modeling choices, training process, and evaluation methodology. We release Training Chronicles (Appendix C) detailing our experience in training BloombergGPT.

Motivation & Objective

• Develop a large language model specialized for the financial domain.
• Construct a large, curated FinPile dataset (financial data) complemented by public data to enable mixed-domain training.
• Evaluate BloombergGPT on standard and internal finance benchmarks as well as general LLM benchmarks.
• Explain data collection, tokenizer design, model architecture, training protocol, and evaluation methods to aid replication.
• Share training insights and challenges to inform future domain-specific LLM initiatives.

Proposed method

• Build a 50B parameter decoder-only model inspired by BLOOM architecture.
• Use a mixed training corpus: 363B tokens from FinPile (financial) and 345B tokens from public datasets, totaling over 700B tokens.
• Employ a Unigram tokenizer with 131,072 vocabulary tokens and ALiBi positional encoding.
• Train with a left-to-right causal objective using 2,048-token sequences and 64x8 A100 GPUs, with ZeRO stage 3 model parallelism.
• Apply mixed-precision training with BF16 and FP32 where appropriate, activation checkpointing, and fused kernels for efficiency.
• Evaluate on public financial NLP benchmarks, internal Bloomberg tasks, and general NLP benchmarks to assess domain-specific and general capabilities.

Experimental results

Research questions

• RQ1 How does BloombergGPT perform on financial NLP benchmarks compared to general-purpose LLMs?
• RQ2 Does mixed-domain training (financial plus public data) improve financial task performance without degrading general NLP capabilities?
• RQ3 What are the effects of dataset construction (FinPile) and tokenizer choice on model performance and efficiency?
• RQ4 What training configurations and optimization strategies reliably stabilize and scale training for a 50B parameter finance-focused LLM?
• RQ5 How do Bloomberg-specific benchmarks (internal tasks) reflect real-world usage versus public benchmarks?

Key findings

• BloombergGPT substantially outperforms existing models on in-domain financial tasks.
• The model maintains competitive or superior performance on general NLP benchmarks despite its financial focus.
• Training uses a 50B parameter decoder with 70 layers and 40 attention heads, trained on ~569B tokens.
• The tokenizer is a large Unigram vocabulary of 131,072 tokens, enabling dense information encoding.
• ALiBi positional encoding and BLOOM-style decoder architecture support efficient long-sequence inference.
• Evaluation includes external financial tasks, internal sentiment and NER probes, and BIG-bench Hard assessments.