[Paper Review] FinBERT: A Pretrained Language Model for Financial Communications

Yi Yang, Mark Christopher Siy Uy+1 more|arXiv (Cornell University)|Jun 15, 2020

Topic Modeling11 references176 citations

TL;DR

FinBERT, a finance-domain BERT model pretrained on 4.9B tokens from financial texts, outperforms generic BERT on three financial sentiment tasks.

ABSTRACT

Contextual pretrained language models, such as BERT (Devlin et al., 2019), have made significant breakthrough in various NLP tasks by training on large scale of unlabeled text re-sources.Financial sector also accumulates large amount of financial communication text.However, there is no pretrained finance specific language models available. In this work,we address the need by pretraining a financial domain specific BERT models, FinBERT, using a large scale of financial communication corpora. Experiments on three financial sentiment classification tasks confirm the advantage of FinBERT over generic domain BERT model. The code and pretrained models are available at https://github.com/yya518/FinBERT. We hope this will be useful for practitioners and researchers working on financial NLP tasks.

Motivation & Objective

• Motivate the need for a finance-domain pretrained language model due to domain-specific sentiment signals in financial texts.
• Build FinBERT by pretraining BERT on large-scale financial corpora.
• Demonstrate FinBERT's superiority over generic BERT on financial sentiment classification tasks.

Proposed method

• Construct FinVocab (finance-focused) and FinBERT variants (BaseVocab/FinVocab; cased/uncased).
• Pretrain FinBERT on ~4.9B tokens from corporate reports, earnings call transcripts, and analyst reports.
• Fine-tune FinBERT with a simple linear classifier and cross-entropy loss for sentiment tasks.
• Compare FinBERT against BERT-Base (cased/uncased) on three datasets.
• Evaluate with accuracy on multiple splits to assess performance gains.

Experimental results

Research questions

• RQ1 Does FinBERT improve financial sentiment classification compared to generic BERT on standard datasets?
• RQ2 How do in-domain vocabulary (FinVocab) and full financial corpora pretraining affect performance?
• RQ3 Are uncased models consistently superior to cased models in financial NLP tasks?
• RQ4 What is the impact of combining different financial corpora on FinBERT performance?

Key findings

• On PhraseBank, the best FinBERT model (uncased FinVocab) achieves 0.872 accuracy, a 4.4% improvement over uncased BERT and 15.4% over cased BERT.
• On FiQA, the best FinBERT model (uncased FinVocab) achieves 0.844 accuracy, a 15.6% improvement over uncased BERT and 29.2% over cased BERT.
• On AnalystTone, the best FinBERT model (uncased FinVocab) achieves 0.887, improving 4.3% over uncased BERT and 5.5% over cased BERT.
• FinBERT models pretrained on all financial corpora perform best, with Analyst Reports contributing strong performance across tasks.