[Paper Review] InvestLM: A Large Language Model for Investment using Financial Domain Instruction Tuning

Yi Yang, Yixuan Tang+1 more|arXiv (Cornell University)|Sep 15, 2023

Stock Market Forecasting Methods24 citations

TL;DR

InvestLM is a financial-domain LLM built by instruction-tuning LLaMA-65B with a curated, domain-specific instruction dataset, achieving competitive expert assessments and strong cross-task generalization on financial NLP benchmarks.

ABSTRACT

We present a new financial domain large language model, InvestLM, tuned on LLaMA-65B (Touvron et al., 2023), using a carefully curated instruction dataset related to financial investment. Inspired by less-is-more-for-alignment (Zhou et al., 2023), we manually curate a small yet diverse instruction dataset, covering a wide range of financial related topics, from Chartered Financial Analyst (CFA) exam questions to SEC filings to Stackexchange quantitative finance discussions. InvestLM shows strong capabilities in understanding financial text and provides helpful responses to investment related questions. Financial experts, including hedge fund managers and research analysts, rate InvestLM's response as comparable to those of state-of-the-art commercial models (GPT-3.5, GPT-4 and Claude-2). Zero-shot evaluation on a set of financial NLP benchmarks demonstrates strong generalizability. From a research perspective, this work suggests that a high-quality domain specific LLM can be tuned using a small set of carefully curated instructions on a well-trained foundation model, which is consistent with the Superficial Alignment Hypothesis (Zhou et al., 2023). From a practical perspective, this work develops a state-of-the-art financial domain LLM with superior capability in understanding financial texts and providing helpful investment advice, potentially enhancing the work efficiency of financial professionals. We release the model parameters to the research community.

Motivation & Objective

• Motivate the need for accessible, high-quality financial domain LLMs for investment tasks.
• Demonstrate that a small, carefully curated set of domain instructions can effectively fine-tune a foundation model.
• Show that domain instruction tuning yields strong generalization to financial NLP benchmarks.
• Provide expert evaluations comparing InvestLM to state-of-the-art commercial models.
• Offer insights into the impact of domain-specific vs generic instruction data on model performance.

Proposed method

• Fine-tune LLaMA-65B using Low-rank Adaptation (LoRA) with rank 16 on selected projection modules.
• Use Linear Rope Scaling to extend context length to 8,192 tokens for long financial texts.
• Train for 15 epochs with a learning rate of 3e-4 and batch size of 16 (InvestLM-65B).
• Construct a 1,335-instruction dataset from CFA, StackExchange QFin, academic journals, textbooks, SEC filings, financial NLP tasks, and investment questions.
• Compare InvestLM against GPT-3.5, GPT-4, Claude-2, and open baselines through expert evaluation and GPT-4 style scoring.

Figure 1: Expert evaluation.

Experimental results

Research questions

• RQ1 Can a small, carefully curated set of financial-domain instructions effectively transform a strong foundation model into a high-quality financial LLM?
• RQ2 How does domain instruction tuning affect performance compared to generic instruction data on financial NLP tasks?
• RQ3 How does InvestLM perform relative to state-of-the-art commercial models on financial benchmarks and expert evaluations?
• RQ4 What is the generalizability of InvestLM to financial NLP tasks not explicitly used in tuning?
• RQ5 What are the behavioral differences between domain-tuned LLaMA-based models and baseline LLaMA in investment scenarios?

Key findings

• InvestLM’s expert-evaluated responses are comparable to or better than GPT-3.5 and GPT-4 across test questions.
• InvestLM shows strong generalizability on several financial NLP benchmarks not used in tuning.
• Domain instruction tuning yields larger gains for smaller models (7B) than for larger models (65B).
• Generic Alpaca-style instructions can harm domain task performance, underscoring the value of domain-specific data.
• InvestLM reduces hallucinations relative to LLaMA and provides concise, logical investment conclusions.
• InvestLM’s results align with Superficial Alignment Hypothesis by showing high-domain performance from targeted instructions.

Figure 2: GPT-4 evaluation.