[Paper Review] AlphaFin: Benchmarking Financial Analysis with Retrieval-Augmented Stock-Chain Framework

Xiang Li, Zhenyu Li+6 more|arXiv (Cornell University)|Mar 19, 2024

Stock Market Forecasting Methods7 citations

TL;DR

The paper presents AlphaFin datasets and the Stock-Chain framework, which combines fine-tuned StockGPT with retrieval-augmented generation to tackle stock trend prediction and financial Q&A, outperforming baselines in ARR and accuracy.

ABSTRACT

The task of financial analysis primarily encompasses two key areas: stock trend prediction and the corresponding financial question answering. Currently, machine learning and deep learning algorithms (ML&DL) have been widely applied for stock trend predictions, leading to significant progress. However, these methods fail to provide reasons for predictions, lacking interpretability and reasoning processes. Also, they can not integrate textual information such as financial news or reports. Meanwhile, large language models (LLMs) have remarkable textual understanding and generation ability. But due to the scarcity of financial training datasets and limited integration with real-time knowledge, LLMs still suffer from hallucinations and are unable to keep up with the latest information. To tackle these challenges, we first release AlphaFin datasets, combining traditional research datasets, real-time financial data, and handwritten chain-of-thought (CoT) data. It has a positive impact on training LLMs for completing financial analysis. We then use AlphaFin datasets to benchmark a state-of-the-art method, called Stock-Chain, for effectively tackling the financial analysis task, which integrates retrieval-augmented generation (RAG) techniques. Extensive experiments are conducted to demonstrate the effectiveness of our framework on financial analysis.

Motivation & Objective

• Formally define the financial analysis task as two tasks: stock trend prediction and financial Q&A.
• Create AlphaFin datasets combining traditional datasets, real-time data, and CoT data to train FinLLMs.
• Propose Stock-Chain with RAG to mitigate hallucinations and enable real-time information in analysis.
• Demonstrate Stock-Chain's effectiveness through extensive experiments and ablation studies.

Proposed method

• Fine-tune StockGPT on AlphaFin datasets using LoRA to handle stock trend prediction and provide explanations.
• Stage-1 stock trend prediction: retrieve documents for each company, form a prompt, and predict up/down; select indices predicted to rise and compute ARR with capitalization-weighted portfolio.
• Stage-2 financial Q&A: build a vector DB, extract knowledge via coarse summaries and RefGPT, perform similarity-based retrieval, and fine-tune StockGPT on Stage-2 data; generate responses with a RAG-enabled prompt.
• RAG implementation includes vector embedding with BGE, cosine similarity retrieval, and continuous updating of the knowledge base.
• Evaluation includes ARR, ACC, and risk metrics for Stage-1; ROUGE and human/GPT-4 judgments for Stage-2.

Experimental results

Research questions

• RQ1 Can AlphaFin-scale FinLLMs achieve state-of-the-art stock trend prediction when combined with retrieval-augmented generation?
• RQ2 Does integrating real-time knowledge through RAG improve financial Q&A quality and reduce hallucinations compared to baseline LLMs?
• RQ3 What is the contribution of AlphaFin components (datasets, CoT data) to the performance of StockGPT and Stock-Chain?
• RQ4 How does Stock-Chain perform against traditional ML/DL models and general FinLLMs in terms of ARR and user-perceived effectiveness?

Key findings

• Stock-Chain achieves the highest ARR (30.8%) and ACC (55.63%) among evaluated models on AlphaFin-Test Stage-1.
• Fine-tuning with both financial reports and CoT data yields the best stock trend prediction results, outperforming raw-data only and single-cohort fine-tuning.
• Stock-Chain with RAG provides superior Stage-2 ROUGE scores (e.g., ROUGE-1 0.4352, ROUGE-2 0.3056, ROUGE-L 0.4031) and strong humanGPT-4 preference results.
• Stock-Chain consistently outperforms baselines including FinGPT and FinMA in financial analysis tasks, with significant ARR gains and favorable preference evaluations.
• Ablation studies show that combining News and Reports data yields the best Stage-2 Q&A performance (ROUGE metrics and content quality).