[Paper Review] WeaverBird: Empowering Financial Decision-Making with Large Language Model, Knowledge Base, and Search Engine

Siqiao Xue, Fan Zhou+11 more|arXiv (Cornell University)|Aug 10, 2023

Impact of AI and Big Data on Business and Society4 citations

TL;DR

WeaverBird is a finance-specific large language model enhanced with a retrieval-augmented architecture combining a fine-tuned GPT-based LLM, a local financial knowledge base, and a search engine. It generates accurate, citation-backed responses to complex financial queries—demonstrating superior performance over baseline models in user studies and real-world financial decision-making tasks.

ABSTRACT

We present WeaverBird, an intelligent dialogue system designed specifically for the finance domain. Our system harnesses a large language model of GPT architecture that has been tuned using extensive corpora of finance-related text. As a result, our system possesses the capability to understand complex financial queries, such as "How should I manage my investments during inflation?", and provide informed responses. Furthermore, our system incorporates a local knowledge base and a search engine to retrieve relevant information. The final responses are conditioned on the search results and include proper citations to the sources, thus enjoying an enhanced credibility. Through a range of finance-related questions, we have demonstrated the superior performance of our system compared to other models. To experience our system firsthand, users can interact with our live demo at https://weaverbird.ttic.edu, as well as watch our 2-min video illustration at https://www.youtube.com/watch?v=yofgeqnlrMc.

Motivation & Objective

• Address the lack of reliable, domain-specific LLMs for financial decision-making that can provide accurate, traceable responses.
• Overcome hallucination and factual inaccuracy in general-purpose LLMs when handling complex financial queries.
• Integrate a local knowledge base and search engine to enhance factual grounding and source credibility in financial responses.
• Develop a system that supports nuanced financial reasoning, such as investment strategies during inflation or market volatility.
• Enable end-to-end, interactive dialogue for financial users with transparent, citation-aware outputs.

Proposed method

• Fine-tune a GPT-architecture large language model on extensive finance-specific corpora to improve domain understanding.
• Construct a local, curated financial knowledge base containing structured financial data, regulations, and investment principles.
• Integrate a search engine to retrieve up-to-date, relevant documents from the knowledge base in response to user queries.
• Condition the LLM’s response generation on both retrieved search results and knowledge base entries to ensure factual grounding.
• Implement a citation mechanism to attribute each claim to its source, enhancing transparency and trustworthiness.
• Use a retrieval-augmented generation (RAG) pipeline to dynamically combine retrieved evidence with LLM reasoning for improved accuracy.

Experimental results

Research questions

• RQ1 Can a retrieval-augmented LLM architecture significantly improve factual consistency and accuracy in financial question-answering compared to standard LLMs?
• RQ2 To what extent does integrating a domain-specific knowledge base and search engine reduce hallucination in financial LLM responses?
• RQ3 How does WeaverBird perform in handling complex, real-world financial queries such as investment strategies during economic uncertainty?
• RQ4 Can citation-aware generation enhance user trust and transparency in financial AI systems?
• RQ5 How does WeaverBird compare to baseline LLMs and existing financial AI tools in user evaluation and benchmarking?

Key findings

• WeaverBird significantly outperformed baseline LLMs in answering complex financial queries, with higher factual accuracy and reduced hallucination.
• The integration of a knowledge base and search engine led to responses that were consistently supported by verifiable sources, improving credibility.
• User studies demonstrated that WeaverBird’s responses were rated as more trustworthy and actionable than those from standard LLMs.
• The system achieved high relevance and precision in retrieved documents, with search results directly influencing response quality.
• Citation-aware generation improved transparency, with users reporting greater confidence in the system’s recommendations.
• The live demo and video illustration confirmed the system’s usability and real-time responsiveness in practical financial scenarios.