[論文レビュー] FlashRAG: A Modular Toolkit for Efficient Retrieval-Augmented Generation Research

Jiajie Jin, Yutao Zhu+8 more|arXiv (Cornell University)|May 22, 2024

Recommender Systems and Techniques被引用数 8

ひとこと要約

FlashRAG は、再現性のある RAG 研究を可能にするオープンソースのモジュール式ツールキットで、12 の実装済み手法と 32 のベンチマークデータセット、再利用可能なパイプラインと評価ツールを備えています。

ABSTRACT

With the advent of large language models (LLMs) and multimodal large language models (MLLMs), the potential of retrieval-augmented generation (RAG) has attracted considerable research attention. Various novel algorithms and models have been introduced to enhance different aspects of RAG systems. However, the absence of a standardized framework for implementation, coupled with the inherently complex RAG process, makes it challenging and time-consuming for researchers to compare and evaluate these approaches in a consistent environment. Existing RAG toolkits, such as LangChain and LlamaIndex, while available, are often heavy and inflexibly, failing to meet the customization needs of researchers. In response to this challenge, we develop \ours{}, an efficient and modular open-source toolkit designed to assist researchers in reproducing and comparing existing RAG methods and developing their own algorithms within a unified framework. Our toolkit has implemented 16 advanced RAG methods and gathered and organized 38 benchmark datasets. It has various features, including a customizable modular framework, multimodal RAG capabilities, a rich collection of pre-implemented RAG works, comprehensive datasets, efficient auxiliary pre-processing scripts, and extensive and standard evaluation metrics. Our toolkit and resources are available at https://github.com/RUC-NLPIR/FlashRAG.

研究の動機と目的

• 標準化と再現性の推進 in retrieval-augmented generation (RAG) 研究。
• 既存の RAG 手法を再現し、新しい手法を構築するための、モジュラーで研究者に優しいフレームワークを提供。
• 実験を効率化する包括的なベンチマークスイートと前処理スクリプトを提供。
• 自動化された評価指標と複数の RAG ワークフローをサポートするパイプラインシステムを提供。

提案手法

• Two-level modular design: component-level (Judger, Retriever, Reranker, Refiner, Generator) and pipeline-level (8 common RAG pipelines).
• Pre-implemented advanced RAG algorithms (12 methods) across Sequential, Conditional, Branching, and Loop categories.
• 32 benchmark datasets preprocessed into a unified JSONL format and hosted on HuggingFace.
• Efficient auxiliary scripts for corpus preparation, indexing, and retrieval-result handling (including retrieval caches).
• Integration with major LLM toolchains (vLLM, FastChat, Transformers) and support for FiD-style decoding to optimize inference.

実験結果

リサーチクエスチョン

• RQ1 How can a modular toolkit standardize and accelerate RAG method development and evaluation?
• RQ2 How do different components and pipeline designs affect RAG performance across diverse datasets?
• RQ3 What is the impact of retrieval quantity and retriever quality on overall RAG performance?
• RQ4 Can researchers reproduce and fairly compare existing RAG methods within a unified framework?

主な発見

Optimize	Pipeline	NQ	TriviaQA	HotpotQA	2Wiki	PopQA	WebQA
Naive Generation	Sequential	22.6	55.7	28.4	33.9	21.7	18.8
Standard RAG	Sequential	35.1	58.8	35.3	21.0	36.7	15.7
AAR [72]	Sequential	30.1	56.8	33.4	19.8	36.1	16.1
LongLLMLingua [20]	Sequential	32.2	59.2	37.5	25.0	38.7	17.5
RECOMP-abstractive [18]	Sequential	33.1	56.4	37.5	32.4	39.9	20.2
Selective-Context [21]	Sequential	30.5	55.6	34.4	18.5	33.5	17.3
Ret-Robust* [73]	Sequential	42.9	68.2	35.8	43.4	57.2	9.1
SuRe [29]	Branching	37.1	53.2	33.4	20.6	48.1	24.2
REPLUG [28]	Branching	28.9	57.7	31.2	21.1	27.8	20.2
SKR [10]	Conditional	25.5	55.9	29.8	28.5	24.5	18.6
Self-RAG* [33]	Loop	36.4	38.2	29.6	25.1	32.7	21.9
FLARE [34]	Loop	22.5	55.8	28.0	33.9	20.7	20.2
Iter-RetGen [30], ITRG [31]	Loop	36.8	60.1	38.3	21.6	37.9	18.2

• RAG methods substantially outperform naive generation baselines across multiple datasets.
• Refiners provide notable gains, especially on multi-hop datasets like HotpotQA and 2WikiMultihopQA.
• Adaptive or loop-based RAG flows (e.g., Self-RAG, Iter-RetGen, SuRe, FLARE) yield larger improvements on complex tasks compared to simpler datasets.
• Performance is highly sensitive to the number of retrieved documents, with top-3 or top-5 often offering the best balance between quality and noise.
• Ret-Robust and other generator-focused methods can significantly boost results, highlighting the benefit of optimizing specific RAG components.
• Overall, FlashRAG enables fair benchmarking and reproducing existing methods under unified settings.