[Paper Review] FIT-RAG: Black-Box RAG with Factual Information and Token Reduction

Yuren Mao, Xuemei Dong+4 more|arXiv (Cornell University)|Mar 21, 2024

Handwritten Text Recognition Techniques4 citations

TL;DR

FIT-RAG is a black-box retrieval-augmented generation framework that enhances LLM effectiveness and efficiency by jointly leveraging factual information and LLM preferences in retrieval, while reducing input tokens via sub-document-level pruning and self-knowledge recognition. It improves Llama2-13B-Chat accuracy by up to 27.5% on PopQA and reduces input tokens by up to 49% across datasets.

ABSTRACT

Due to the extraordinarily large number of parameters, fine-tuning Large Language Models (LLMs) to update long-tail or out-of-date knowledge is impractical in lots of applications. To avoid fine-tuning, we can alternatively treat a LLM as a black-box (i.e., freeze the parameters of the LLM) and augment it with a Retrieval-Augmented Generation (RAG) system, namely black-box RAG. Recently, black-box RAG has achieved success in knowledge-intensive tasks and has gained much attention. Existing black-box RAG methods typically fine-tune the retriever to cater to LLMs' preferences and concatenate all the retrieved documents as the input, which suffers from two issues: (1) Ignorance of Factual Information. The LLM preferred documents may not contain the factual information for the given question, which can mislead the retriever and hurt the effectiveness of black-box RAG; (2) Waste of Tokens. Simply concatenating all the retrieved documents brings large amounts of unnecessary tokens for LLMs, which degenerates the efficiency of black-box RAG. To address these issues, this paper proposes a novel black-box RAG framework which utilizes the factual information in the retrieval and reduces the number of tokens for augmentation, dubbed FIT-RAG. FIT-RAG utilizes the factual information by constructing a bi-label document scorer. Besides, it reduces the tokens by introducing a self-knowledge recognizer and a sub-document-level token reducer. FIT-RAG achieves both superior effectiveness and efficiency, which is validated by extensive experiments across three open-domain question-answering datasets: TriviaQA, NQ and PopQA. FIT-RAG can improve the answering accuracy of Llama2-13B-Chat by 14.3\% on TriviaQA, 19.9\% on NQ and 27.5\% on PopQA, respectively. Furthermore, it can save approximately half of the tokens on average across the three datasets.

Motivation & Objective

• To address the limitations of existing black-box RAG systems that ignore factual information in retrieval and waste tokens through full-document concatenation.
• To improve retrieval effectiveness by incorporating both factual relevance and LLM preference as dual labels in the retrieval process.
• To enhance token efficiency by reducing input augmentation through self-knowledge recognition and sub-document-level token reduction.
• To achieve a balanced trade-off between answer accuracy and computational efficiency in knowledge-intensive question-answering tasks.
• To enable practical deployment of black-box RAG in real-world applications with limited computational resources and strict token constraints.

Proposed method

• Proposes a bi-label document scorer that jointly optimizes for factual relevance and LLM preference using two separate labels during retrieval.
• Introduces a self-knowledge recognizer to detect whether the LLM can answer the question without external retrieval, avoiding unnecessary augmentation.
• Employs a sub-document-level token reducer that selects only the most relevant sub-documents based on score aggregation and filtering.
• Uses a sophisticated prompt template that encourages reasoning and explanation, improving LLM performance on retrieved knowledge.
• Applies a retrieval strategy that selects top-k documents based on combined scores from the bi-label scorer and applies token reduction only to the most relevant sub-documents.
• Employs a two-stage process: first, retrieve candidate documents; second, apply sub-document filtering and token reduction before feeding to the LLM.

Experimental results

Research questions

• RQ1 How can factual information be effectively integrated into the retrieval process of black-box RAG to improve answer accuracy?
• RQ2 To what extent can token reduction be achieved without sacrificing retrieval effectiveness in black-box RAG?
• RQ3 What is the optimal number of input documents that balances accuracy and efficiency in the token reduction pipeline?
• RQ4 How does prompt design influence the performance of black-box RAG with reduced input tokens?
• RQ5 Can a self-knowledge recognizer effectively avoid unnecessary retrieval and augmentation in black-box RAG?

Key findings

• FIT-RAG improves the answering accuracy of Llama2-13B-Chat by 14.3% on TriviaQA, 19.9% on NQ, and 27.5% on PopQA, demonstrating superior effectiveness.
• The method reduces input tokens by 49% on TriviaQA, 37% on NQ, and 49% on PopQA, achieving significant efficiency gains.
• Using 10 input documents for the token reducer achieves the best trade-off between accuracy and token reduction, as performance declines beyond this point.
• The comprehensive prompt template outperforms both simple and CoT prompts by 2.7% and 1.5% in accuracy, respectively, showing its effectiveness.
• The self-knowledge recognizer successfully identifies cases where no external retrieval is needed, reducing unnecessary augmentation.
• The bi-label document scorer effectively balances factual relevance and LLM preference, preventing retrieval of misleading or irrelevant documents.