[Paper Review] Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs

Jiejun Tan, Zhicheng Dou+4 more|arXiv (Cornell University)|Feb 19, 2024

Private Equity and Venture Capital4 citations

TL;DR

This paper proposes SlimPLM, a method that uses a small proxy language model to determine when and what knowledge to retrieve for large language models (LLMs), reducing unnecessary retrieval and lowering inference costs. By generating heuristic answers from the proxy model, SlimPLM identifies missing knowledge and formulates targeted retrieval queries, achieving state-of-the-art performance on five question-answering datasets with lower LLM inference costs.

ABSTRACT

The integration of large language models (LLMs) and search engines represents a significant evolution in knowledge acquisition methodologies. However, determining the knowledge that an LLM already possesses and the knowledge that requires the help of a search engine remains an unresolved issue. Most existing methods solve this problem through the results of preliminary answers or reasoning done by the LLM itself, but this incurs excessively high computational costs. This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in LLMs with a slim proxy model, to enhance the LLM's knowledge acquisition process. We employ a proxy model which has far fewer parameters, and take its answers as heuristic answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM. We only conduct retrieval for the missing knowledge in questions that the LLM does not know. Extensive experimental results on five datasets with two LLMs demonstrate a notable improvement in the end-to-end performance of LLMs in question-answering tasks, achieving or surpassing current state-of-the-art models with lower LLM inference costs.

Motivation & Objective

• To address the challenge of determining when retrieval is necessary for LLMs to avoid unnecessary computation and irrelevant results.
• To reduce the high computational cost of existing retrieval-augmented generation (RAG) methods that rely on multiple LLM inferences.
• To improve the accuracy and efficiency of LLM-based question-answering by identifying knowledge gaps using a lightweight proxy model.
• To enable selective, targeted retrieval only for missing knowledge, minimizing latency and resource usage.

Proposed method

• A slim proxy language model with far fewer parameters than the main LLM is used to generate a heuristic answer for each user question.
• The quality of the heuristic answer is evaluated by a lightweight model to determine whether retrieval is necessary.
• High-quality heuristic answers indicate that the LLM already possesses sufficient knowledge, avoiding retrieval; low-quality answers trigger retrieval.
• The heuristic answer is decomposed into sub-questions that reflect specific knowledge gaps, which are then used to generate targeted retrieval queries.
• Only queries requiring retrieval are passed to the retrieval system, ensuring relevant and focused knowledge acquisition.
• The method is integrated into existing RAG frameworks with minimal computational overhead and no fine-tuning of the main LLM.

Experimental results

Research questions

• RQ1 Can a small proxy model effectively detect knowledge gaps in large language models?
• RQ2 Does using a proxy model reduce the computational cost of retrieval-augmented generation while maintaining or improving performance?
• RQ3 Can heuristic answers from a small model guide the generation of precise retrieval queries for missing knowledge?
• RQ4 How does the performance of SlimPLM compare to state-of-the-art RAG methods in terms of accuracy and inference cost?
• RQ5 Is the proxy model’s assessment of retrieval necessity robust across diverse question-answering datasets?

Key findings

• SlimPLM achieves or surpasses state-of-the-art performance on five benchmark question-answering datasets using only a small proxy model and minimal LLM inference.
• The method reduces LLM inference costs by avoiding retrieval for questions the model can answer independently.
• The proxy model’s heuristic answers are effective in identifying knowledge gaps, with high consistency in detecting when retrieval is needed.
• Targeted query generation from sub-questions improves retrieval relevance and downstream answer quality.
• The approach maintains low latency and computational overhead, making it suitable for real-time applications.
• Empirical results show significant performance gains over baseline RAG methods, especially in reducing hallucination and retrieval noise.