[Paper Review] Large Language Models for Information Retrieval: A Survey

Yutao Zhu, Huaying Yuan+8 more|arXiv (Cornell University)|Aug 14, 2023

Topic ModelingComputer Science92 citations

TL;DR

This survey reviews how large language models (LLMs) are applied to information retrieval (IR) across query rewriting, retrievers, rerankers, readers, and search agents, highlighting methods, challenges, and directions.

ABSTRACT

As a primary means of information acquisition, information retrieval (IR) systems, such as search engines, have integrated themselves into our daily lives. These systems also serve as components of dialogue, question-answering, and recommender systems. The trajectory of IR has evolved dynamically from its origins in term-based methods to its integration with advanced neural models. While the neural models excel at capturing complex contextual signals and semantic nuances, thereby reshaping the IR landscape, they still face challenges such as data scarcity, interpretability, and the generation of contextually plausible yet potentially inaccurate responses. This evolution requires a combination of both traditional methods (such as term-based sparse retrieval methods with rapid response) and modern neural architectures (such as language models with powerful language understanding capacity). Meanwhile, the emergence of large language models (LLMs), typified by ChatGPT and GPT-4, has revolutionized natural language processing due to their remarkable language understanding, generation, generalization, and reasoning abilities. Consequently, recent research has sought to leverage LLMs to improve IR systems. Given the rapid evolution of this research trajectory, it is necessary to consolidate existing methodologies and provide nuanced insights through a comprehensive overview. In this survey, we delve into the confluence of LLMs and IR systems, including crucial aspects such as query rewriters, retrievers, rerankers, and readers. Additionally, we explore promising directions, such as search agents, within this expanding field.

Motivation & Objective

• Consolidate and analyze recent advancements in LLMs applied to IR components (query rewriters, retrievers, rerankers, readers).
• Explain how LLMs are integrated into IR pipelines and identify practical challenges such as data scarcity and hallucination.
• Discuss techniques (prompting, fine-tuning, corpus-enhanced methods) and evaluation aspects.
• Explore future directions including search agents and end-to-end LLM-driven IR models.

Proposed method

• Survey of IR background and LLM concepts relevant to IR.
• Categorization of IR modules (query rewriter, retriever, reranker, reader) and their LLM-enhanced variants.
• Discussion of prompting strategies, fine-tuning, and knowledge distillation for IR tasks.
• Analysis of corpus-enhanced vs LLM-only approaches for query rewriting.
• Overview of search agents (e.g., WebGPT) as unified LLM-driven IR paradigms.

Experimental results

Research questions

• RQ1 How are LLMs currently integrated into IR modules (query rewriter, retriever, reranker, reader) and what are the benefits and limitations?
• RQ2 What prompting strategies, training regimes, and corpus integrations yield the best IR performance with LLMs across ad-hoc and conversational search?
• RQ3 What are the key challenges (data scarcity, hallucination, interpretability) when deploying LLMs in IR, and how can they be mitigated?
• RQ4 What is the role and potential of search agents in automating IR tasks and improving user experience?
• RQ5 What future directions and open problems should guide IR research with LLMs?

Key findings

• LLMs are used to enhance query rewriting for ad-hoc and conversational search by expanding and clarifying queries.
• Corpus-enhanced LLM-based rewriting methods leverage external documents to improve relevance and factuality of rewrites.
• Three main rewriting approaches exist: prompting, fine-tuning, and knowledge distillation, with prompting being prominent in IR applications.
• LLMs contribute to IR through modules beyond rewriting, including retrievers, rerankers, and readers, enabling more context-aware and generated responses.
• Search agents model automated search and result understanding, enabling end-to-end LLM-driven IR workflows (e.g., WebGPT).
• The field emphasizes balancing traditional IR efficiency (e.g., BM25) with neural semantic capabilities to address data scarcity and robustness.