[Paper Review] Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search

Kelong Mao, Zhicheng Dou+4 more|arXiv (Cornell University)|Mar 12, 2023

Topic Modeling4 citations

TL;DR

This paper proposes LLM4CS, a prompting framework that leverages large language models (LLMs) to improve conversational search by generating multiple query rewrites and hypothetical responses, then aggregating them into a robust representation of user intent. The method achieves state-of-the-art performance on CAsT-19, CAsT-20, and CAsT-21 benchmarks, outperforming both existing methods and even human rewrites in intent understanding and retrieval effectiveness.

ABSTRACT

Precisely understanding users' contextual search intent has been an important challenge for conversational search. As conversational search sessions are much more diverse and long-tailed, existing methods trained on limited data still show unsatisfactory effectiveness and robustness to handle real conversational search scenarios. Recently, large language models (LLMs) have demonstrated amazing capabilities for text generation and conversation understanding. In this work, we present a simple yet effective prompting framework, called LLM4CS, to leverage LLMs as a text-based search intent interpreter to help conversational search. Under this framework, we explore three prompting methods to generate multiple query rewrites and hypothetical responses, and propose to aggregate them into an integrated representation that can robustly represent the user's real contextual search intent. Extensive automatic evaluations and human evaluations on three widely used conversational search benchmarks, including CAsT-19, CAsT-20, and CAsT-21, demonstrate the remarkable performance of our simple LLM4CS framework compared with existing methods and even using human rewrites. Our findings provide important evidence to better understand and leverage LLMs for conversational search.

Motivation & Objective

• To address the challenge of accurately capturing users' contextual search intent in long-tailed, diverse conversational search sessions.
• To overcome data scarcity in conversational search by leveraging the zero-shot reasoning and generation capabilities of large language models (LLMs).
• To improve the robustness and effectiveness of conversational query rewriting (CQR) by incorporating hypothetical response generation and ensemble aggregation.
• To demonstrate that LLMs can surpass traditional models and even human rewrites in understanding complex, multi-turn search intent.

Proposed method

• The framework uses three tailored prompting methods to generate multiple query rewrites and longer hypothetical responses from the same conversation context.
• It aggregates the generated rewrites and responses into a unified, intent-rich representation using a learnable aggregation mechanism.
• The method leverages the strong contextual understanding of LLMs to resolve coreference and omission issues common in conversational queries.
• It employs a two-stage process: (1) LLM generation of diverse intent expressions, and (2) ensemble aggregation to filter out incorrect or noisy interpretations.
• The framework is evaluated using standard conversational search benchmarks (CAsT-19, CAsT-20, CAsT-21) with both automatic and human evaluation.
• The aggregation strategy enhances robustness by combining multiple plausible interpretations of the same user intent.

Experimental results

Research questions

• RQ1 Can LLMs effectively understand and reconstruct users' contextual search intent in multi-turn conversations?
• RQ2 Does generating both query rewrites and hypothetical responses improve intent representation compared to rewriting alone?
• RQ3 Can ensemble aggregation of multiple LLM-generated outputs enhance robustness and performance in conversational search?
• RQ4 How does the LLM4CS framework compare to human-annotated rewrites and existing CQR/CDR models in intent accuracy and retrieval effectiveness?

Key findings

• LLM4CS achieved state-of-the-art performance on CAsT-19, CAsT-20, and CAsT-21, outperforming both existing methods and human rewrites in retrieval effectiveness.
• Human evaluation showed that 85.5%, 89.4%, and 84.8% of LLM4CS rewrites correctly conveyed the user’s intent on CAsT-19, CAsT-20, and CAsT-21, respectively, compared to 75.1%, 62.0%, and 58.6% for T5QR.
• Less than 3% of LLM4CS rewrites contained coreference errors, compared to ~10% for T5QR, demonstrating superior handling of pronouns and references.
• The inclusion of hypothetical responses significantly improved performance, indicating that modeling plausible answers enhances intent understanding.
• Aggregation of multiple generations effectively filtered out incorrect interpretations and strengthened valid ones, improving robustness.