[Paper Review] Generating Multi-Aspect Queries for Conversational Search

Zahra Abbasiantaeb, Lupart, Simon+1 more|arXiv (Cornell University)|Mar 28, 2024

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes a Generate-then-Retrieve (GTR) framework for conversational information search that leverages large language models (LLMs) to generate multiple, diverse queries from a single LLM-generated answer, improving retrieval over single-query methods. Experiments show that query generation based on the LLM's answer significantly outperforms standard retrieval, especially in complex, multi-fact queries, with a new GPT-3.5-based assessment pool enabling fairer evaluation due to higher judgment coverage.

ABSTRACT

Conversational information seeking (CIS) systems aim to model the user's information need within the conversational context and retrieve the relevant information. One major approach to modeling the conversational context aims to rewrite the user utterance in the conversation to represent the information need independently. Recent work has shown the benefit of expanding the rewritten utterance with relevant terms. In this work, we hypothesize that breaking down the information of an utterance into multi-aspect rewritten queries can lead to more effective retrieval performance. This is more evident in more complex utterances that require gathering evidence from various information sources, where a single query rewrite or query representation cannot capture the complexity of the utterance. To test this hypothesis, we conduct extensive experiments on five widely used CIS datasets where we leverage LLMs to generate multi-aspect queries to represent the information need for each utterance in multiple query rewrites. We show that, for most of the utterances, the same retrieval model would perform better with more than one rewritten query by 85% in terms of nDCG@3. We further propose a multi-aspect query generation and retrieval framework, called MQ4CS. Our extensive experiments show that MQ4CS outperforms the state-of-the-art query rewriting methods. We make our code and our new dataset of generated multi-aspect queries publicly available.

Motivation & Objective

• To address the limitation of single-query rewriting in conversational IR, which fails to capture complex, multi-fact user information needs.
• To improve retrieval performance in conversational search by generating multiple, diverse queries from an LLM-generated answer rather than relying on a single rewritten query.
• To develop a more reliable evaluation benchmark for TREC iKAT by generating a new, high-coverage assessment pool using GPT-3.5 judgments, overcoming the low judged passage rate in official evaluations.
• To investigate whether LLMs' reasoning and knowledge can be leveraged to generate more effective, retrieval-optimized queries for complex conversational queries.

Proposed method

• Propose an AQD (Answer then Query then Retrieve) pipeline: first generate a grounded answer using an LLM, then decompose that answer into multiple search queries.
• Use the LLM to generate up to five diverse, fact-specific queries from the generated answer to cover different aspects of the user’s information need.
• Perform independent retrieval for each generated query, then interleave and re-rank results using a hybrid retrieval strategy.
• Introduce AQD_A, a variant that re-ranks the top passages from each query using the original LLM-generated answer as a reranking signal.
• Implement a QD (Query then Decide) baseline that directly prompts the LLM to generate queries before answering, to compare query quality.
• Create a new, high-coverage relevance assessment pool using GPT-3.5 for 133 conversational turns, with 19,413 relevance judgments, to enable fairer evaluation of retrieval performance.

Figure 1: An example conversation with a complex user request. The system in this case generates three distinct queries from the initial answer and searches for every query in the passage collection. Then it reads and reasons over the top passages to generate the final grounded answer.

Experimental results

Research questions

• RQ1 Can we leverage LLMs’ internal knowledge and reasoning capacity to enhance conversational passage retrieval in a Generate-then-Retrieve pipeline?
• RQ2 Can we leverage LLMs’ reasoning capabilities to generate multiple, related, and searchable queries that improve retrieval effectiveness?
• RQ3 How does the performance of multi-query generation compare to single-query rewriting and direct query generation in complex conversational search?
• RQ4 To what extent do GPT-3.5-generated relevance judgments agree with human assessments, and can they support a fairer evaluation of retrieval systems?

Key findings

• The AQD and AQD_A methods significantly outperform standard retrieval baselines and single-query methods, especially in deeper conversations (turn >14), where context modeling becomes more complex.
• The AQD approach, which generates queries from the LLM’s answer, produces more effective queries than directly prompting the LLM to generate queries or using the answer as a query.
• The official TREC iKAT evaluation pool suffers from a very low rate of judged passages in the top 10 (around 2 per run), leading to unfair and misleading performance comparisons.
• The new GPT-3.5-based assessment pool shows 66.84% agreement with human relevance judgments in binary classification, validating its reliability for evaluation.
• The AQD and AQD_A models achieve superior retrieval performance when evaluated on the new GPT-3.5 pool, demonstrating that the official evaluation was biased against their methods due to low judgment coverage.
• The study confirms that generating multiple queries from an LLM-generated answer leads to better coverage of relevant facts, especially in chain-of-thought reasoning scenarios involving multiple entities or attributes.

Figure 2: Judged@10 metric in TREC iKAT official relevance assessments for the first 10 passages returned by each model.