[Paper Review] Query Reformulation using Query History for Passage Retrieval in Conversational Search

Sheng-Chieh Lin, Jheng-Hong Yang+4 more|arXiv (Cornell University)|May 5, 2020

Topic ModelingComputer Science50 references20 citations

TL;DR

This paper proposes a multi-stage pipeline for conversational passage retrieval that combines a traditional IR system with two novel query reformulation methods: Historical Query Expansion (HQE) and Neural Transfer Reformulation (NTR). HQE leverages past queries for expansion, while NTR transfers human conversational understanding to a neural model, achieving an 18% improvement over the top TREC 2019 system in NDCG@3 and reducing the performance gap to manually rewritten queries to just 4 points when fused.

ABSTRACT

Passage retrieval in a conversational context is essential for many downstream applications; it is however extremely challenging due to limited data resources. To address this problem, we present an effective multi-stage pipeline for passage ranking in conversational search that integrates a widely-used IR system with a conversational query reformulation module. Along these lines, we propose two simple yet effective query reformulation approaches: historical query expansion (HQE) and neural transfer reformulation (NTR). Whereas HQE applies query expansion, a traditional IR query reformulation technique, NTR transfers human knowledge of conversational query understanding to a neural query reformulation model. The proposed HQE method was the top-performing submission of automatic systems in CAsT Track at TREC 2019. Building on this, our NTR approach improves an additional 18% over that best entry in terms of NDCG@3. We further analyze the distinct behaviors of the two approaches, and show that fusing their output reduces the performance gap (measured in NDCG@3) between the manually-rewritten and automatically-generated queries to 4 from 22 points when compared with the best CAsT submission.

Motivation & Objective

• To improve passage retrieval in conversational search, where limited training data hinders performance.
• To address the challenge of understanding and reformulating user queries in multi-turn conversations.
• To develop effective query reformulation techniques that enhance retrieval effectiveness without requiring large-scale labeled conversational data.
• To bridge the performance gap between automatically generated and manually rewritten queries in conversational search.
• To integrate traditional IR methods with neural approaches for improved ranking in conversational retrieval.

Proposed method

• The framework employs a multi-stage pipeline combining a standard IR system with a query reformulation module.
• Historical Query Expansion (HQE) reformulates queries by expanding them using terms from previous queries in the conversation history.
• Neural Transfer Reformulation (NTR) uses a neural model fine-tuned to transfer human-annotated conversational query understanding patterns into query reformulation.
• The two reformulation methods are fused to combine their complementary strengths in query understanding.
• The system is evaluated using the CAsT Track at TREC 2019, using NDCG@3 as the primary metric.
• The method leverages existing IR infrastructure while adding lightweight, effective query reformulation components.

Experimental results

Research questions

• RQ1 How effective is historical query expansion (HQE) in improving passage retrieval for conversational search?
• RQ2 To what extent can neural transfer learning improve query reformulation in low-resource conversational settings?
• RQ3 How do HQE and NTR compare in their ability to emulate human query rewriting?
• RQ4 Can fusing HQE and NTR outputs significantly reduce the performance gap between automatic and manual query reformulation?
• RQ5 What is the impact of query reformulation on NDCG@3 in a real conversational retrieval benchmark?

Key findings

• The HQE method achieved the top performance among automatic systems in the CAsT Track at TREC 2019.
• The NTR method improved upon the best CAsT submission by 18% in NDCG@3, demonstrating strong gains from neural transfer.
• Fusing HQE and NTR outputs reduced the performance gap in NDCG@3 between automatically generated and manually rewritten queries from 22 points to just 4 points.
• The two methods exhibit distinct behaviors, with HQE relying on lexical overlap from history and NTR capturing semantic intent through transfer learning.
• The fusion of both approaches leads to more robust and human-like query reformulation, significantly narrowing the gap to manual rewrites.
• The results confirm that combining traditional IR techniques with neural transfer learning yields superior performance in low-resource conversational retrieval.