[Paper Review] Large Language Model based Long-tail Query Rewriting in Taobao Search

Wenjun Peng, Guiyang Li+7 more|arXiv (Cornell University)|Nov 7, 2023

Caching and Content Delivery4 citations

TL;DR

This paper proposes \method, a three-stage framework that leverages large language models (LLMs) for long-tail query rewriting in Taobao search, combining multi-instruction supervised fine-tuning, offline feedback via Taobao's retrieval system, and objective alignment using contrastive learning. The method significantly improves retrieval quality and boosts GMV, #Trans, and UV—especially for long-tail and 'nothing' queries—achieving a 18.66% increase in transaction volume for zero-result queries in online A/B tests.

ABSTRACT

In the realm of e-commerce search, the significance of semantic matching cannot be overstated, as it directly impacts both user experience and company revenue. Along this line, query rewriting, serving as an important technique to bridge the semantic gaps inherent in the semantic matching process, has attached wide attention from the industry and academia. However, existing query rewriting methods often struggle to effectively optimize long-tail queries and alleviate the phenomenon of "few-recall" caused by semantic gap. In this paper, we present BEQUE, a comprehensive framework that Bridges the sEmantic gap for long-tail QUEries. In detail, BEQUE comprises three stages: multi-instruction supervised fine tuning (SFT), offline feedback, and objective alignment. We first construct a rewriting dataset based on rejection sampling and auxiliary tasks mixing to fine-tune our large language model (LLM) in a supervised fashion. Subsequently, with the well-trained LLM, we employ beam search to generate multiple candidate rewrites, and feed them into Taobao offline system to obtain the partial order. Leveraging the partial order of rewrites, we introduce a contrastive learning method to highlight the distinctions between rewrites, and align the model with the Taobao online objectives. Offline experiments prove the effectiveness of our method in bridging semantic gap. Online A/B tests reveal that our method can significantly boost gross merchandise volume (GMV), number of transaction (#Trans) and unique visitor (UV) for long-tail queries. BEQUE has been deployed on Taobao, one of most popular online shopping platforms in China, since October 2023.

Motivation & Objective

• To address the semantic gap in e-commerce search, particularly for long-tail queries that often return no results ('nothing' queries).
• To improve query rewriting quality by aligning LLM-generated rewrites with real-world online objectives such as GMV, transaction count, and hitrate.
• To overcome limitations of prior methods that rely on small models or generic LLM prompting without task-specific fine-tuning or feedback integration.
• To develop a scalable, production-ready framework that enhances retrieval performance while maintaining relevance and controllability in real-world search systems.

Proposed method

• Constructs a high-quality rewriting dataset using rejection sampling and auxiliary tasks (quality classification, product title prediction, chain-of-thought) to enable multi-instruction supervised fine-tuning (SFT) of the LLM.
• Employs beam search with the fine-tuned LLM to generate multiple candidate rewrites per query for downstream evaluation.
• Uses Taobao's offline system to simulate online retrieval and collect partial order feedback based on relevance, increment, and hitrate metrics.
• Applies a contrastive learning-based objective alignment module (PRO) to highlight differences between rewrites and align model outputs with Taobao's online optimization goals.
• Integrates offline feedback and objective alignment into a unified training loop to improve model generalization and retrieval performance.
• Deploys the final model in production on Taobao search since October 2023, with online A/B testing validating its real-world impact.

Figure 1. Framework of Taobao search engine.

Experimental results

Research questions

• RQ1 Can a large language model be effectively fine-tuned to generate high-quality rewrites for long-tail queries that bridge semantic gaps in e-commerce search?
• RQ2 How can offline feedback from a real e-commerce retrieval system be used to improve the alignment of LLM-generated rewrites with online business objectives?
• RQ3 What is the impact of different optimization objectives (relevance, increment, hitrate) on the performance of query rewriting models in real-world settings?
• RQ4 To what extent can multi-instruction fine-tuning and auxiliary task learning enhance the model’s ability to understand and rewrite rare, long-tail queries?

Key findings

• Online A/B tests show \method increased GMV by 0.40%, #Trans by 0.34%, and UV by 0.33% across all queries, demonstrating broad system-wide improvements.
• For rewritten queries (27% of total PV), the model achieved a 2.96% increase in GMV, 1.36% in #Trans, and 1.22% in UV, indicating strong performance on the target population.
• The method achieved a 18.66% increase in #Trans and 6.25% in UV for 'nothing' queries, proving its effectiveness in resolving the most challenging semantic gap cases.
• When optimized for increment, the model significantly improved both increment and hitrate, even at the cost of a marginal decrease in relevance, showing strong trade-off efficiency.
• The objective alignment stage using contrastive learning effectively captured partial order among rewrites, enabling better optimization toward business KPIs than baseline methods.
• The framework outperformed the prior state-of-the-art model CLE-QR in all key metrics, with the largest gains observed in long-tail and zero-result query scenarios.

Figure 2. Framework of \method .