[Paper Review] Personalized Query Auto-Completion Through a Lightweight Representation of the User Context

Manojkumar Rangasamy Kannadasan, Grigor Aslanyan|arXiv (Cornell University)|May 3, 2019

Information Retrieval and Search Behavior4 citations

TL;DR

This paper proposes a lightweight, scalable method for personalized query auto-completion using fastText-based embeddings of user context queries to measure semantic similarity with candidate completions. By integrating these embedding-derived features into a LambdaMART ranking model, the approach achieves a 20–30% improvement over the Most Popular Completion baseline and up to 10% improvement over a baseline ranker on sessions with user context, outperforming text-based personalization features.

ABSTRACT

Query Auto-Completion (QAC) is a widely used feature in many domains, including web and eCommerce search, suggesting full queries based on a prefix typed by the user. QAC has been extensively studied in the literature in the recent years, and it has been consistently shown that adding personalization features can significantly improve the performance of QAC. In this work we propose a novel method for personalized QAC that uses lightweight embeddings learnt through fastText. We construct an embedding for the user context queries, which are the last few queries issued by the user. We also use the same model to get the embedding for the candidate queries to be ranked. We introduce ranking features that compute the distance between the candidate queries and the context queries in the embedding space. These features are then combined with other commonly used QAC ranking features to learn a ranking model. We apply our method to a large eCommerce search engine (eBay) and show that the ranker with our proposed feature significantly outperforms the baselines on all of the offline metrics measured, which includes Mean Reciprocal Rank (MRR), Success Rate (SR), Mean Average Precision (MAP), and Normalized Discounted Cumulative Gain (NDCG). Our baselines include the Most Popular Completion (MPC) model as well as a ranking model without our proposed features. The ranking model with the proposed features results in a $20-30\\%$ improvement over the MPC model on all metrics. We obtain up to a $5\\%$ improvement over the baseline ranking model for all the sessions, which goes up to about $10\\%$ when we restrict to sessions that contain the user context. Moreover, our proposed features also significantly outperform text based personalization features studied in the literature before, and adding text based features on top of our proposed embedding based features results only in minor improvements.

Motivation & Objective

• To improve query auto-completion personalization by leveraging user context from recent queries.
• To develop a scalable, lightweight representation of user context using fastText embeddings for semantic similarity modeling.
• To integrate these embeddings into a learning-to-rank framework for better candidate query ranking.
• To evaluate the effectiveness of embedding-based features compared to traditional text-based personalization features.
• To demonstrate significant performance gains in offline metrics on a large-scale commercial search engine (eBay).

Proposed method

• User context is represented as a sequence of the last few queries issued by the user, treated as a single document for embedding learning.
• FastText is used to learn dense, subword-aware vector representations for both context queries and candidate queries.
• Cosine similarity between the user context embedding and each candidate query embedding is computed as a key ranking feature.
• These embedding-based features are combined with standard QAC ranking features (e.g., popularity, time-sensitivity) in a LambdaMART ranker.
• The model is trained and evaluated on a large-scale eBay search dataset, with ablation studies on feature contributions.
• Partial dependence plots are used to analyze the influence of embedding similarity on ranking outcomes.

Experimental results

Research questions

• RQ1 Can a lightweight, fastText-based embedding of user context queries improve query auto-completion ranking performance?
• RQ2 How does the performance of embedding-based features compare to traditional text-based personalization features in QAC?
• RQ3 To what extent do embedding-based features improve ranking metrics when combined with existing features?
• RQ4 Does the model show stronger improvements on sessions where user context is available?
• RQ5 What is the contribution of individual context query embeddings to the final ranking score?

Key findings

• The proposed embedding-based ranking model achieves a 20–30% improvement over the Most Popular Completion (MPC) baseline on all offline metrics, including MRR, SR, MAP, and NDCG.
• On sessions containing user context, the model improves by up to 10% over the baseline ranker, demonstrating the value of context-aware personalization.
• The embedding-based features outperform text-based personalization features, with a 1.5% improvement in MAP@3 on the full dataset and 3% on the context-only dataset.
• Adding text-based features on top of embedding features results in only minor gains, indicating that embedding features capture most of the relevant signal.
• Partial dependence plots confirm a strong, monotonic relationship between cosine similarity in the embedding space and ranking performance.
• The model shows statistically significant improvements across all metrics, with 95% confidence intervals confirming the reliability of the results.