[Paper Review] A Hierarchical Recurrent Encoder-Decoder For Generative Context-Aware Query Suggestion

Alessandro Sordoni, Yoshua Bengio+4 more|arXiv (Cornell University)|Jul 8, 2015

Topic Modeling36 references4 citations

TL;DR

This paper proposes a hierarchical recurrent encoder-decoder model for generative, context-aware query suggestion that handles variable-length query sequences and produces synthetic suggestions for rare or long-tail queries. By leveraging word-level embeddings and sequence modeling, the model outperforms existing context-aware methods in next-query prediction and generates high-quality, diverse suggestions through end-to-end training and efficient decoding.

ABSTRACT

Users may strive to formulate an adequate textual query for their information need. Search engines assist the users by presenting query suggestions. To preserve the original search intent, suggestions should be context-aware and account for the previous queries issued by the user. Achieving context awareness is challenging due to data sparsity. We present a probabilistic suggestion model that is able to account for sequences of previous queries of arbitrary lengths. Our novel hierarchical recurrent encoder-decoder architecture allows the model to be sensitive to the order of queries in the context while avoiding data sparsity. Additionally, our model can suggest for rare, or long-tail, queries. The produced suggestions are synthetic and are sampled one word at a time, using computationally cheap decoding techniques. This is in contrast to current synthetic suggestion models relying upon machine learning pipelines and hand-engineered feature sets. Results show that it outperforms existing context-aware approaches in a next query prediction setting. In addition to query suggestion, our model is general enough to be used in a variety of other applications.

Motivation & Objective

• Address the challenge of data sparsity in context-aware query suggestion, especially for rare or long-tail queries.
• Enable the generation of synthetic query suggestions not present in training data, improving coverage beyond co-occurrence-based methods.
• Model user query sessions as sequential contexts to preserve reformulation patterns and improve suggestion relevance.
• Develop a neural architecture that is robust to variable-length query histories and sensitive to the order of past queries.
• Achieve efficient, scalable inference using standard decoding techniques while maintaining high-quality, contextually grounded suggestions.

Proposed method

• Employ a hierarchical recurrent encoder-decoder architecture with two levels: word-level and query-level RNNs to model sequences of queries.
• Use word embeddings and query embeddings to capture semantic relationships, enabling clustering of similar terms and queries in the embedding space.
• Train the model end-to-end by maximizing the likelihood of observed query sessions, allowing joint optimization of context representation and generation.
• Generate suggestions by autoregressive decoding, sampling one word at a time conditioned on the full history of previous queries.
• Leverage shared statistical weight across words in the context to mitigate data sparsity, treating the query sequence as a distributed representation.
• Introduce a special end-of-query token to signal completion and form final suggestions from generated word sequences.

Experimental results

Research questions

• RQ1 Can a neural sequence model effectively capture long-term context from variable-length query sessions to improve next-query prediction?
• RQ2 To what extent can a generative model produce high-quality, synthetic suggestions for rare or unseen queries without relying on training data co-occurrence?
• RQ3 How does the hierarchical RNN architecture compare to existing count-based or feature-engineered models in terms of coverage and relevance for long-tail queries?
• RQ4 Can the model generate diverse and contextually appropriate reformulations that reflect generalization or specification patterns in user behavior?
• RQ5 Does the model’s ability to generate suggestions through differentiable, end-to-end training lead to measurable improvements in retrieval performance?

Key findings

• The proposed model outperforms existing context-aware models in next-query prediction, as measured by Mean Reciprocal Rank (MRR), demonstrating improved ranking quality.
• The model generates synthetic suggestions that are judged superior to baseline methods in a user study, indicating higher perceived relevance and usefulness.
• The hierarchical RNN architecture effectively models long-range dependencies and reformulation patterns in query sessions, even with variable-length contexts.
• The model exhibits clustering behavior in learned embeddings, where semantically similar queries and terms are positioned close in the vector space.
• By modeling the context as a sequence of words sharing statistical weight, the model reduces data sparsity and generalizes well to rare queries.
• The model is flexible and extensible, capable of being adapted to other NLP tasks such as auto-completion, language modeling, and next-word prediction.