[Paper Review] Refining Recency Search Results with User Click Feedback

Taesup Moon, Wei Chu+3 more|arXiv (Cornell University)|Mar 19, 2011

Advanced Bandit Algorithms Research23 references3 citations

TL;DR

This paper proposes a real-time re-ranking approach for recency search using user click feedback to adapt ranking models to rapidly changing user intent. By leveraging a random exploration bucket to collect unbiased click data, the authors train online learning models that dynamically adjust document rankings based on click-through rates, significantly improving relevance for time-sensitive queries compared to static, editorially trained models.

ABSTRACT

Traditional machine-learned ranking systems for web search are often trained to capture stationary relevance of documents to queries, which has limited ability to track non-stationary user intention in a timely manner. In recency search, for instance, the relevance of documents to a query on breaking news often changes significantly over time, requiring effective adaptation to user intention. In this paper, we focus on recency search and study a number of algorithms to improve ranking results by leveraging user click feedback. Our contributions are three-fold. First, we use real search sessions collected in a random exploration bucket for \emph{reliable} offline evaluation of these algorithms, which provides an unbiased comparison across algorithms without online bucket tests. Second, we propose a re-ranking approach to improve search results for recency queries using user clicks. Third, our empirical comparison of a dozen algorithms on real-life search data suggests importance of a few algorithmic choices in these applications, including generalization across different query-document pairs, specialization to popular queries, and real-time adaptation of user clicks.

Motivation & Objective

• To address the challenge of tracking rapidly evolving user intent in recency search, where document relevance shifts over time.
• To improve ranking quality for time-sensitive queries by leveraging real-time user click feedback instead of relying solely on static editorial judgments.
• To develop and evaluate a re-ranking system that adapts dynamically to temporal changes in document relevance using online learning.
• To provide a reliable offline evaluation framework for online learning algorithms in web search using a random exploration bucket.
• To identify key algorithmic choices—such as generalization across query-document pairs and real-time adaptation—that significantly impact performance.

Proposed method

• A random exploration bucket was implemented in a live commercial search engine to collect unbiased user click data by randomly shuffling the top four results for a subset of recency-classified queries.
• Click-through rates (CTR) for query-document pairs were modeled using a linear combination of features, including the baseline ranking score from an editorial-judgment-based system.
• Online learning algorithms were trained on real user click feedback to dynamically update ranking parameters in real time, enabling adaptation to changing user preferences.
• A re-ranking approach was applied to the output of the existing recency ranking function, refining results based on CTR predictions derived from click data.
• The evaluation framework allowed offline comparison of multiple algorithms without requiring online A/B testing, ensuring unbiased performance assessment.
• The system incorporated both a global model for generalization and query-document-specific bias terms to balance performance across popular and long-tail queries.

Experimental results

Research questions

• RQ1 How can user click feedback be effectively leveraged to improve ranking for time-sensitive, recency-based queries?
• RQ2 What algorithmic choices—such as generalization, specialization, and real-time adaptation—most significantly impact performance in dynamic recency ranking?
• RQ3 Can a random exploration bucket provide a reliable, unbiased evaluation mechanism for online learning algorithms in web search?
• RQ4 How does online learning based on click feedback compare to batch learning using editorial judgments in terms of ranking quality for recency queries?
• RQ5 What is the impact of modeling CTR with both global and query-specific components on ranking performance across diverse query distributions?

Key findings

• The use of real-time click feedback significantly improved ranking quality for recency queries compared to static, editorially trained models.
• Online learning algorithms that adapt in real time outperformed batch methods, demonstrating the importance of dynamic adaptation to shifting user intent.
• Generalization across query-document pairs and specialization to popular queries were identified as critical factors for effective performance across the full query distribution.
• The random exploration bucket provided a reliable offline evaluation mechanism, eliminating positional bias and enabling unbiased comparison of online learning algorithms without online A/B testing.
• Empirical evaluation on 399,880 search sessions showed that CTR-based re-ranking with online learning led to measurable improvements in ranking effectiveness for time-sensitive queries.
• The study demonstrated that user click data can capture subtle, rapidly evolving relevance shifts that are difficult for human editors to anticipate or label accurately.