[Paper Review] A Simple yet Effective Framework for Active Learning to Rank

Qingzhong Wang, Haifang Li+8 more|arXiv (Cornell University)|May 20, 2022

Machine Learning and Algorithms52 references4 citations

TL;DR

This paper proposes a simple yet effective active learning framework for learning to rank (LTR) in web search, combining Ranking Entropy (RE) to identify uncertain low-frequency queries and Prediction Variance (PV) to prioritize high-diversity, high-frequency queries. The hybrid RE+PV strategy improves LTR model performance by up to 1.38% in DCG with the same labeling budget, reducing annotation costs while increasing training efficiency at scale in Baidu's search engine.

ABSTRACT

While China has become the biggest online market in the world with around 1 billion internet users, Baidu runs the world largest Chinese search engine serving more than hundreds of millions of daily active users and responding billions queries per day. To handle the diverse query requests from users at web-scale, Baidu has done tremendous efforts in understanding users' queries, retrieve relevant contents from a pool of trillions of webpages, and rank the most relevant webpages on the top of results. Among these components used in Baidu search, learning to rank (LTR) plays a critical role and we need to timely label an extremely large number of queries together with relevant webpages to train and update the online LTR models. To reduce the costs and time consumption of queries/webpages labeling, we study the problem of Activ Learning to Rank (active LTR) that selects unlabeled queries for annotation and training in this work. Specifically, we first investigate the criterion -- Ranking Entropy (RE) characterizing the entropy of relevant webpages under a query produced by a sequence of online LTR models updated by different checkpoints, using a Query-By-Committee (QBC) method. Then, we explore a new criterion namely Prediction Variances (PV) that measures the variance of prediction results for all relevant webpages under a query. Our empirical studies find that RE may favor low-frequency queries from the pool for labeling while PV prioritizing high-frequency queries more. Finally, we combine these two complementary criteria as the sample selection strategies for active learning. Extensive experiments with comparisons to baseline algorithms show that the proposed approach could train LTR models achieving higher Discounted Cumulative Gain (i.e., the relative improvement ΔDCG4=1.38%) with the same budgeted labeling efforts.

Motivation & Objective

• To reduce the high cost and time of labeling massive query-webpage pairs in large-scale web search by selecting the most informative samples for active learning.
• To address the bias of uncertainty-based active learning toward low-frequency queries due to insufficient training signals in Baidu's search environment.
• To improve LTR model training efficiency by identifying queries that offer maximum information gain through complementary criteria.
• To develop a scalable, lightweight selection strategy suitable for real-time deployment across hundreds of millions of queries daily.
• To validate the effectiveness of combining uncertainty (RE) and diversity (PV) criteria in improving ranking performance under budgeted labeling constraints.

Proposed method

• The framework uses Ranking Entropy (RE) to measure uncertainty in LTR predictions by applying a Query-By-Committee (QBC) method across multiple model checkpoints, identifying queries with inconsistent ranking outputs.
• Prediction Variance (PV) is introduced as a new criterion to quantify the diversity of predicted ranking scores across relevant webpages under a query, serving as a proxy for true ranking diversity.
• RE and PV are combined into a hybrid selection strategy to balance exploration of uncertain low-frequency queries and exploitation of informative high-frequency queries.
• The method selects queries with high combined RE+PV scores for labeling, ensuring both model uncertainty and ranking diversity are prioritized.
• The approach is deployed in Baidu’s production search system, with daily model updates and online A/B testing on 0.6% of traffic to evaluate real-world performance.
• Offline experiments use ground-truth annotations to compute DCG and R01 improvements, while online experiments compare RE+PV against random selection using real traffic metrics.

Experimental results

Research questions

• RQ1 How does Ranking Entropy (RE) perform in identifying informative queries for active learning in LTR, and what biases does it introduce in a real-world search setting?
• RQ2 Can Prediction Variance (PV) effectively capture the diversity of true ranking labels and serve as a reliable surrogate for information gain in query selection?
• RQ3 To what extent does combining RE and PV improve LTR model performance compared to using either criterion alone or random selection?
• RQ4 Does the hybrid RE+PV strategy reduce labeling costs while maintaining or improving ranking quality in large-scale web search systems?
• RQ5 How does the proposed method perform on low-frequency queries, which are typically underrepresented and harder to train on due to lack of supervisory signals?

Key findings

• The use of Ranking Entropy (RE) alone leads to a bias toward low-frequency queries due to insufficient training signals, resulting in higher uncertainty scores even when such queries offer limited information gain.
• Prediction Variance (PV) correlates strongly with the true diversity of ranking scores across webpages and serves as an effective proxy for information gain, favoring high-frequency, high-diversity queries.
• The combined RE+PV strategy achieves a relative improvement of up to 1.38% in DCG@4 in offline experiments and 0.35% in online experiments compared to random selection, under the same labeling budget.
• The proposed method discovers 43% more valid training pairs than random selection, significantly improving training efficiency and model generalization.
• Online results show consistent improvements: DCG@4 increases by up to 0.35% across all queries and up to 0.85% on low-frequency queries, while R01 decreases by up to 2.6% on low-frequency queries, indicating better relevance in top results.
• The framework is scalable and practical for real-world deployment, enabling daily model updates with minimal computational overhead on a pool of hundreds of millions of queries.