[Paper Review] Learning Optimal Card Ranking from Query Reformulation

Liangjie Hong, Yue Shi+1 more|arXiv (Cornell University)|Jun 22, 2016

Web Data Mining and Analysis31 references3 citations

TL;DR

This paper proposes a learning-to-label framework that leverages user query reformulation as implicit feedback to generate effective relevance labels for query-card pairs in mobile search. By modeling card contributions to reformulation behavior, the method outperforms human-annotated labels and click-based signals, with the learning-to-label (LtL) strategy achieving the best performance by learning discriminative, query-specific card relevance weights.

ABSTRACT

Mobile search has recently been shown to be the major contributor to the growing search market. The key difference between mobile search and desktop search is that information presentation is limited to the screen space of the mobile device. Thus, major search engines have adopted a new type of search result presentation, known as extit{information cards}, in which each card presents summarized results from one domain/vertical, for a given query, to augment the standard blue-links search results. While it has been widely acknowledged that information cards are particularly suited to mobile user experience, it is also challenging to optimize such result sets. Typically, user engagement metrics like query reformulation are based on whole ranked list of cards for each query and most traditional learning to rank algorithms require per-item relevance labels. In this paper, we investigate the possibility of interpreting query reformulation into effective relevance labels for query-card pairs. We inherit the concept of conventional learning-to-rank, and propose pointwise, pairwise and listwise interpretations for query reformulation. In addition, we propose a learning-to-label strategy that learns the contribution of each card, with respect to a query, where such contributions can be used as labels for training card ranking models. We utilize a state-of-the-art ranking model and demonstrate the effectiveness of proposed mechanisms on a large-scale mobile data from a major search engine, showing that models trained from labels derived from user engagement can significantly outperform ones trained from human judgment labels.

Motivation & Objective

• To address the challenge of labeling query-card relevance in mobile search, where traditional click-based or human-annotated labels are insufficient due to low-click cards and temporal relevance.
• To explore whether query reformulation—defined as consecutive similar queries—can serve as a reliable proxy for user satisfaction and card relevance.
• To develop a scalable, principled labeling strategy that maps user-level reformulation behavior to per-card relevance scores for training card ranking models.
• To demonstrate that models trained on reformulation-derived labels significantly outperform those trained on human judgments or click-through rates (CTR).

Proposed method

• Proposes three label interpretation strategies: pointwise (NPL, DPL, MPL), pairwise (APL), and listwise (LL), each mapping reformulation events to per-card relevance scores.
• Introduces the learning-to-label (LtL) method, which trains a model to estimate each card's contribution to query reformulation, using regression weights as labels.
• Uses a state-of-the-art ranking model to evaluate label quality, training on labels derived from reformulation data rather than human judgments or CTR.
• Applies a threshold-based similarity metric to identify consecutive queries as reformulation events, forming the basis for positive/negative label assignment.
• Employs a functional label definition (Equation 4) that combines click and view signals into a total value score to rank cards by engagement contribution.
• Validates label quality via downstream ranking performance, comparing LtL against pointwise, pairwise, and listwise baselines on a large-scale mobile search dataset.

Experimental results

Research questions

• RQ1 Can query reformulation be effectively leveraged as a proxy for relevance in mobile card ranking, especially when click signals are absent or unreliable?
• RQ2 How can reformulation events—observed at the query level—be meaningfully mapped to per-card relevance scores for training ranking models?
• RQ3 Does a learning-to-label strategy that models card contributions to reformulation outperform traditional labeling methods such as human judgment or CTR?
• RQ4 What is the relative performance of different label interpretation strategies (pointwise, pairwise, listwise) when trained on reformulation data?
• RQ5 Can the resulting card ranking models trained on reformulation-derived labels achieve superior performance compared to models trained on CTR or human-annotated labels?

Key findings

• The learning-to-label (LtL) strategy achieves the highest performance across all metrics, outperforming both human-judgment-based labels and CTR-based labels.
• LtL generates labels with high discriminative power by learning card contributions to reformulation, resulting in superior TPR and low TNR.
• The pointwise method DPL performs well by preserving ranking order between positive and negative examples, achieving a strong F-measure.
• APL, a pairwise method, achieves second-best performance with a balanced trade-off between good and bad pairs, while maintaining low computational complexity.
• NPL and MPL underperform due to label interference and over-penalization of single-card movements, respectively, highlighting the importance of preserving ranking structure.
• Regression weights from LtL provide interpretable insights—e.g., NewsCard and WebCard are highly relevant for 'Barack Obama', while NavigationCard dominates for 'Apple'—validating the model’s functional interpretability.