[Paper Review] Online Learning to Rank with List-level Feedback for Image Filtering

Chang Li, Artem Grotov+2 more|arXiv (Cornell University)|Dec 12, 2018

Domain Adaptation and Few-Shot Learning36 references4 citations

TL;DR

This paper proposes two online learning to rank methods—PGLearn and RegLearn—for image filtering using only list-level feedback, where user feedback is aggregated at the list level rather than for individual items. RegLearn, which uses regression to predict list-level feedback and back-propagates through relevance scores, outperforms PGLearn, especially with larger list sizes, and effectively learns discounted weights from ideal or low-noise feedback.

ABSTRACT

Online learning to rank (OLTR) via implicit feedback has been extensively studied for document retrieval in cases where the feedback is available at the level of individual items. To learn from item-level feedback, the current algorithms require certain assumptions about user behavior. In this paper, we study a more general setup: OLTR with list-level feedback, where the feedback is provided only at the level of an entire ranked list. We propose two methods that allow online learning to rank in this setup. The first method, PGLearn, uses a ranking model to generate policies and optimizes it online using policy gradients. The second method, RegLearn, learns to combine individual document relevance scores by directly predicting the observed list-level feedback through regression. We evaluate the proposed methods on the image filtering task, in which deep neural networks (DNNs) are used to rank images in response to a set of standing queries. We show that PGLearn does not perform well in OLTR with list-level feedback. RegLearn, instead, shows good performance in both online and offline metrics.

Motivation & Objective

• To address the challenge of online learning to rank (OLTR) in image filtering when only list-level feedback is available, rather than item-level feedback.
• To develop methods that can learn from pixel-based features using deep neural networks, avoiding reliance on textual features common in traditional OLTR.
• To evaluate the effectiveness of OLTR methods under list-level feedback, particularly in scenarios with high feedback noise or limited user interaction data.
• To compare the performance of policy gradient-based (PGLearn) and regression-based (RegLearn) approaches in online and offline metrics for image ranking.

Proposed method

• PGLearn uses a policy gradient approach, modeling the ranked list as an action and optimizing the ranking policy based on list-level feedback using reinforcement learning.
• RegLearn directly predicts list-level feedback (e.g., click-through rate or nDCG) by combining individual document relevance scores via a regression head.
• Both methods use deep neural networks as the underlying ranking model to process image features and generate relevance scores.
• RegLearn employs back-propagation to update the ranking model based on the difference between predicted and observed list-level feedback.
• The methods are evaluated using the MSCOCO dataset with simulated feedback under different configurations: ideal (nDCG@k) and noisy (CTR@k) setups.
• Exploration strategies such as ε-greedy with ε=0.1 are used to balance exploration and exploitation during online training.

Experimental results

Research questions

• RQ1 Can online learning to rank be effectively applied to image filtering when feedback is only available at the list level rather than for individual items?
• RQ2 How do PGLearn and RegLearn compare in performance when learning from list-level feedback, especially as list size increases?
• RQ3 Can RegLearn learn the correct discounted weights (e.g., for nDCG) and examination probabilities (e.g., for click models) from list-level feedback?
• RQ4 How does feedback noise affect the ability of RegLearn to learn a reliable ranking model?
• RQ5 Can prior knowledge of user behavior (e.g., examination probabilities) improve the performance of RegLearn?

Key findings

• RegLearn with ε=0.1 achieves average nDCG@5 of 0.67±0.01 and 0.66±0.00 under perfect and locating click configurations, respectively, showing strong online performance.
• PGLearn performs poorly when list size exceeds 2, failing to generalize to larger lists despite being designed for policy gradient optimization.
• RegLearn learns discounted weights close to the ground truth (Euclidean distance 0.079) under ideal nDCG feedback, indicating accurate modeling of ranking quality.
• Under noisy click feedback, RegLearn’s performance degrades: Euclidean distances to ground truth weights are 0.231 (perfect) and 0.372 (locating), and it fails entirely under the most noisy (entertaining) configuration.
• OracleLearn, which uses prior knowledge of examination probabilities, consistently improves performance, suggesting that incorporating user behavior priors enhances learning.
• Pure exploration (ε=1) prevents RegLearn from training effectively, indicating that balanced exploration (ε=0.1) is essential for successful learning.