[Paper Review] Toward Understanding Privileged Features Distillation in Learning-to-Rank

Shuo Yang, Sujay Sanghavi+3 more|arXiv (Cornell University)|Sep 19, 2022

Advanced Bandit Algorithms Research4 citations

TL;DR

This paper investigates privileged feature distillation (PFD) in learning-to-rank, where a teacher model trained on both regular and privileged features (e.g., click history) distills knowledge into a student model that only uses regular features. The key finding is a non-monotonic performance curve: as privileged feature predictive power increases, student performance initially improves but eventually declines due to high-variance teacher predictions.

ABSTRACT

In learning-to-rank problems, a privileged feature is one that is available during model training, but not available at test time. Such features naturally arise in merchandised recommendation systems; for instance, "user clicked this item" as a feature is predictive of "user purchased this item" in the offline data, but is clearly not available during online serving. Another source of privileged features is those that are too expensive to compute online but feasible to be added offline. Privileged features distillation (PFD) refers to a natural idea: train a "teacher" model using all features (including privileged ones) and then use it to train a "student" model that does not use the privileged features. In this paper, we first study PFD empirically on three public ranking datasets and an industrial-scale ranking problem derived from Amazon's logs. We show that PFD outperforms several baselines (no-distillation, pretraining-finetuning, self-distillation, and generalized distillation) on all these datasets. Next, we analyze why and when PFD performs well via both empirical ablation studies and theoretical analysis for linear models. Both investigations uncover an interesting non-monotone behavior: as the predictive power of a privileged feature increases, the performance of the resulting student model initially increases but then decreases. We show the reason for the later decreasing performance is that a very predictive privileged teacher produces predictions with high variance, which lead to high variance student estimates and inferior testing performance.

Motivation & Objective

• To investigate the effectiveness and limitations of privileged feature distillation (PFD) in learning-to-rank, especially when privileged features are unavailable at test time.
• To understand why PFD outperforms baselines like no-distillation, self-distillation, and generalized distillation (GenD) in practice.
• To uncover the underlying mechanisms behind PFD’s performance, particularly the role of teacher prediction variance.
• To analyze the conditions under which PFD fails, despite strong predictive power of privileged features.

Proposed method

• Train a teacher model using both regular and privileged features to generate soft labels for training data.
• Train a student model using only regular features, with a loss that combines ground-truth labels and teacher-predicted labels via a weighted combination.
• Use ablation studies to analyze the impact of the distillation loss weight (α) and label sparsity on performance.
• Conduct theoretical analysis on linear models to derive the estimation error of the student model and identify variance sources.
• Model the data generation process with latent variables, where label variance is decomposed into noise and explainable components from privileged features.
• Derive closed-form expressions for the expected estimation error of the student model under PFD, showing the trade-off between bias reduction and variance increase.

Experimental results

Research questions

• RQ1 Why does PFD outperform multiple baselines, including self-distillation and generalized distillation, across diverse learning-to-rank datasets?
• RQ2 How does the predictive power of privileged features affect the performance of the resulting student model in PFD?
• RQ3 What causes the observed non-monotonic performance curve in PFD, where the most predictive privileged features do not yield the best student performance?
• RQ4 Why is generalized distillation (GenD) — where the teacher uses only privileged features — inferior to PFD in practice?
• RQ5 How does the variance of teacher predictions influence the generalization error of the student model in PFD?

Key findings

• PFD outperforms all baselines — including no-distillation, self-distillation, generalized distillation, and fine-tuned pretraining — on three public learning-to-rank datasets (Yahoo, Istella, MSLRWeb30k) and an industrial-scale Amazon search log dataset.
• The performance of PFD exhibits a non-monotonic relationship with the predictive power of privileged features: performance increases initially but then decreases as the features become more predictive.
• The decline in performance for highly predictive privileged features is caused by high variance in the teacher’s predictions, which leads to high-variance student parameter estimates and worse generalization.
• Theoretical analysis shows that PFD reduces student estimation variance by allowing the student to learn from the part of label variance explained by privileged features, improving generalization.
• Generalized distillation (GenD) underperforms PFD because the teacher’s predictions become uninformative noise when privileged and regular features are independent, and because the teacher lacks regular features to improve prediction quality.
• Empirical results on Amazon data confirm that using the most predictive feature (e.g., 'add-to-cart') does not yield the best PFD performance, validating the non-monotonic behavior observed in theory and synthetic experiments.