[Paper Review] Ranking Distillation: Learning Compact Ranking Models With High Performance for Recommender System

Jiaxi Tang, Ke Wang|arXiv (Cornell University)|Sep 19, 2018

Recommender Systems and Techniques41 references20 citations

TL;DR

This paper introduces Ranking Distillation (RD), a knowledge distillation technique that trains a compact student model to mimic a larger teacher model in ranking tasks, such as recommender systems. By leveraging both ground-truth labels and the teacher’s top-K ranking supervision, the student achieves near-teacher performance with less than half the model size, significantly improving inference efficiency without sacrificing effectiveness.

ABSTRACT

We propose a novel way to train ranking models, such as recommender systems, that are both effective and efficient. Knowledge distillation (KD) was shown to be successful in image recognition to achieve both effectiveness and efficiency. We propose a KD technique for learning to rank problems, called \emph{ranking distillation (RD)}. Specifically, we train a smaller student model to learn to rank documents/items from both the training data and the supervision of a larger teacher model. The student model achieves a similar ranking performance to that of the large teacher model, but its smaller model size makes the online inference more efficient. RD is flexible because it is orthogonal to the choices of ranking models for the teacher and student. We address the challenges of RD for ranking problems. The experiments on public data sets and state-of-the-art recommendation models showed that RD achieves its design purposes: the student model learnt with RD has a model size less than half of the teacher model while achieving a ranking performance similar to the teacher model and much better than the student model learnt without RD.

Motivation & Objective

• To address the trade-off between effectiveness and inference efficiency in ranking models, particularly in large-scale recommender systems.
• To adapt knowledge distillation—proven effective in classification—to ranking problems, where the goal is relative ordering rather than class prediction.
• To design a flexible, model-agnostic distillation framework that enables small student models to match the performance of large teacher models.
• To balance supervision from ground-truth labels and the teacher model’s ranking output, ensuring robust generalization.
• To evaluate the method on real-world datasets using state-of-the-art ranking models, demonstrating both model compression and performance preservation.

Proposed method

• The method trains a smaller student model to minimize two losses: one based on ground-truth relevance labels and another based on the relative ranking order produced by a larger teacher model.
• The teacher model generates a soft ranking output over items for each query, and the student learns to reproduce the relative ordering of the top-K items from this output.
• A hybrid weighting scheme is proposed to assign higher importance to items ranked higher by the teacher, improving distillation fidelity.
• The distillation loss combines KL divergence between the student’s and teacher’s ranking distributions, with a focus on preserving the relative order of top-ranked items.
• The approach is orthogonal to model architecture, allowing it to be applied to any teacher-student pair of ranking models, including deep neural networks.
• The method dynamically balances trust in ground-truth labels and teacher supervision using a weighted combination of losses, improving generalization.

Experimental results

Research questions

• RQ1 Can knowledge distillation be effectively adapted to ranking problems, where the goal is relative ordering rather than classification?
• RQ2 How can the teacher model’s ranking output be meaningfully used to supervise a smaller student model in a way that preserves ranking performance?
• RQ3 What weighting schemes for the teacher’s top-K rankings yield the best performance in distillation for ranking tasks?
• RQ4 To what extent can a student model achieve performance comparable to a large teacher model while being significantly smaller and faster for inference?
• RQ5 How does the proposed method compare to standard training of the student model without distillation in terms of model size and ranking effectiveness?

Key findings

• The student model trained with Ranking Distillation achieved a model size less than half that of the teacher model while maintaining ranking performance nearly identical to the teacher.
• On the Gowalla dataset, the best-performing RD variant (hybrid weighting) achieved P@10 of 0.0878, nDCG@10 of 0.1283, and MAP of 0.0969—significantly outperforming the student model trained without distillation.
• On the Foursquare dataset, the hybrid weighting scheme improved P@10 from 0.0424 (uniform weighting) to 0.0444, with corresponding gains in nDCG@10 and MAP.
• The position importance weighting scheme outperformed uniform and individual model weighting, showing that top-ranked items from the teacher model are more informative for distillation.
• The hybrid weighting strategy, combining position importance and ranking discrepancy, yielded the best results across both datasets, confirming its effectiveness in balancing supervision.
• The method demonstrated strong generalization across different ranking models and datasets, proving its flexibility and robustness in real-world recommendation scenarios.