[Paper Review] Neural Ranking Models with Weak Supervision

Mostafa Dehghani, Hamed Zamani+3 more|arXiv (Cornell University)|Apr 28, 2017

Topic Modeling29 references4 citations

TL;DR

This paper proposes training neural ranking models using weak supervision from unsupervised IR models like BM25, which generates large-scale training data without human labeling. By leveraging billions of weakly labeled query-document pairs, the authors achieve over 13% and 35% MAP improvements over BM25 on Robust04 and ClueWeb, respectively, demonstrating that pre-training on weakly supervised data significantly boosts performance, especially with limited labeled data.

ABSTRACT

Despite the impressive improvements achieved by unsupervised deep neural networks in computer vision and NLP tasks, such improvements have not yet been observed in ranking for information retrieval. The reason may be the complexity of the ranking problem, as it is not obvious how to learn from queries and documents when no supervised signal is available. Hence, in this paper, we propose to train a neural ranking model using weak supervision, where labels are obtained automatically without human annotators or any external resources (e.g., click data). To this aim, we use the output of an unsupervised ranking model, such as BM25, as a weak supervision signal. We further train a set of simple yet effective ranking models based on feed-forward neural networks. We study their effectiveness under various learning scenarios (point-wise and pair-wise models) and using different input representations (i.e., from encoding query-document pairs into dense/sparse vectors to using word embedding representation). We train our networks using tens of millions of training instances and evaluate it on two standard collections: a homogeneous news collection(Robust) and a heterogeneous large-scale web collection (ClueWeb). Our experiments indicate that employing proper objective functions and letting the networks to learn the input representation based on weakly supervised data leads to impressive performance, with over 13% and 35% MAP improvements over the BM25 model on the Robust and the ClueWeb collections. Our findings also suggest that supervised neural ranking models can greatly benefit from pre-training on large amounts of weakly labeled data that can be easily obtained from unsupervised IR models.

Motivation & Objective

• To investigate whether unsupervised IR models like BM25 can serve as a weak supervision signal for training effective neural ranking models.
• To evaluate the impact of different input representations and learning objectives (point-wise vs. pair-wise) under weak supervision.
• To determine whether pre-training on weakly supervised data improves performance of supervised ranking models, especially when labeled data is scarce.
• To identify key design factors enabling high performance in weakly supervised neural ranking.

Proposed method

• Using BM25 to automatically generate binary relevance labels (1 for relevant, 0 for non-relevant) for query-document pairs across millions of queries.
• Training feed-forward neural networks on billions of weakly labeled instances, with input representations ranging from raw embeddings to dense/sparse vectors.
• Employing both point-wise and pair-wise learning objectives, with a focus on ranking optimization rather than calibrated scoring to reduce overfitting to weak labels.
• Allowing the network to learn query and document representations end-to-end, rather than relying on fixed feature engineering.
• Pre-training deep neural networks on weakly supervised data before fine-tuning on small amounts of labeled data to improve generalization.
• Evaluating models on standard ad-hoc retrieval benchmarks: Robust04 and ClueWeb09-B.

Experimental results

Research questions

• RQ1 RQ1: Can labels from an unsupervised IR model such as BM25 be used as weak supervision signal to train an effective neural ranker?
• RQ2 RQ2: What input representation and learning objective is most suitable for learning in such a setting?
• RQ3 RQ3: Can a supervised learning model benefit from a weak supervision step, especially in cases when labeled data is limited?

Key findings

• The best-performing neural ranking model achieved over 13% MAP improvement over BM25 on the Robust04 collection and over 35% improvement on the ClueWeb collection.
• Even without any human-labeled data, the neural model outperformed the BM25 model used to generate the weak labels, indicating generalization beyond the weak signal.
• Pre-training on weakly supervised data significantly improved the performance of supervised models when only limited labeled data was available.
• The most effective models learned input representations end-to-end rather than relying on fixed feature sets, enabling better generalization.
• Defining an objective function focused on ranking rather than calibrated scoring helped the network avoid overfitting to imperfections in the weakly labeled data.
• Large-scale training data, made possible by weak supervision, was critical for the success of deep neural networks in this setting.