[Paper Review] Modeling Document Interactions for Learning to Rank with Regularized Self-Attention

Shuo Sun, Kevin Duh|arXiv (Cornell University)|May 8, 2020

Topic Modeling30 references4 citations

TL;DR

This paper proposes a regularized self-attention mechanism for learning to rank that models interactions among documents in a set, improving ranking performance over standard neural ranking models. By introducing targeted regularization to attention weights, the model learns more meaningful document interactions, achieving state-of-the-art results on LETOR benchmark datasets.

ABSTRACT

Learning to rank is an important task that has been successfully deployed in many real-world information retrieval systems. Most existing methods compute relevance judgments of documents independently, without holistically considering the entire set of competing documents. In this paper, we explore modeling documents interactions with self-attention based neural networks. Although self-attention networks have achieved state-of-the-art results in many NLP tasks, we find empirically that self-attention provides little benefit over baseline neural learning to rank architecture. To improve the learning of self-attention weights, We propose simple yet effective regularization terms designed to model interactions between documents. Evaluations on publicly available Learning to Rank (LETOR) datasets show that training self-attention network with our proposed regularization terms can significantly outperform existing learning to rank methods.

Motivation & Objective

• To address the limitation of existing learning to rank models that compute relevance scores independently without considering competing documents.
• To investigate whether self-attention mechanisms can effectively model document interactions in ranking tasks.
• To improve self-attention's performance in ranking by introducing regularization terms that encourage meaningful attention patterns between documents.
• To develop a self-contained, end-to-end learning to rank model that leverages document set context for better relevance scoring.

Proposed method

• The model uses a ListNet-based neural network architecture that takes a set of document feature vectors as input, where each vector is derived from query-document pairs via a feature function φ.
• Self-attention layers compute query, key, and value representations for each document, enabling attention weights to capture inter-document relationships across the entire document set.
• A novel regularization term is introduced in the loss function to constrain attention weights, promoting more meaningful interactions between documents by penalizing uninformative or uniform attention patterns.
• The regularization is formulated as a weighted sum of attention matrix deviations from uniformity, encouraging attention to focus on relevant document pairs.
• The model is trained using a surrogate listwise loss (cross-entropy between predicted and ground-truth top-one probability distributions) with the added regularization.
• At inference, the model computes relevance scores for all documents simultaneously, using the full set context, and ranks them based on predicted scores.

Experimental results

Research questions

• RQ1 Can self-attention mechanisms effectively model document interactions in learning to rank, beyond what standard neural models achieve?
• RQ2 Why does vanilla self-attention fail to improve ranking performance on standard benchmarks despite its theoretical capability for modeling interactions?
• RQ3 How can regularization be designed to guide self-attention to learn more meaningful document interactions?
• RQ4 Does incorporating document set context through regularized self-attention lead to measurable improvements in standard IR metrics like NDCG and MAP?

Key findings

• The proposed regularized self-attention model significantly outperforms standard ListNet and other strong baselines on the Yahoo, MSLR-WEB10K, and Istella LETOR datasets.
• Empirical analysis shows that unregularized self-attention learns attention weights that are nearly uniform, indicating poor interaction modeling.
• The regularization term effectively reduces attention uniformity and improves the model's ability to distinguish relevant document pairs.
• The model achieves state-of-the-art performance on all three benchmark datasets, demonstrating the effectiveness of incorporating document set context via regularized attention.
• The improvement is consistent across multiple evaluation metrics, including NDCG and MAP, confirming the benefit of holistic document interaction modeling.