[Paper Review] Deep Relevance Ranking Using Enhanced Document-Query Interactions

Ryan McDonald, Georgios-Ioannis Brokos+1 more|arXiv (Cornell University)|Sep 5, 2018

Topic Modeling37 references16 citations

TL;DR

This paper proposes enhanced deep relevance ranking models that integrate context-sensitive term encodings and multiple interaction views into the Deep Relevance Matching Model (drmm), enabling end-to-end training with rich contextual representations. The proposed models, including abel-drmm and posit-drmm with multi-view inputs, outperform BM25, drmm, and PACRR on BioASQ and TREC Robust 2004, achieving state-of-the-art results with significant gains in mean average precision and nDCG.

ABSTRACT

We explore several new models for document relevance ranking, building upon the Deep Relevance Matching Model (DRMM) of Guo et al. (2016). Unlike DRMM, which uses context-insensitive encodings of terms and query-document term interactions, we inject rich context-sensitive encodings throughout our models, inspired by PACRR's (Hui et al., 2017) convolutional n-gram matching features, but extended in several ways including multiple views of query and document inputs. We test our models on datasets from the BIOASQ question answering challenge (Tsatsaronis et al., 2015) and TREC ROBUST 2004 (Voorhees, 2005), showing they outperform BM25-based baselines, DRMM, and PACRR.

Motivation & Objective

• To improve document relevance ranking by incorporating context-sensitive representations into the drmm framework, which traditionally uses context-insensitive term encodings.
• To enable end-to-end training of deep relevance models by designing differentiable interaction mechanisms that support rich contextual embeddings.
• To investigate the impact of multiple views—context-sensitive, context-insensitive, and exact match—on query-document term interaction modeling.
• To evaluate the proposed models on biomedical and general-domain retrieval benchmarks, demonstrating performance gains over strong baselines.
• To analyze the contribution of context-sensitive encodings and multi-view representations through attention visualization and ablation studies.

Proposed method

• Introduces multi-view term encodings: context-sensitive (via CNN/RNN), context-insensitive (pre-trained embeddings), and exact match representations for query and document terms.
• Employs differentiable convolutional n-gram matching features inspired by PACRR, extended to support context-sensitive representations.
• Applies query-focused attention mechanisms to learn document-aware query term representations, enhancing relevance modeling.
• Uses max-pooling and average-k-max pooling to transform rich term representations into fixed-width vectors, emphasizing dense term matches.
• Integrates gating mechanisms to weight query term scores based on term importance, preserving document-aware relevance estimation.
• Employs end-to-end training with backpropagation, enabling joint optimization of context-sensitive encodings and interaction layers.

Experimental results

Research questions

• RQ1 Can context-sensitive term encodings significantly improve relevance ranking performance compared to context-insensitive representations in deep learning models?
• RQ2 How do multiple views of query-document interactions (context-sensitive, context-insensitive, exact match) affect model performance and generalization?
• RQ3 To what extent do differentiable interaction mechanisms enable effective end-to-end training of deep relevance models with rich contextual embeddings?
• RQ4 How do attention-based document representations and pooling strategies enhance the modeling of term match density and relevance?
• RQ5 What is the relative contribution of context-sensitive encodings versus auxiliary features (e.g., BM25) in improving model performance?

Key findings

• The proposed posit-drmm+mv model achieves a mean average precision (MAP) of 48.1 on the BioASQ development set, significantly outperforming BM25 (43.7) and the original drmm (42.5).
• abel-drmm+mv achieves a MAP of 47.6 on BioASQ, and removing context-sensitive encodings reduces performance to 46.3, demonstrating their key contribution.
• On TREC Robust 2004, posit-drmm+mv achieves a MAP of 0.428 and nDCG@20 of 0.612, outperforming BM25 and PACRR, with statistically significant gains.
• Ablation studies show that removing context-sensitive encodings reduces MAP by 1.3 points (from 48.1 to 47.0) in posit-drmm, confirming their importance.
• Visualization of attention scores reveals that context-sensitive representations better capture semantic alignment, even when exact matches are absent.
• The model with multi-view inputs (mv) shows consistent improvements over single-view variants, with statistically significant gains in MAP and nDCG@20 on both datasets.