[Paper Review] An Updated Duet Model for Passage Re-ranking

Bhaskar Mitra, Nick Craswell|arXiv (Cornell University)|Mar 18, 2019

Topic Modeling22 references34 citations

TL;DR

This paper presents Duet v2, an updated neural passage re-ranking model that integrates simple modifications (IDF-weighted interactions, word embeddings, ReLU activations, and an MLP fusion with bagging) and demonstrates improved MS MARCO performance through ablations.

ABSTRACT

We propose several small modifications to Duet---a deep neural ranking model---and evaluate the updated model on the MS MARCO passage ranking task. We report significant improvements from the proposed changes based on an ablation study.

Motivation & Objective

• Motivate improvements to the Duet neural ranking model for MS MARCO passage ranking.
• Propose simple architectural and input representation changes to enhance performance and training efficiency.
• Quantify the impact of each modification via ablation studies and compare to state-of-the-art non-BERT baselines.

Proposed method

• Replace character-level n-graph encoding with word embeddings in the distributed sub-model to speed up training.
• Incorporate IDF weighting into the local interaction matrix to emphasize discriminative query terms.
• Replace Tanh with ReLU activations across the model for faster training and potential performance gains.
• Use a multi-layer perceptron to jointly fuse vector outputs from local and distributed sub-models (instead of a single scalar combination).
• Apply bagging by training multiple Duet v2 models with different seeds and data samples to ensemble predictions.
• Train with cross-entropy loss over triplets (q, p+, p−) using Adam optimizer and fixed hyperparameters; trim inputs; limit vocabulary; fixed hidden sizes.

Experimental results

Research questions

• RQ1 Does IDF weighting of the query-document interaction improve ranking performance on MS MARCO?
• RQ2 Do non-linear activations (ReLU) and an MLP-based fusion of sub-model outputs outperform the original Duet design?
• RQ3 Does bagging multiple Duet v2 models yield additional gains in MS MARCO passage ranking?
• RQ4 How does the updated Duet v2 compare to non-BERT baselines and to BERT-based approaches on MS MARCO?

Key findings

• Duet v2 achieves MRR@10 of 0.243 on the dev set and 0.245 on the eval set.
• Ensemble of eight Duet v2 models yields MRR@10 of 0.252 (dev) and 0.253 (eval).
• An ablation removing IDF weighting degrades MRR by about 33%.
• Replacing Tanh with ReLU caused about a 26% degradation in MRR when disabled.
• Using a linear combination of local and distributed scores (instead of an MLP) degrades MRR by about 14%.
• Bagging yields an additional ~3% improvement in MRR.
• Duet v2 approaches comparable performance to other non-BERT top methods on MS MARCO and trains much faster (1.5 hours on a Tesla K40).