[Paper Review] The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models

Ronak Pradeep, Rodrigo Nogueira+1 more|arXiv (Cornell University)|Jan 14, 2021

Topic ModelingComputer Science61 references40 citations

TL;DR

The paper introduces Expando-Mono-Duo, a multi-stage text ranking design using pretrained sequence-to-sequence models, including document expansion and two reranking stages (monoT5 and duoT5), validated across multiple benchmarks with open-source implementations.

ABSTRACT

We propose a design pattern for tackling text ranking problems, dubbed "Expando-Mono-Duo", that has been empirically validated for a number of ad hoc retrieval tasks in different domains. At the core, our design relies on pretrained sequence-to-sequence models within a standard multi-stage ranking architecture. "Expando" refers to the use of document expansion techniques to enrich keyword representations of texts prior to inverted indexing. "Mono" and "Duo" refer to components in a reranking pipeline based on a pointwise model and a pairwise model that rerank initial candidates retrieved using keyword search. We present experimental results from the MS MARCO passage and document ranking tasks, the TREC 2020 Deep Learning Track, and the TREC-COVID challenge that validate our design. In all these tasks, we achieve effectiveness that is at or near the state of the art, in some cases using a zero-shot approach that does not exploit any training data from the target task. To support replicability, implementations of our design pattern are open-sourced in the Pyserini IR toolkit and PyGaggle neural reranking library.

Motivation & Objective

• Propose a reusable design pattern for text ranking that combines document expansion with seq-to-seq models in a multi-stage pipeline.
• Demonstrate effectiveness across multiple ad hoc retrieval tasks and domains, including zero-shot settings.
• Provide an end-to-end description and ablations to quantify component contributions.
• Offer open-source implementations to support replication and adoption.

Proposed method

• Extend a standard multi-stage ranking architecture with an initial document expansion stage (Expando) using a pretrained sequence-to-sequence model to generate query-like expansions appended to each document.
• Implement first-stage keyword retrieval via inverted indexing (BM25) to produce a candidate set.
• Introduce a two-stage reranking pipeline with monoT5 (pointwise) followed by duoT5 (pairwise) to refine rankings using pretrained seq-to-seq transformers.
• Describe input-output formats for monoT5 and duoT5, including how probabilities are produced and how pairwise scores are aggregated (Sum, Sum-Log, Sym-Sum, Sym-Sum-Log).
• Discuss practical considerations for long texts and the choice to exclude expansion queries from reranking inputs to avoid noise and input length issues.
• Provide deployment details and replicate experiments using Pyserini IR toolkit and PyGaggle neural reranking library.

Experimental results

Research questions

• RQ1 Does the Expando-Mono-Duo pattern achieve state-of-the-art or near-state-of-the-art effectiveness across multiple ad hoc retrieval tasks?
• RQ2 Can document expansion via a seq-to-seq model improve initial retrieval performance without requiring task-specific fine-tuning?
• RQ3 What is the impact of the Mono and Duo components and their aggregation strategies on final ranking quality?
• RQ4 How does zero-shot performance compare when applying the pattern to new tasks/datasets?
• RQ5 What are practical considerations (e.g., input length, computational cost) when deploying Expando-Mono-Duo?

Key findings

• Document expansion with a seq-to-seq model enriches representations prior to indexing, improving first-stage retrieval.
• A two-stage reranking pipeline using monoT5 and duoT5 yields effectiveness at or near the state of the art across several benchmarks.
• Zero-shot setups can achieve strong results without task-specific fine-tuning, demonstrating the pattern's generality.
• Ablation analyses quantify the contributions of Expando, Mono, and Duo components to overall performance.
• The approach is implemented in open-source tools (Pyserini IR toolkit and PyGaggle), supporting replicability.