[Paper Review] Evaluating Interpolation and Extrapolation Performance of Neural Retrieval Models

Jingtao Zhan, Xiaohui Xie+5 more|arXiv (Cornell University)|Apr 25, 2022

Domain Adaptation and Few-Shot Learning4 citations

TL;DR

This paper proposes a novel evaluation protocol to separately assess interpolation and extrapolation performance of neural retrieval models by resampling training and test data based on query embedding similarity. Results show that while representation-based models perform nearly as well as interaction-based models in interpolation, they significantly underperform in extrapolation, highlighting the need for distinct evaluation of generalization capacity beyond standard benchmarks.

ABSTRACT

A retrieval model should not only interpolate the training data but also extrapolate well to the queries that are different from the training data. While neural retrieval models have demonstrated impressive performance on ad-hoc search benchmarks, we still know little about how they perform in terms of interpolation and extrapolation. In this paper, we demonstrate the importance of separately evaluating the two capabilities of neural retrieval models. Firstly, we examine existing ad-hoc search benchmarks from the two perspectives. We investigate the distribution of training and test data and find a considerable overlap in query entities, query intent, and relevance labels. This finding implies that the evaluation on these test sets is biased toward interpolation and cannot accurately reflect the extrapolation capacity. Secondly, we propose a novel evaluation protocol to separately evaluate the interpolation and extrapolation performance on existing benchmark datasets. It resamples the training and test data based on query similarity and utilizes the resampled dataset for training and evaluation. Finally, we leverage the proposed evaluation protocol to comprehensively revisit a number of widely-adopted neural retrieval models. Results show models perform differently when moving from interpolation to extrapolation. For example, representation-based retrieval models perform almost as well as interaction-based retrieval models in terms of interpolation but not extrapolation. Therefore, it is necessary to separately evaluate both interpolation and extrapolation performance and the proposed resampling method serves as a simple yet effective evaluation tool for future IR studies.

Motivation & Objective

• To investigate whether existing ad-hoс retrieval benchmarks truly evaluate generalization or are biased toward interpolation.
• To develop a method for isolating and measuring interpolation and extrapolation performance without requiring new annotations or test sets.
• To re-evaluate widely used neural retrieval models under separate interpolation and extrapolation regimes to understand their generalization behavior.
• To demonstrate that extrapolation performance correlates strongly with out-of-distribution generalization, unlike interpolation.

Proposed method

• Queries from existing benchmarks (e.g., MS MARCO, TREC DL) are embedded into a vector space using pre-trained models.
• Query similarity is computed via embedding cosine distance to define training-test pairs as similar (interpolation) or dissimilar (extrapolation).
• The dataset is resampled to create new training and test splits: one emphasizing similar queries (interpolation), the other emphasizing dissimilar queries (extrapolation).
• Models are trained and evaluated on these resampled splits to isolate performance in each regime.
• The method is validated by comparing extrapolation performance to out-of-distribution generalization on a separate test set, showing strong correlation.

Experimental results

Research questions

• RQ1 To what extent do current ad-hoc retrieval benchmarks evaluate interpolation rather than true extrapolation?
• RQ2 How do different neural retrieval model architectures (e.g., representation-based vs. interaction-based) perform under interpolation versus extrapolation regimes?
• RQ3 Does the proposed resampling-based evaluation protocol effectively isolate and measure extrapolation performance?
• RQ4 How well does extrapolation performance on the resampled data correlate with out-of-distribution generalization on a held-out test set?

Key findings

• Existing benchmarks like MS MARCO and TREC Deep Learning Tracks exhibit substantial overlap in query entities, intents, and relevance labels between training and test sets, indicating strong bias toward interpolation.
• Representation-based models (e.g., dense retrieval) achieve near-peak performance in interpolation but degrade significantly in extrapolation compared to interaction-based models like ColBERT.
• Extrapolation performance on the resampled dataset strongly correlates with out-of-distribution generalization performance, validating the method’s effectiveness.
• Interpolation performance shows weak correlation with out-of-distribution performance, suggesting it is a poor proxy for generalization.
• The results demonstrate that models can interpolate well on similar queries but fail to generalize to truly novel queries, revealing a critical gap in current evaluation practices.