[Paper Review] End-to-End Retrieval in Continuous Space

Daniel Gillick, Alessandro Presta+1 more|arXiv (Cornell University)|Nov 19, 2018

Topic ModelingComputer Science22 references71 citations

TL;DR

The paper demonstrates end-to-end retrieval in continuous space using dual encoders and off-line ANN search, achieving substantial gains over discrete retrieval baselines on two similar-question datasets. It also introduces negative sampling strategies and discusses evaluation for retrieval systems.

ABSTRACT

Most text-based information retrieval (IR) systems index objects by words or phrases. These discrete systems have been augmented by models that use embeddings to measure similarity in continuous space. But continuous-space models are typically used just to re-rank the top candidates. We consider the problem of end-to-end continuous retrieval, where standard approximate nearest neighbor (ANN) search replaces the usual discrete inverted index, and rely entirely on distances between learned embeddings. By training simple models specifically for retrieval, with an appropriate model architecture, we improve on a discrete baseline by 8% and 26% (MAP) on two similar-question retrieval tasks. We also discuss the problem of evaluation for retrieval systems, and show how to modify existing pairwise similarity datasets for this purpose.

Motivation & Objective

• Motivate end-to-end continuous retrieval as an alternative to discrete inverted-index systems.
• Propose a dual-encoder architecture that encodes queries and candidates independently for efficient ANN retrieval.
• Explore loss functions and negative sampling strategies to train retrieval-optimized embeddings.
• Evaluate end-to-end retrieval on pairwise datasets converted to retrieval tasks to assess real-world performance.

Proposed method

• Define a Dual Encoder framework with a query encoder and a candidate encoder producing d-dimensional embeddings.
• Use cosine similarity as the retrieval score and apply a learned affine transform to scale the similarity to a logit.
• Experiment with multiple negative-sampling losses, with in-batch sampled softmax giving best results.
• Train with mini-batch gradient descent, using a large batch size to provide diverse negatives.
• Convert pairwise datasets into incomplete retrieval tasks by transitive closure to obtain test queries and candidates.
• Evaluate using MAP@100 on retrieved top-K results and compare to traditional baselines like TFIDF and BM25.

Experimental results

Research questions

• RQ1 Can end-to-end continuous retrieval outperform traditional discrete retrieval baselines on real retrieval tasks?
• RQ2 How do different negative sampling losses affect end-to-end retrieval performance?
• RQ3 Does a multi-task dual-encoder setup improve retrieval across related tasks?
• RQ4 What is the effect of batch size on MAP@100 for end-to-end retrieval?

Key findings

• End-to-end dual-encoder retrieval outperforms discrete baselines such as TFIDF and BM25 on Quora and AskUbuntu datasets in MAP@100.
• In-batch sampled softmax loss yields the best retrieval results and faster convergence among tested strategies.
• Larger batch sizes improve MAP@100, with 2, 10, 100, and 1000 yielding progressively higher scores in experiments.
• Multi-task training with data from multiple queries tasks provides strong retrieval performance.
• Best overall results are achieved with a multi-task dual encoder combining Paralex, Quora, and AskUbuntu data.