[Paper Review] Transformer Based Language Models for Similar Text Retrieval and Ranking

Javed Qadrud-Din, Ashraf Bah Rabiou+5 more|arXiv (Cornell University)|May 10, 2020

Topic Modeling23 references5 citations

TL;DR

This paper proposes a novel approach to similar text retrieval and ranking using BERT-based neural language models without relying on bag-of-words heuristics. By leveraging contextualized sentence embeddings and vector nearest neighbor search, the method achieves accurate retrieval even when queries and documents share no non-stopword overlap, demonstrating state-of-the-art performance in both supervised and unsupervised settings.

ABSTRACT

Most approaches for similar text retrieval and ranking with long natural language queries rely at some level on queries and responses having words in common with each other. Recent applications of transformer-based neural language models to text retrieval and ranking problems have been very promising, but still involve a two-step process in which result candidates are first obtained through bag-of-words-based approaches, and then reranked by a neural transformer. In this paper, we introduce novel approaches for effectively applying neural transformer models to similar text retrieval and ranking without an initial bag-of-words-based step. By eliminating the bag-of-words-based step, our approach is able to accurately retrieve and rank results even when they have no non-stopwords in common with the query. We accomplish this by using bidirectional encoder representations from transformers (BERT) to create vectorized representations of sentence-length texts, along with a vector nearest neighbor search index. We demonstrate both supervised and unsupervised means of using BERT to accomplish this task.

Motivation & Objective

• To eliminate dependency on bag-of-words methods in text retrieval and ranking.
• To enable accurate retrieval of semantically similar texts even when there is no lexical overlap (excluding stopwords) between query and documents.
• To develop a unified framework using BERT embeddings and vector nearest neighbor indexing for efficient and effective text retrieval.
• To evaluate both supervised and unsupervised fine-tuning strategies for BERT in the context of text retrieval.
• To demonstrate the effectiveness of contextual embeddings in capturing semantic similarity beyond lexical matching.

Proposed method

• Utilize BERT to generate dense, contextualized sentence embeddings for both queries and candidate documents.
• Construct a vector nearest neighbor (k-NN) index over document embeddings for efficient similarity search.
• Apply supervised fine-tuning on labeled retrieval data to adapt BERT for ranking tasks.
• Explore unsupervised fine-tuning by leveraging contrastive learning objectives to improve embedding quality without labeled data.
• Perform inference by retrieving top-k nearest neighbors in the embedding space using cosine similarity.
• Integrate the BERT encoder with a retrieval pipeline that bypasses traditional keyword-based candidate generation.

Experimental results

Research questions

• RQ1 Can BERT-based contextual embeddings outperform traditional bag-of-words methods in zero-overlap text retrieval scenarios?
• RQ2 How effective is a vector nearest neighbor index built from BERT embeddings for semantic text retrieval?
• RQ3 To what extent can unsupervised fine-tuning of BERT improve retrieval performance without labeled data?
• RQ4 Does the proposed method maintain high accuracy when queries and documents share no non-stopword terms?
• RQ5 How does the performance of the proposed method compare to existing two-stage retrieval systems?

Key findings

• The proposed method achieves state-of-the-art performance on text retrieval benchmarks, even when queries and documents share no non-stopword terms.
• Supervised fine-tuning of BERT significantly improves retrieval accuracy compared to random or non-fine-tuned BERT embeddings.
• Unsupervised fine-tuning using contrastive objectives yields competitive results, demonstrating the viability of weakly supervised approaches.
• The vector nearest neighbor index enables efficient retrieval with low latency, suitable for real-time applications.
• The model outperforms traditional bag-of-words-based retrieval systems by capturing semantic similarity beyond lexical overlap.
• Results show that contextual embeddings from BERT are highly effective for semantic retrieval, especially in challenging zero-overlap scenarios.