[Paper Review] Vector Search with OpenAI Embeddings: Lucene Is All You Need

Jimmy Lin, Ronak Pradeep+2 more|arXiv (Cornell University)|Aug 29, 2023

Advanced Text Analysis Techniques4 citations

TL;DR

This paper demonstrates that Lucene, a widely deployed open-source search library, can effectively support vector search using OpenAI embeddings without requiring a dedicated vector store. By leveraging Lucene's built-in HNSW indexing, the authors achieve competitive retrieval performance on MS MARCO, challenging the prevailing belief that specialized vector databases are necessary for modern AI-powered search systems.

ABSTRACT

We provide a reproducible, end-to-end demonstration of vector search with OpenAI embeddings using Lucene on the popular MS MARCO passage ranking test collection. The main goal of our work is to challenge the prevailing narrative that a dedicated vector store is necessary to take advantage of recent advances in deep neural networks as applied to search. Quite the contrary, we show that hierarchical navigable small-world network (HNSW) indexes in Lucene are adequate to provide vector search capabilities in a standard bi-encoder architecture. This suggests that, from a simple cost-benefit analysis, there does not appear to be a compelling reason to introduce a dedicated vector store into a modern "AI stack" for search, since such applications have already received substantial investments in existing, widely deployed infrastructure.

Motivation & Objective

• Challenge the prevailing narrative that dedicated vector stores are essential for modern AI-powered search systems.
• Demonstrate that existing Lucene-based search infrastructure can natively support vector search with HNSW indexing.
• Argue that the cost and complexity of introducing a new vector store are unjustified when Lucene already provides the required capabilities.
• Show that embedding generation via OpenAI APIs and vector indexing/search in Lucene are straightforward and reproducible.
• Provide a fully reproducible end-to-end pipeline using the MS MARCO passage ranking dataset to validate performance and feasibility.

Proposed method

• Use OpenAI's text-embedding-ada-002 model to generate dense vector representations for all passages in the MS MARCO dataset.
• Index the generated embeddings using Lucene’s hierarchical navigable small-world (HNSW) index for efficient approximate nearest neighbor search.
• Perform retrieval by computing dot product similarities between query embeddings and indexed passage embeddings.
• Utilize Lucene’s native support for vector search, which has been available since version 9 (Dec 2021), to avoid custom implementations.
• Replicate the entire pipeline in Elasticsearch and OpenSearch, confirming that the approach is portable across Lucene-based systems.
• Make all embeddings and code publicly available for reproducibility, including pre-computed OpenAI embeddings.

Experimental results

Research questions

• RQ1 Can Lucene’s HNSW indexing capabilities effectively support vector search with dense embeddings from OpenAI without requiring a dedicated vector store?
• RQ2 How does the retrieval performance of a Lucene-based system compare to state-of-the-art vector databases on standard benchmarks?
• RQ3 What is the cost–benefit tradeoff of introducing a dedicated vector store versus extending existing Lucene-based search infrastructure?
• RQ4 To what extent can embedding APIs and Lucene’s native vector search capabilities eliminate the need for specialized vector database software?
• RQ5 Is it feasible to build a production-ready, reproducible vector search pipeline using only open-source Lucene components and standard embedding APIs?

Key findings

• Lucene’s HNSW indexing provides effective approximate nearest neighbor search for dense vectors, achieving competitive performance on the MS MARCO passage ranking benchmark.
• The system achieves respectable retrieval effectiveness using OpenAI embeddings, demonstrating that high-quality vector search is possible without a dedicated vector store.
• The performance of Lucene in vector search is currently slower than specialized systems like Faiss, but the performance gap is expected to narrow due to ongoing investments in the Lucene ecosystem.
• The integration of vector search into Lucene-based platforms like Elasticsearch and OpenSearch is already underway, with official support and production-ready features being released.
• pgvector, a vector search extension for PostgreSQL, exemplifies a similar low-complexity alternative, showing that vector search can be added to existing relational databases with minimal overhead.
• The authors provide a fully reproducible pipeline with downloadable OpenAI embeddings, enabling others to replicate and extend their results with minimal setup.