[Paper Review] Towards Practical Visual Search Engine within Elasticsearch

Cun Mu, Jun Zhao+3 more|arXiv (Cornell University)|Jun 23, 2018

Advanced Image and Video Retrieval Techniques4 references9 citations

TL;DR

This paper presents a novel visual search system built within Elasticsearch by encoding high-dimensional image feature vectors into string tokens, enabling efficient approximate nearest neighbor search using Elasticsearch's inverted index. The proposed vector-to-string encoding method achieves high precision (up to 98.2% at Precision@24) with low latency (under 0.5s), enabling scalable, cost-effective, and multimodal search in e-commerce environments.

ABSTRACT

In this paper, we describe our end-to-end content-based image retrieval system built upon Elasticsearch, a well-known and popular textual search engine. As far as we know, this is the first time such a system has been implemented in eCommerce, and our efforts have turned out to be highly worthwhile. We end up with a novel and exciting visual search solution that is extremely easy to be deployed, distributed, scaled and monitored in a cost-friendly manner. Moreover, our platform is intrinsically flexible in supporting multimodal searches, where visual and textual information can be jointly leveraged in retrieval. The core idea is to encode image feature vectors into a collection of string tokens in a way such that closer vectors will share more string tokens in common. By doing that, we can utilize Elasticsearch to efficiently retrieve similar images based on similarities within encoded sting tokens. As part of the development, we propose a novel vector to string encoding method, which is shown to substantially outperform the previous ones in terms of both precision and latency. First-hand experiences in implementing this Elasticsearch-based platform are extensively addressed, which should be valuable to practitioners also interested in building visual search engine on top of Elasticsearch.

Motivation & Objective

• To enable efficient content-based image retrieval within Elasticsearch, a system originally designed for textual search.
• To overcome Elasticsearch's limitation in handling high-dimensional vector similarity search without requiring in-memory storage of feature vectors.
• To develop a scalable, cost-effective, and production-ready visual search solution compatible with existing e-commerce infrastructure.
• To support multimodal search by natively integrating visual and textual queries within a single index.
• To demonstrate that vector-to-string encoding can achieve high precision and low latency for real-world e-commerce applications.

Proposed method

• Image features are extracted using deep learning models and embedded into high-dimensional Euclidean space.
• A novel vector-to-string encoding method maps image feature vectors into string tokens such that similar vectors share more common tokens.
• Encoded string tokens are indexed into Elasticsearch using its inverted index, enabling fast retrieval of candidate images based on token overlap.
• At query time, the query vector is encoded into string tokens, and a candidate set is retrieved via Elasticsearch's token-based search.
• The candidate set is re-ranked using exact Euclidean distance to the query vector for final precision.
• The system supports multimodal search by jointly indexing textual product metadata and visual feature tokens in a single Elasticsearch index.

Experimental results

Research questions

• RQ1 Can a visual search system be effectively built on top of Elasticsearch, a system designed for textual search?
• RQ2 How can high-dimensional image feature vectors be efficiently encoded into string tokens to enable similarity search using Elasticsearch's inverted index?
• RQ3 What is the trade-off between precision and latency in vector-to-string encoding for visual search in e-commerce?
• RQ4 Can the system support seamless multimodal search combining visual and textual queries?
• RQ5 How does the proposed encoding method compare to prior approaches in terms of retrieval accuracy and performance?

Key findings

• The proposed vector-to-string encoding method significantly outperforms prior approaches in both precision and latency, achieving up to 98.20% Precision@24 with sub-0.5s average latency.
• Settings with 256 centroids and 64 subvectors achieved 96.06% Precision@24 in under 0.5 seconds, meeting the key performance threshold.
• The system achieved 98.97% Precision@24 at 0.7970s latency with 256 centroids and 256 subvectors, demonstrating strong scalability.
• The method enables cost-effective deployment by avoiding the need to load millions of high-dimensional vectors into RAM.
• The system supports seamless multimodal search by indexing both visual and textual data in a single Elasticsearch index.
• The approach enables production deployment that is easily distributed, scaled, and monitored in real-world e-commerce environments.