[Paper Review] DeepSeek: Content Based Image Search & Retrieval.

Tanya Piplani, David Bamman|arXiv (Cornell University)|Jan 9, 2018

Multimodal Machine Learning Applications7 references3 citations

TL;DR

This paper proposes DeepSeek, a content-based image retrieval system that uses natural language queries to retrieve semantically relevant images. It employs two methods: caption-based retrieval using fine-tuned ResNet-101 and skip-thought vectors, and end-to-end embedding space retrieval via learned image-text embeddings; the latter achieves higher precision at top-5 retrieval (91.2%) despite lower top-1 precision (68.3%), demonstrating the effectiveness of joint embedding learning for semantic image search.

ABSTRACT

Most of the internet today is composed of digital media that includes videos and images. With pixels becoming the currency in which most transactions happen on the internet, it is becoming increasingly important to have a way of browsing through this ocean of information with relative ease. YouTube has 400 hours of video uploaded every minute and many million images are browsed on Instagram, Facebook, etc. Inspired by recent advances in the field of deep learning and success that it has gained on various problems like image captioning and, machine translation , word2vec , skip thoughts, etc, we present DeepSeek a natural language processing based deep learning model that allows users to enter a description of the kind of images that they want to search, and in response the system retrieves all the images that semantically and contextually relate to the query. Two approaches are described in the following sections.

Motivation & Objective

• Address the limitations of existing image search systems that rely on image-based queries, limited keyword matching, or metadata, which often fail to capture semantic meaning.
• Enable accurate and efficient content-based image retrieval using natural language descriptions as input queries.
• Overcome the fundamental flaw in image-based retrieval by allowing users to search with descriptive text instead of requiring a similar image.
• Develop a unified embedding space where both image and text features are semantically aligned for improved retrieval performance.

Proposed method

• Utilize a fine-tuned ResNet-101 backbone pretrained on MS-COCO object detection to extract deep convolutional features from images.
• Generate image captions using a Long Short-Term Memory (LSTM)-based language model trained on the MS-COCO dataset to produce semantically rich textual descriptions.
• Encode generated captions and user queries into dense vector representations using the skip-thought model to enable semantic similarity computation.
• Perform image retrieval by minimizing L2 distance between the query embedding and caption embeddings of images in the dataset.
• Implement an end-to-end embedding space retrieval method where images and their corresponding captions are jointly embedded into a shared semantic space using a Siamese-like architecture.
• Optimize retrieval using a contrastive loss objective to pull positive image-text pairs closer and push negative pairs apart in the embedding space.

Experimental results

Research questions

• RQ1 Can a text-based query system outperform traditional image-based or keyword-based image retrieval in terms of semantic relevance and user expressivity?
• RQ2 How effective is the skip-thought vector model in capturing semantic meaning from image captions for downstream retrieval tasks?
• RQ3 Does joint embedding of images and captions into a shared space improve retrieval accuracy compared to caption-based retrieval alone?
• RQ4 To what extent does the choice of retrieval metric (e.g., precision@1 vs. precision@5) reflect the true performance of semantic image retrieval models?

Key findings

• The embedding space retrieval (ESR) method achieved a precision@5 of 91.2%, outperforming caption-based retrieval (CBR) in top-5 accuracy despite lower top-1 precision.
• Caption-based retrieval achieved a higher precision@1 (72.9%) than embedding-based retrieval (68.3%), indicating better initial ranking accuracy for the top result.
• The skip-thought model effectively encoded image captions into meaningful vector representations, enabling high semantic similarity between semantically related queries and images.
• The system demonstrated strong qualitative performance, correctly retrieving images that matched both object presence and semantic relationships (e.g., 'a dog watching a television').
• The retrieval pipeline was not GPU-optimized, and timing results (1.7–2.1 seconds per query) suggest significant performance gains are possible with hardware acceleration.
• The model’s performance was quantitatively evaluated on the MS-COCO dataset using standard metrics (BLEU, METEOR, ROUGE-L, CIDEr), showing competitive results compared to state-of-the-art caption generation models.