[Paper Review] Recent Advance in Content-based Image Retrieval: A Literature Survey

Wengang Zhou, Houqiang Li+1 more|arXiv (Cornell University)|Jun 19, 2017

Advanced Image and Video Retrieval TechniquesComputer Science169 references160 citations

TL;DR

This survey categorizes and evaluates CBIR techniques from 2003 to 2016, detailing image representation, indexing, and ranking modules, and outlines future research directions.

ABSTRACT

The explosive increase and ubiquitous accessibility of visual data on the Web have led to the prosperity of research activity in image search or retrieval. With the ignorance of visual content as a ranking clue, methods with text search techniques for visual retrieval may suffer inconsistency between the text words and visual content. Content-based image retrieval (CBIR), which makes use of the representation of visual content to identify relevant images, has attracted sustained attention in recent two decades. Such a problem is challenging due to the intention gap and the semantic gap problems. Numerous techniques have been developed for content-based image retrieval in the last decade. The purpose of this paper is to categorize and evaluate those algorithms proposed during the period of 2003 to 2016. We conclude with several promising directions for future research.

Motivation & Objective

• Summarize the CBIR pipeline and its five core modules: query formation, image representation, database indexing, image scoring, and search reranking.
• Evaluate key techniques for each module, including feature extraction, visual codebooks, spatial context, and quantization.
• Compare hand-crafted and learning-based features and their impact on scalability and retrieval accuracy.
• Discuss ground-truth datasets, evaluation metrics, and practical industrial CBIR systems to identify strengths and limitations.

Proposed method

• Review the five-module CBIR framework and off-line/on-line stages.
• Discuss feature extraction options: hand-crafted local/global features and learning-based features (including CNNs, patch-based and region-based representations).
• Explain visual codebook learning, including k-means, hierarchical k-means, ANN-based quantization, and alternative hashing/virtual vocabularies.
• Describe spatial context embedding and how visual phrases and spatial pyramid concepts enhance discriminability.
• Outline feature quantization strategies, including hard vs. soft quantization and scalable search structures for large-scale databases.

Experimental results

Research questions

• RQ1 What are effective query formation schemes (e.g., query by example, sketch, color map, text-based structured queries) in CBIR?
• RQ2 How should images be represented and encoded to balance accuracy and scalability in large-scale databases?
• RQ3 What indexing and similarity measurement techniques best support efficient and accurate CBIR at scale?
• RQ4 How can reranking and semi-structured search improve final retrieval results and address semantic gaps?
• RQ5 What datasets and evaluation metrics best reflect progress and guide future CBIR research?

Key findings

• SIFT-based local features and Bag-of-Visual-Words were foundational for scalable CBIR, enabling robust matching and indexing.
• Learning-based features, especially deep CNNs, have emerged as powerful representations for retrieval, with approaches ranging from global descriptors to region-based and patch-level features.
• Spatial context and visual phrases significantly improve discriminative capability beyond traditional bag-of-words representations.
• Efficient indexing and approximate nearest neighbor search (e.g., KD-trees, vocab trees, FLANN) are essential for scalable large-scale CBIR systems, often complemented by hashing and binary codes.
• There is a clear shift toward end-to-end or task-tailored deep models (e.g., CNN fine-tuning for landmarks, region-based deep descriptors, and hashing-based retrieval) to better bridge the semantic gap.
• Commercial CBIR systems (e.g., TinEye, Ditto) demonstrate the practicality and scale of CBIR approaches in real-world deployments.