[Paper Review] Content-Based Sub-Image Retrieval with Relevance Feedback

Jie Luo, Mário A. Nascimento|ArXiv.org|Apr 26, 2009

Image Retrieval and Classification Techniques17 references3 citations

TL;DR

This paper introduces a novel relevance feedback approach for content-based sub-image retrieval (CBsIR), where the goal is to find images in a database that contain a query image as a sub-region. By modeling images as overlapping tiles and iteratively reweighting tile penalties based on user feedback, the method achieves stable 70% average recall within top 20 results after just 5 iterations, with each iteration taking ~2 seconds on a standard desktop.

ABSTRACT

The typical content-based image retrieval problem is to find images within a database that are similar to a given query image. This paper presents a solution to a different problem, namely that of content based sub-image retrieval, i.e., finding images from a database that contains another image. Note that this is different from finding a region in a (segmented) image that is similar to another image region given as a query. We present a technique for CBsIR that explores relevance feedback, i.e., the user's input on intermediary results, in order to improve retrieval efficiency. Upon modeling images as a set of overlapping and recursive tiles, we use a tile re-weighting scheme that assigns penalties to each tile of the database images and updates the tile penalties for all relevant images retrieved at each iteration using both the relevant and irrelevant images identified by the user. Each tile is modeled by means of its color content using a compact but very efficient method which can, indirectly, capture some notion of texture as well, despite the fact that only color information is maintained. Performance evaluation on a largely heterogeneous dataset of over 10,000 images shows that the system can achieve a stable average recall value of 70% within the top 20 retrieved (and presented) images after only 5 iterations, with each such iteration taking about 2 seconds on an off-the-shelf desktop computer.

Motivation & Objective

• To address the challenge of retrieving images that contain a given query image as a sub-region, rather than retrieving full images similar to the query.
• To improve retrieval effectiveness in sub-image retrieval by incorporating user relevance feedback, which is underexplored in existing literature.
• To develop a scalable and efficient system that models images using overlapping tiles and dynamically adjusts similarity based on user-provided positive and negative examples.
• To evaluate the method on a large, heterogeneous dataset of over 10,000 images to demonstrate robustness and performance.

Proposed method

• Images are modeled as a set of overlapping, recursive tiles to capture spatial and color-based content at multiple scales.
• Each tile is represented using the BIC (Bag of Image Colors) method, which uses compact color histograms to encode color content and indirectly capture texture.
• A tile re-weighting scheme assigns penalties to tiles based on their relevance; penalties are updated iteratively using feedback from the user on retrieved results.
• The system refines image similarity by weighting tile distances with their corresponding penalties, giving more importance to tiles that are consistently relevant.
• User feedback is used to identify both relevant and irrelevant images, which are then used to update tile penalties across all relevant images.
• The process is iterative: after each feedback round, the system re-ranks database images based on the updated similarity measure.

Experimental results

Research questions

• RQ1 Can relevance feedback significantly improve retrieval performance in content-based sub-image retrieval (CBsIR), where the goal is to find images containing a query image as a sub-region?
• RQ2 How does the tile re-weighting mechanism, based on user feedback, enhance the accuracy and convergence of sub-image retrieval?
• RQ3 What is the impact of using different numbers of quantized colors in the BIC representation on retrieval performance and system efficiency?
• RQ4 How does the system maintain efficiency and scalability when processing large image databases with complex content?
• RQ5 To what extent can user feedback reduce the gap between low-level image features and high-level semantic intent in sub-image retrieval?

Key findings

• The system achieves a stable average recall of 70% within the top 20 retrieved images after only 5 iterations of relevance feedback.
• Using 16 quantized colors in the BIC representation reduces query processing time to about 1 second per iteration, compared to 2.5 seconds with 64 colors, while maintaining comparable retrieval effectiveness.
• The method with 16 colors learns more slowly but eventually reaches similar performance levels, while 64-color representation offers faster initial learning but higher computational cost.
• The storage cost for metadata is only 2.7 MB when using 16 colors—about 20% of the image database size—demonstrating efficient storage usage.
• The system processes each query in under 2 seconds per iteration on a standard desktop, making it practical for interactive use.
• The tile re-weighting mechanism effectively captures user intent by penalizing irrelevant tiles and reinforcing relevant ones, significantly improving ranking accuracy over iterations.