[Paper Review] From A Glance to "Gotcha": Interactive Facial Image Retrieval with Progressive Relevance Feedback

Xinru Yang, Haozhi Qi+2 more|arXiv (Cornell University)|Jul 30, 2020

Image Retrieval and Classification Techniques30 references4 citations

TL;DR

This paper proposes an end-to-end interactive facial image retrieval framework that uses progressive relevance feedback to help users iteratively refine their mental image of a suspect using only attribute-level feedback. By progressively disclosing feedback and leveraging deep features from SE-ResNet on CelebA, the system achieves 98.66% ranking percentile, outperforming full disclosure settings and demonstrating strong performance with minimal annotation cost.

ABSTRACT

Facial image retrieval plays a significant role in forensic investigations where an untrained witness tries to identify a suspect from a massive pool of images. However, due to the difficulties in describing human facial appearances verbally and directly, people naturally tend to depict by referring to well-known existing images and comparing specific areas of faces with them and it is also challenging to provide complete comparison at each time. Therefore, we propose an end-to-end framework to retrieve facial images with relevance feedback progressively provided by the witness, enabling an exploitation of history information during multiple rounds and an interactive and iterative approach to retrieving the mental image. With no need of any extra annotations, our model can be applied at the cost of a little response effort. We experiment on exttt{CelebA} and evaluate the performance by ranking percentile and achieve 99\% under the best setting. Since this topic remains little explored to the best of our knowledge, we hope our work can serve as a stepping stone for further research.

Motivation & Objective

• To address the challenge of facial image retrieval when the target is only a mental image in the user’s mind, especially in forensic contexts.
• To reduce reliance on costly manual annotations by enabling instant, attribute-based relevance feedback without prior image labeling.
• To model human-like cognitive behavior in facial comparison by progressively revealing feedback over multiple rounds.
• To improve retrieval accuracy by integrating image features and attribute representations in a supervised, interactive learning framework.

Proposed method

• The system employs a deep learning-based framework that fuses image embeddings from SE-ResNet with attribute vectors to represent facial images.
• It uses a progressive disclosure mechanism where relevance feedback is gradually revealed across rounds, starting with masked feedback and increasing transparency over time.
• Relevance feedback is collected in real-time after each candidate image is shown, with users indicating differences using attribute-level comparisons (e.g., 'with glasses' vs. 'without').
• The model is trained end-to-end using a contrastive loss that optimizes for ranking the target image higher in the retrieval list.
• Feature extraction is performed using SE-ResNet pre-trained on VGGFace2 and fine-tuned on CelebA to enhance attribute-aware representation learning.

Experimental results

Research questions

• RQ1 Can progressive disclosure of relevance feedback improve facial image retrieval performance compared to full disclosure?
• RQ2 How does the integration of attribute-level feedback with deep image features affect retrieval accuracy in an interactive setting?
• RQ3 Does the proposed framework achieve high performance without requiring manual annotations on the dataset?
• RQ4 To what extent does the model’s performance reflect human-like cognitive progression in facial comparison?

Key findings

• The progressive disclosure mechanism achieved the best performance, with a final ranking percentile of 98.66% on the CelebA dataset.
• The model achieved a test loss of 0.0297 at round 5 under progressive disclosure, significantly lower than 0.0589 under full disclosure with attributes.
• Using SE-ResNet features fine-tuned on CelebA yielded the highest performance (98.66% percentile), outperforming OpenFace (87.02%) and SE-ResNet on VGGFace2 (95.80%).
• Despite a drop in matched attributes from 37 to 34 between rounds 3 and 4, the candidate image at round 4 was visually a better match, indicating that feature fusion improves performance beyond attribute count.
• The system converged earlier for male targets (often by round 3), while female targets allowed for later refinement, likely due to greater facial diversity in the dataset.