[Paper Review] MosAIc: Finding Artistic Connections across Culture with Conditional Image Retrieval

Mark Hamilton, Stephanie Fu+10 more|arXiv (Cornell University)|Jul 14, 2020

Generative Adversarial Networks and Image Synthesis83 references4 citations

TL;DR

MosAIc introduces Conditional Image Retrieval (CIR), a method that combines visual similarity search with user-defined conditions (e.g., culture, media) to discover semantically and visually related artworks across cultures and time periods. The approach uses pruned Random Projection Trees to efficiently retrieve conditionally filtered neighbors, enabling non-parametric style transfer and revealing GAN 'blind spots' in data distribution modeling.

ABSTRACT

We introduce MosAIc, an interactive web app that allows users to find pairs of semantically related artworks that span different cultures, media, and millennia. To create this application, we introduce Conditional Image Retrieval (CIR) which combines visual similarity search with user supplied filters or "conditions". This technique allows one to find pairs of similar images that span distinct subsets of the image corpus. We provide a generic way to adapt existing image retrieval data-structures to this new domain and provide theoretical bounds on our approach's efficiency. To quantify the performance of CIR systems, we introduce new datasets for evaluating CIR methods and show that CIR performs non-parametric style transfer. Finally, we demonstrate that our CIR data-structures can identify "blind spots" in Generative Adversarial Networks (GAN) where they fail to properly model the true data distribution.

Motivation & Objective

• To enable interactive discovery of semantically and visually related artworks across diverse cultures, media, and time periods.
• To address the challenge of efficiently restricting image retrieval to specific subsets (e.g., by culture or medium) without rebuilding index structures.
• To develop a general method for adapting existing KNN data-structures to support conditional queries using arbitrary logical predicates.
• To evaluate CIR performance on new datasets and demonstrate its utility in identifying distributional failures in GANs.
• To create an interactive web application (MosAIc) that visualizes artistic connections across global art collections.

Proposed method

• Introduces Conditional Image Retrieval (CIR), where retrieval is constrained by user-specified conditions such as culture or media.
• Adapts tree-based KNN structures (e.g., Random Projection Trees) by pruning nodes that do not belong to the target condition subset using an inverted index.
• Proves a theoretical lower bound on the number of nodes that can be pruned in Random Projection Trees under conditional queries.
• Employs deep features from pre-trained models (e.g., ResNet50, MobileNetV2) as embeddings for image similarity computation using cosine distance.
• Uses an inverted index to map conditions (e.g., '18th-century Chinese') to relevant image indices, enabling fast filtering during retrieval.
• Applies the resulting Conditional K-Nearest Neighbor (CKNN) structure to detect 'blind spots' in GANs by analyzing neighbor distributions in feature space.

Experimental results

Research questions

• RQ1 Can conditional image retrieval uncover meaningful artistic connections across culturally and temporally distant artworks?
• RQ2 How can existing KNN data-structures be efficiently adapted to support conditional queries without rebuilding for each condition?
• RQ3 To what extent can CIR perform non-parametric style transfer by retrieving content-similar images in the 'style' of a target subset?
• RQ4 Can CKNN structures reveal distributional discrepancies in GAN-generated data, such as 'blind spots' where the model fails to generate realistic samples?
• RQ5 How does the proposed CIR method compare in efficiency and accuracy to baseline approaches like query-then-filter or brute-force search?

Key findings

• CIR successfully retrieves semantically and visually similar artworks from vastly different cultures and time periods, such as a 1980 BC Egyptian boat model and an 18th-century Chinese raft painting.
• The pruned Random Projection Tree approach achieves significant speedups over query-then-filter and brute-force baselines, with measured space efficiency of 6.4 MB for a conditional index on a 1M-image dataset.
• CIR performs non-parametric style transfer by retrieving content-similar images from a target condition subset, demonstrating transfer without model fine-tuning.
• The CKNN data-structure effectively identifies 'blind spots' in ProGAN trained on CelebA HQ, revealing regions where the model fails to generate realistic faces, particularly around brimmed hats.
• The method outperforms baseline strategies in conditional retrieval speed and scalability, especially when conditions are specific or the target subset is small.
• The new datasets introduced for CIR evaluation—FEI Face Database and two large-scale art collections—enable robust benchmarking of conditional retrieval performance.