[Paper Review] Semantic Image Search for Robotic Applications

Tomas Kulvičius, Irene Markelić+2 more|arXiv (Cornell University)|Apr 2, 2020

Multimodal Machine Learning Applications3 citations

TL;DR

This paper proposes SIMSEA, a bi-modal semantic image search method that combines textual linguistic cues with visual features to improve image search precision and resolve polysemy. By pairing a target object with context-specific descriptors (e.g., 'drinking' for 'glass'), it reduces irrelevant results from standard text-based searches, achieving higher precision than Google search in most cases, especially for ambiguous terms.

ABSTRACT

Generalization in robotics is one of the most important problems. New generalization approaches use internet databases in order to solve new tasks. Modern search engines can return a large amount of information according to a query within milliseconds. However, not all of the returned information is task relevant, partly due to the problem of polysemes. Here we specifically address the problem of object generalization by using image search. We suggest a bi-modal solution, combining visual and textual information, based on the observation that humans use additional linguistic cues to demarcate intended word meaning. We evaluate the quality of our approach by comparing it to human labelled data and find that, on average, our approach leads to improved results in comparison to Google searches, and that it can treat the problem of polysemes.

Motivation & Objective

• Address the challenge of polysemy in internet-based image search, which hinders robotic generalization.
• Improve precision of image search results for robotic applications by reducing irrelevant results from ambiguous queries.
• Develop a method that mimics human linguistic refinement to disambiguate word meanings in image search.
• Enable automatic cleaning of image search results using visual consistency and linguistic context.
• Provide a scalable, semi-automated approach to generate high-quality training data for object recognition in robotics.

Proposed method

• Conduct multiple image searches using a base term paired with context-specific linguistic cues (e.g., 'drinking glass' instead of 'glass').
• Extract visual features using PHOW (Pyramid Histogram of Words) to represent images for comparison.
• Compute inter-image similarity using Hellinger distance between PHOW feature vectors.
• Rank candidate images based on their frequency of appearance across multiple subsearches, assuming higher co-occurrence indicates greater relevance.
• Use the ranking score as a proxy for human relevance, filtering out images that appear inconsistently across subsearches.
• Apply the method to real-world categories (e.g., 'apple', 'glass', 'milk') and compare results to human-labeled relevance and standard Google search.

Experimental results

Research questions

• RQ1 Can combining linguistic cues with visual features improve the precision of image search for robotic applications?
• RQ2 To what extent can this method resolve polysemy in image search compared to standard text-based search?
• RQ3 How well does the proposed ranking mechanism reflect human-based relevance judgments?
• RQ4 Does the method perform consistently across different object categories, especially those with high ambiguity?
• RQ5 Can this approach automatically generate cleaner training data for object recognition in robotics?

Key findings

• SIMSEA outperforms standard Google image search in precision for most categories, particularly for ambiguous terms like 'glass' and 'apple'.
• For the 'glass' category, SIMSEA successfully filtered out 42% of irrelevant results (e.g., vision aids, material) present in Google’s initial results.
• The method achieved a precision of approximately 0.7 across most human subjects, with only one outlier (TP1) skewing the average.
• The 'milk' category showed lower performance due to high human disagreement, indicating inherent ambiguity in visual depiction of liquids.
• The ranking based on subsearch co-occurrence frequency correlated well with human relevance judgments, validating its use as a proxy for relevance.
• The approach effectively reduces polysemous results by leveraging linguistic context, demonstrating its strength in disambiguation tasks.