[Paper Review] What Looks Good with my Sofa: Multimodal Search Engine for Interior Design

Ivona Tautkutė, Aleksandra Możejko+4 more|arXiv (Cornell University)|Jul 21, 2017

Advanced Image and Video Retrieval Techniques29 references4 citations

TL;DR

This paper proposes a multimodal interior design search engine that combines visual and textual queries to retrieve furniture and decor items with high stylistic and aesthetic similarity to a user's sofa or room. By blending visual search (using YOLO-9000 and deep features) with text-based retrieval (using word2vec-like embeddings), the system improves average style similarity by 11% over baseline methods.

ABSTRACT

In this paper, we propose a multi-modal search engine for interior design that combines visual and textual queries. The goal of our engine is to retrieve interior objects, e.g. furniture or wall clocks, that share visual and aesthetic similarities with the query. Our search engine allows the user to take a photo of a room and retrieve with a high recall a list of items identical or visually similar to those present in the photo. Additionally, it allows to return other items that aesthetically and stylistically fit well together. To achieve this goal, our system blends the results obtained using textual and visual modalities. Thanks to this blending strategy, we increase the average style similarity score of the retrieved items by 11%. Our work is implemented as a Web-based application and it is planned to be opened to the public.

Motivation & Objective

• To address the limitation of purely visual or textual search in interior design, where neither captures stylistic and contextual fit well.
• To develop a system that retrieves not only visually similar items but also those that aesthetically and stylistically complement the query object.
• To improve retrieval quality by fusing visual and textual modalities using a novel blending strategy in feature space.
• To implement and publicly release a working web-based prototype of the Style Search Engine for real-world use.

Proposed method

• The system uses YOLO-9000 for object detection in user-uploaded room images to localize key items like sofas.
• Visual search retrieves items based on deep neural network features (e.g. normalized activations from a pre-trained CNN) in a feature space.
• Textual queries are processed using embeddings that capture semantic and stylistic co-occurrence patterns, similar to word2vec, to find stylistically compatible items.
• A feature similarity blending method re-ranks results by computing distances in visual and textual feature spaces, combining them into a unified ranked list.
• The blending strategy uses weighted similarity scores from both modalities to improve overall stylistic relevance.
• The system is implemented as a Flask-based web application, allowing users to upload images or select from a dataset.

Experimental results

Research questions

• RQ1 Can combining visual and textual modalities improve the retrieval of aesthetically and stylistically compatible interior design items compared to unimodal approaches?
• RQ2 How effective is a feature-space blending strategy in enhancing the stylistic similarity of retrieved results?
• RQ3 To what extent does adding textual context (e.g. 'white', 'Scandinavian') improve retrieval quality beyond visual search alone?
• RQ4 Can a probabilistic co-occurrence metric based on real room datasets effectively quantify stylistic compatibility between items?
• RQ5 How does the performance of the multimodal system compare to vanilla visual and text-only search in terms of style similarity?

Key findings

• The proposed feature similarity blending method increased the average style similarity score of retrieved results by 11% compared to vanilla visual search.
• Vanilla visual search alone achieved a mean style similarity score of 0.2295, while the blended method outperformed this baseline.
• Textual search alone achieved a slightly lower score of 0.2243, indicating that text alone is less effective than visual search for this task.
• Using the detected object class (e.g. 'sofa') as a text query already improved results over visual-only search, showing the value of textual context.
• Adding additional descriptors like 'white' or 'decorative' further enhanced performance, demonstrating the benefit of richer textual input.
• The web application prototype successfully demonstrated the system’s functionality and is planned for public release after user testing.