[Paper Review] Constructing an Interaction Behavior Model for Web Image Search

Xiaohui Xie, Jiaxin Mao+4 more|arXiv (Cornell University)|May 8, 2018

Advanced Image and Video Retrieval Techniques34 references3 citations

TL;DR

This paper proposes the Grid-based User Browsing Model (GUBM), an unsupervised interaction behavior model for web image search that leverages cursor hovering and click data to estimate user examination behavior and result relevance. GUBM outperforms state-of-the-art models in behavior prediction and ranking effectiveness, demonstrating that hover interactions are strong indicators of relevance and can significantly improve topical relevance and image quality metrics.

ABSTRACT

User interaction behavior is a valuable source of implicit relevance feedback. In Web image search a different type of search result presentation is used than in general Web search, which leads to different interaction mechanisms and user behavior. For example, image search results are self-contained, so that users do not need to click the results to view the landing page as in general Web search, which generates sparse click data. Also, two-dimensional result placement instead of a linear result list makes browsing behaviors more complex. Thus, it is hard to apply standard user behavior models (e.g., click models) developed for general Web search to Web image search. In this paper, we conduct a comprehensive image search user behavior analysis using data from a lab-based user study as well as data from a commercial search log. We then propose a novel interaction behavior model, called grid-based user browsing model (GUBM), whose design is motivated by observations from our data analysis. GUBM can both capture users' interaction behavior, including cursor hovering, and alleviate position bias. The advantages of GUBM are two-fold: (1) It is based on an unsupervised learning method and does not need manually annotated data for training. (2) It is based on user interaction features on search engine result pages (SERPs) and is easily transferable to other scenarios that have a grid-based interface such as video search engines. We conduct extensive experiments to test the performance of our model using a large-scale commercial image search log. Experimental results show that in terms of behavior prediction (perplexity), and topical relevance and image quality (normalized discounted cumulative gain (NDCG)), GUBM outperforms state-of-the-art baseline models as well as the original ranking. We make the implementation of GUBM and related datasets publicly available for future studies.

Motivation & Objective

• To understand user interaction patterns in web image search, particularly the role of cursor hovering beyond clicks.
• To address the limitations of existing click models in image search due to sparse clicks and non-linear result layouts.
• To develop a behavior model that captures grid-based browsing patterns and alleviates position bias in image search.
• To create a transferable, unsupervised model that uses only observable interaction features and does not require annotated data.
• To improve result ranking by integrating hovering behavior as a proxy for user examination and relevance.

Proposed method

• The model is built on three core assumptions: users examine results in a directional grid traversal, hovering indicates examination, and users may skip results in both vertical and horizontal directions.
• GUBM uses observable interaction sequences—clicks and hovers—on a grid-based SERP to estimate internal parameters such as examination probability and query-image relevance.
• The model employs an unsupervised learning framework that does not require manually labeled relevance data, enabling training on large-scale commercial search logs.
• It models user browsing as a Markov process over grid positions, with transition probabilities informed by interaction frequency and sequence patterns.
• The model computes a likelihood score for interaction sequences and uses it to infer the relevance of image results.
• GUBM is designed to be transferable to other grid-based interfaces, such as video or product search engines.

Experimental results

Research questions

• RQ1 How do user interaction behaviors, particularly hovering, correlate with examination and relevance in web image search?
• RQ2 What are the dominant patterns of user browsing in a grid-based image search interface?
• RQ3 Can hovering be used as a reliable signal for relevance estimation in image search, especially given the sparsity of clicks?
• RQ4 How can user behavior models be adapted to the unique layout and interaction patterns of image search compared to traditional Web search?
• RQ5 To what extent can an unsupervised model using only interaction data outperform supervised baselines in ranking effectiveness?

Key findings

• Hovering over image results is 8 to 10 times more frequent than clicking, indicating it is a rich and underutilized signal for relevance.
• Users tend to examine results in a directional manner, often skipping items both vertically and horizontally, which challenges the sequential examination assumption of traditional click models.
• GUBM significantly outperforms the original ranking and state-of-the-art baseline models in behavior prediction, as measured by perplexity, on a large-scale commercial image search log.
• In terms of relevance evaluation, GUBM improves normalized discounted cumulative gain (NDCG) for both topical relevance and image quality.
• The integration of hover data into the model leads to a measurable performance gain, confirming that hovering is a strong proxy for user examination.
• The model is robust and transferable, with potential applications in other grid-based search interfaces such as video or product search.