[Paper Review] Analyzing Web Search Behavior for Software Engineering Tasks

Nikitha Rao, Chetan Bansal+3 more|arXiv (Cornell University)|Dec 19, 2019

Web Data Mining and Analysis38 references4 citations

TL;DR

This paper presents the first large-scale analysis of web search behavior for software engineering tasks using Bing search logs, employing distant supervision to classify SE-related queries with 93% F1 score. It identifies six core search intents—API, Debug, HowTo, Learn, Installation, and Navigational—and reveals that SE queries are less effective, require more effort, and form over 2.6% of total web search traffic, highlighting the need for specialized search tools for developers.

ABSTRACT

Web search plays an integral role in software engineering (SE) to help with various tasks such as finding documentation, debugging, installation, etc. In this work, we present the first large-scale analysis of web search behavior for SE tasks using the search query logs from Bing, a commercial web search engine. First, we use distant supervision techniques to build a machine learning classifier to extract the SE search queries with an F1 score of 93%. We then perform an analysis on one million search sessions to understand how software engineering related queries and sessions differ from other queries and sessions. Subsequently, we propose a taxonomy of intents to identify the various contexts in which web search is used in software engineering. Lastly, we analyze millions of SE queries to understand the distribution, search metrics and trends across these SE search intents. Our analysis shows that SE related queries form a significant portion of the overall web search traffic. Additionally, we found that there are six major intent categories for which web search is used in software engineering. The techniques and insights can not only help improve existing tools but can also inspire the development of new tools that aid in finding information for SE related tasks.

Motivation & Objective

• To understand how software engineering (SE) related web searches differ from general web searches in behavior and effectiveness.
• To develop a scalable method for identifying SE-related search queries at scale despite lack of labeled data.
• To propose a comprehensive taxonomy of user search intents in SE tasks to guide tool and search engine design.
• To analyze the distribution, popularity, and trends of SE search intents across metrics like success rate, effort, and device types.
• To provide actionable insights for improving developer productivity through better search tools and personalized experiences.

Proposed method

• Apply distant supervision with heuristics and seed keywords to train a machine learning classifier that distinguishes SE-related queries from non-SE queries, achieving 93% F1 score.
• Analyze one million search sessions from Bing logs to compare SE and non-SE query behavior in terms of reformulation, click-through, and dwell time.
• Propose a taxonomy of seven search intent categories: API, Debug, HowTo, Learn, Installation, Navigational, and Miscellaneous, based on intent analysis of SE queries.
• Develop a heuristic-based model to automatically classify SE queries into the proposed intent categories using query patterns and keywords.
• Conduct large-scale analysis on six million SE queries to study intent distribution across popularity, success rate, effort estimation, device types, and temporal trends.
• Use temporal, device-level, and session-based analysis to uncover behavioral patterns in SE search usage across different contexts.

Experimental results

Research questions

• RQ1 How do software engineering-related search queries and sessions differ in behavior and effectiveness from general web search queries?
• RQ2 What are the dominant user intents behind web searches for software engineering tasks?
• RQ3 How do search metrics such as popularity, success rate, and effort vary across different SE search intents?
• RQ4 How do SE search patterns vary across different devices and over time of day?
• RQ5 To what extent do SE-related queries constitute a significant portion of overall web search traffic?

Key findings

• SE-related queries account for over 2.6% of all web search sessions, indicating a substantial and under-recognized segment of web search traffic.
• SE-related queries are less effective than general queries, with higher rates of query reformulation, fewer clicks, and shorter dwell times, indicating user frustration and task difficulty.
• The six primary search intents—API, Debug, HowTo, Learn, Installation, and Navigational—account for the majority of SE search behavior, with distinct patterns across time and devices.
• Debug, HowTo, API, and Installation queries are more prevalent on desktop devices, while Learn and Navigational queries are more common on mobile devices.
• Temporal trends show that HowTo and API queries peak during the day, while Installation and Navigational queries increase steadily and peak at night, suggesting different task timing preferences.
• The study demonstrates that existing general web search engines are suboptimal for SE tasks, underscoring the need for specialized search engines and context-aware tools for developers.