[Paper Review] A Systematic Literature Review of Automated Query Reformulations in Source Code Search.

Mohammad Masudur Rahman, Chanchal K. Roy|arXiv (Cornell University)|Aug 22, 2021

Software Engineering Research170 references4 citations

TL;DR

This systematic literature review analyzes 70 primary studies on automated query reformulation in source code search, identifying eight key methodologies—such as term weighting and thesaurus lookup—while revealing major limitations like vocabulary mismatch and weak evaluation. The study highlights critical gaps in generalizability and developer burden, offering future research directions to improve code search efficiency for developers.

ABSTRACT

Software developers often fix critical bugs to ensure the reliability of their software. They might also need to add new features to their software at a regular interval to stay competitive in the market. These bugs and features are reported as change requests (i.e., technical documents written by software users). Developers consult these documents to implement the required changes in the software code. As a part of change implementation, they often choose a few important keywords from a change request as an ad hoc query. Then they execute the query with a code search engine (e.g., Lucene) and attempt to find out the exact locations within the software code that need to be changed. Unfortunately, even experienced developers often fail to choose the right queries. As a consequence, the developers often experience difficulties in detecting the appropriate locations within the code and spend the majority of their time in numerous trials and errors. There have been many studies that attempt to support developers in constructing queries by automatically reformulating their ad hoc queries. In this systematic literature review, we carefully select 70 primary studies on query reformulations from 2,970 candidate studies, perform an in-depth qualitative analysis using the Grounded Theory approach, and then answer six important research questions. Our investigation has reported several major findings. First, to date, eight major methodologies (e.g., term weighting, query-term co-occurrence analysis, thesaurus lookup) have been adopted in query reformulation. Second, the existing studies suffer from several major limitations (e.g., lack of generalizability, vocabulary mismatch problem, weak evaluation, the extra burden on the developers) that might prevent their wide adoption. Finally, we discuss several open issues in search query reformulations and suggest multiple future research opportunities.

Motivation & Objective

• To address the challenge of developers struggling to formulate effective ad hoc queries for source code search, leading to time-consuming trial-and-error processes.
• To identify and categorize the methodologies used in automated query reformulation to improve search precision and recall in software maintenance tasks.
• To evaluate the limitations of existing approaches, including vocabulary mismatch, weak evaluation practices, and lack of generalizability.
• To identify open issues and propose future research opportunities in automated query reformulation for source code search.

Proposed method

• Conducted a systematic literature review of 2,970 candidate studies, selecting 70 primary studies based on predefined inclusion and exclusion criteria.
• Applied the Grounded Theory approach to perform an in-depth qualitative analysis of the selected studies to identify recurring themes and methodologies.
• Categorized query reformulation techniques into eight major methodologies: term weighting, query-term co-occurrence analysis, thesaurus lookup, and others.
• Evaluated the quality and rigor of existing studies based on evaluation practices, including test datasets, metrics, and reproducibility.
• Synthesized findings to identify recurring limitations and research gaps in current approaches to query reformulation.

Experimental results

Research questions

• RQ1 What are the dominant methodologies used in automated query reformulation for source code search?
• RQ2 What are the key limitations of existing query reformulation approaches in real-world software development contexts?
• RQ3 How do current studies evaluate the effectiveness of query reformulation techniques, and what are the shortcomings in their evaluation strategies?
• RQ4 What are the open issues and challenges that hinder the adoption of query reformulation systems by developers?
• RQ5 What future research directions can improve the generalizability and practical utility of automated query reformulation in code search?

Key findings

• Eight major methodologies—such as term weighting, query-term co-occurrence analysis, and thesaurus lookup—have been adopted in query reformulation research.
• Existing studies suffer from significant limitations, including poor generalizability across different codebases and programming languages.
• The vocabulary mismatch problem remains a persistent challenge, where query terms do not align with relevant codebase terms.
• Evaluation practices in the studies are often weak, with insufficient use of standardized benchmarks, reproducible results, or real-world validation.
• Many approaches place an extra burden on developers, such as requiring manual configuration or domain-specific tuning, reducing practical adoption.
• Despite progress, no single methodology consistently outperforms others across diverse software systems, indicating a need for more robust and adaptive solutions.