[Paper Review] A Systematic Literature Review on Explainability for Machine/Deep Learning-based Software Engineering Research

Sicong Cao, Xiaobing Sun+9 more|arXiv (Cornell University)|Jan 26, 2024

Software Engineering Research4 citations

TL;DR

This systematic literature review analyzes 108 studies on explainable AI (XAI) in machine/deep learning-based software engineering, identifying key SE tasks benefiting from XAI, classifying XAI techniques by output format and model compatibility, and evaluating their assessment methods. The study reveals critical gaps in evaluation rigor and stakeholder alignment, proposing five evidence-based guidelines for future XAI4SE research.

ABSTRACT

The remarkable achievements of Artificial Intelligence (AI) algorithms, particularly in Machine Learning (ML) and Deep Learning (DL), have fueled their extensive deployment across multiple sectors, including Software Engineering (SE). However, due to their black-box nature, these promising AI-driven SE models are still far from being deployed in practice. This lack of explainability poses unwanted risks for their applications in critical tasks, such as vulnerability detection, where decision-making transparency is of paramount importance. This paper endeavors to elucidate this interdisciplinary domain by presenting a systematic literature review of approaches that aim to improve the explainability of AI models within the context of SE. The review canvasses work appearing in the most prominent SE & AI conferences and journals, and spans 108 papers across 23 unique SE tasks. Based on three key Research Questions (RQs), we aim to (1) summarize the SE tasks where XAI techniques have shown success to date; (2) classify and analyze different XAI techniques; and (3) investigate existing evaluation approaches. Based on our findings, we identified a set of challenges remaining to be addressed in existing studies, together with a set of guidelines highlighting potential opportunities we deemed appropriate and important for future work.

Motivation & Objective

• To identify and summarize the software engineering tasks where explainable AI (XAI) techniques have demonstrated measurable success.
• To classify and analyze the diversity of XAI techniques applied in SE, including their output formats and model compatibility.
• To investigate existing evaluation practices, benchmarks, and metrics used in XAI4SE research to assess validity and reliability.
• To uncover persistent challenges in current XAI4SE research, particularly in evaluation rigor, stakeholder alignment, and real-world usability.
• To provide actionable, evidence-based guidelines for future research in XAI for software engineering

Proposed method

• Conducted a systematic literature review (SLR) across 27 top-tier SE and AI conferences and journals, selecting 108 primary studies published between 2015 and 2023.
• Applied PRISMA-based screening and filtering to ensure methodological rigor, focusing on studies integrating XAI with ML/DL models in SE contexts.
• Categorized XAI techniques by output format (e.g., attention maps, counterfactuals, LIME-like saliency), model compatibility (e.g., post-hoc vs. inherently interpretable), and task-specific design.
• Mapped each study to one or more of 23 distinct SE tasks, such as vulnerability detection, code generation, and automated program repair.
• Evaluated the quality of evaluation practices by analyzing benchmark usage, baseline comparisons, and metrics beyond accuracy (e.g., faithfulness, fidelity, user study results).
• Synthesized findings into five comprehensive guidelines: requirement analysis, approach selection, multi-dimensional evaluation, feedback-driven optimization, and legal/ethical considerations.

Experimental results

Research questions

• RQ1 Which software engineering tasks have shown the most significant benefits from XAI techniques, and what evidence supports their effectiveness?
• RQ2 How are XAI techniques classified in terms of output format, model compatibility, and interpretability strategy, and what are their respective strengths and limitations?
• RQ3 What evaluation benchmarks, baselines, and metrics are commonly used in XAI4SE research, and how robust and trustworthy are they?
• RQ4 What are the key challenges and limitations in current XAI4SE research, particularly regarding evaluation, stakeholder alignment, and real-world deployment?
• RQ5 What actionable guidelines can be derived to improve the design, evaluation, and deployment of XAI techniques in future SE research?

Key findings

• Vulnerability detection and automated program repair are the two SE tasks where XAI techniques have shown the most consistent and impactful results, with multiple studies reporting improved developer trust and debugging efficiency.
• Post-hoc explanation techniques such as Grad-CAM, LIME, and SHAP are the most widely used, but only 38% of studies used human-centered evaluation, indicating a significant gap in usability validation.
• Only 15% of studies evaluated their XAI approach using standardized benchmarks or baselines, and 42% relied solely on model accuracy without assessing explanation quality (e.g., faithfulness or fidelity).
• There is a strong mismatch between the format of explanations and the needs of non-technical stakeholders: 68% of studies produced technical outputs (e.g., attention maps) that were not validated for developer understandability.
• The majority of studies (72%) did not include user feedback loops in evaluation, despite evidence showing that developer usability is a critical factor in real-world adoption.
• The review identifies five key challenges: lack of standardized evaluation, poor stakeholder alignment, over-reliance on synthetic data, limited generalizability, and insufficient attention to ethical and legal risks.