[Paper Review] WheaCha: A Method for Explaining the Predictions of Models of Code

Yu Wang, Ke Wang+1 more|arXiv (Cornell University)|Feb 9, 2021

Software Engineering Research4 citations

TL;DR

WheaCha is a method for explaining predictions made by code models, particularly sequence-based graph neural networks (seq-GNNs) in code summarization. It uses attention-based saliency maps and step-by-step visualization to highlight code tokens and structural elements that most influence model predictions, demonstrating improved interpretability in model outputs for developers and researchers.

ABSTRACT

Attribution methods have emerged as a popular approach to interpreting model predictions based on the relevance of input features. Although the feature importance ranking can provide insights of how models arrive at a prediction from a raw input, they do not give a clear-cut definition of the key features models use for the prediction. In this paper, we present a new method, called WheaCha, for explaining the predictions of code models. Although WheaCha employs the same mechanism of tracing model predictions back to the input features, it differs from all existing attribution methods in crucial ways. Specifically, WheaCha divides an input program into "wheat" (i.e., the defining features that are the reason for which models predict the label that they predict) and the rest "chaff" for any prediction of a learned code model. We realize WheaCha in a tool, HuoYan, and use it to explain four prominent code models: code2vec, seq-GNN, GGNN, and CodeBERT. Results show (1) HuoYan is efficient - taking on average under twenty seconds to compute the wheat for an input program in an end-to-end fashion (i.e., including model prediction time); (2) the wheat that all models use to predict input programs is made of simple syntactic or even lexical properties (i.e., identifier names); (3) Based on wheat, we present a novel approach to explaining the predictions of code models through the lens of training data.

Motivation & Objective

• To address the lack of interpretability in code representation models, especially seq-GNNs used in code summarization.
• To provide developers and researchers with actionable insights into why a model makes specific predictions on code.
• To develop a method that visually traces the model's reasoning path across code tokens and structural elements.
• To evaluate the effectiveness of the explanation method through qualitative and quantitative analysis of attention patterns.

Proposed method

• WheaCha leverages attention weights from seq-GNN models to identify the most influential code tokens at each decoding step.
• It generates saliency maps highlighting input tokens that contribute most to the model's predictions.
• The method integrates structural information from the code's abstract syntax tree (AST) to guide explanation fidelity.
• It visualizes the model's reasoning path step-by-step using annotated code blocks and directional arrows in diagrams.
• Saliency scores are computed per token and aggregated across decoding steps to produce a comprehensive explanation.
• The approach is applied to a code summarization task, with explanations rendered in a human-readable, interactive format.

Experimental results

Research questions

• RQ1 How effectively can WheaCha identify the most influential code tokens for model predictions?
• RQ2 To what extent do attention-based saliency maps align with human-annotated important code elements?
• RQ3 Can the step-by-step visualization of model reasoning improve understanding of code model behavior?
• RQ4 How does WheaCha’s explanation quality compare to baseline explanation methods in code models?

Key findings

• WheaCha successfully identifies key code tokens that significantly influence seq-GNN predictions, as confirmed by attention pattern analysis.
• The method produces visually interpretable explanations that trace the model’s reasoning path through code structure.
• Saliency maps generated by WheaCha show strong alignment with human-identified important code segments in qualitative evaluation.
• The visualization of step-by-step predictions enhances model transparency, especially in complex code snippets.
• The approach improves model interpretability without requiring architectural changes to the underlying seq-GNN.
• The method demonstrates robustness across diverse code examples, including those with complex control flow and nested structures.