[Paper Review] Is AI different for SE

Amritanshu Agrawal, Tim Menzies|arXiv (Cornell University)|Dec 9, 2019

Software Engineering Research61 references4 citations

TL;DR

This paper investigates whether AI tools designed for software engineering (SE) differ fundamentally from general-purpose AI tools. By testing a SE-optimized tool across 120 datasets—including SE-specific tasks like defect prediction and bug report analysis—it finds that SE tools outperform standard AI methods on rare, infrequent issues (e.g., edge-case bugs), but fail on common AI benchmarks. The key contribution is a simple rule: use SE-specific tools when dealing with rare events, and avoid them when targets are frequent, highlighting the need for AI tools tailored to SE’s unique data characteristics.

ABSTRACT

What AI tools are needed for SE? Ideally, we should have simple rules that peek at data, then say use this or use that To find such a rule, we explored 120 different data sets addressing numerous problems, including bad smell detection, predicting Github issue close time, bug report analysis, defect prediction and dozens of other non-SE problems. To this data, we apply a SE-based tool that (a)~out-performs the state-of-the-art for these SE problems yet (b)~fails very badly on standard AI problems. In those results, we can find a simple rule for when to use/avoid the SE-based tool. SE data is often about infrequent issues, like the occasional defect, or the rarely exploited security violation, or the requirement that holds for one special case. But as we show, standard AI tools work best when the target is relatively more frequent. Also, we can exploit these special properties of SE, to great effect (to rapidly find better optimizations for SE tasks via a tactic called dodging, explained in this paper). More generally, this result says we need a new kind of SE research for developing new AI tools that are more suited to SE problems.

Motivation & Objective

• To determine whether AI tools designed for software engineering (SE) differ significantly from general-purpose AI tools.
• To identify the conditions under which SE-specific tools outperform or underperform compared to standard AI approaches.
• To uncover the underlying data characteristics that make SE problems distinct from typical AI problems.
• To develop a simple decision rule for selecting between SE-based and general AI tools based on data properties.
• To advocate for a new research direction in AI focused on SE-specific tooling for rare-event detection.

Proposed method

• Evaluated a SE-optimized tool across 120 diverse datasets, including SE tasks (e.g., defect prediction, issue close time prediction) and non-SE problems.
• Compared the SE tool’s performance against state-of-the-art AI models on both SE and non-SE datasets.
• Identified that the SE tool excels on tasks with infrequent target events (e.g., rare defects, edge-case violations), but fails on standard AI benchmarks with more frequent targets.
• Proposed a decision rule: use SE-based tools when the target event is rare; avoid them when the target is relatively frequent.
• Introduced a novel optimization technique called 'dodging' to rapidly improve SE tool performance by exploiting the unique properties of SE data.
• Analyzed data characteristics such as class imbalance and event rarity to explain the performance gap between SE and general AI tools.

Experimental results

Research questions

• RQ1 Under what conditions does a SE-optimized AI tool outperform general-purpose AI models?
• RQ2 How do data characteristics like event frequency and class imbalance affect the performance of SE-specific versus general AI tools?
• RQ3 Can a simple rule be derived to determine when to use SE-based tools instead of standard AI approaches?
• RQ4 What properties of SE data make it fundamentally different from typical AI training data?
• RQ5 How can SE-specific data properties be exploited to improve tool performance, as demonstrated by the 'dodging' technique?

Key findings

• The SE-based tool significantly outperforms state-of-the-art AI models on SE-specific tasks such as defect prediction and bug report analysis.
• Despite its strong performance on SE problems, the SE tool performs poorly on standard AI benchmarks, indicating a mismatch with typical AI data distributions.
• The key differentiator is event frequency: SE tools excel on rare events (e.g., occasional defects), while standard AI tools perform better on more frequent targets.
• A simple rule emerges: use SE-based tools when the target event is rare; avoid them when the target is relatively common.
• The 'dodging' optimization tactic effectively improves SE tool performance by leveraging the unique properties of SE data, such as infrequent but critical issues.
• The results imply that general-purpose AI tools are ill-suited for many SE problems due to their reliance on frequent, balanced data, necessitating a new class of SE-tailored AI tools.