[Paper Review] Smelling out Code Clones: Clone Detection Tool Evaluation and Corresponding Challenges

Rachel Gauci|arXiv (Cornell University)|Mar 2, 2015

Software Engineering Research4 references3 citations

TL;DR

This paper evaluates clone detection tools, identifying challenges in tool evaluation such as lack of reference corpora, configuration sensitivity, and flawed benchmarking. It introduces EvaClone, a framework that automatically determines optimal tool configurations using a fitness function, improving recall and precision by up to 21.9% on Java systems and 10.6% on C systems, thus enhancing the validity of comparative studies.

ABSTRACT

Software clones have been an active area of research for the past two decades. However, although numerous clone detection tools are now available, only a small fraction of the literature has focused on tool evaluation, and this is in fact still an open problem. This is mostly due to the fact that standard information retrieval metrics such as recall and precision require a priori knowledge of clones already in the system. Detection tools also typically have a large number of parameters which are difficult to fine-tune for optimal performance on a particular software system, and different outputs produced by different tools add to the complexity of comparing one tool to another. In this review, we further explore the reasons why tool evaluation is still an open challenge, and present the current tools and frameworks targeted at mitigating these problems, focusing on the current standard benchmarks used to evaluate modern clone detection tools, and also presenting a recent method aimed at finding optimal tool configurations.

Motivation & Objective

• To investigate the persistent challenges in evaluating clone detection tools, particularly due to lack of reference corpora and configuration sensitivity.
• To assess the limitations of existing benchmarks like Bellon’s Framework and the Mutation & Injection Framework in ensuring valid tool comparisons.
• To propose and evaluate EvaClone, a framework that automates the identification of optimal tool configurations to improve evaluation reliability.
• To demonstrate that default configurations in tool evaluation significantly degrade performance, and that optimal tuning can substantially improve recall and precision.
• To highlight the need for standardized, representative benchmarks and fitness functions that balance recall and precision in tool evaluation.

Proposed method

• Uses Bellon’s Framework and the Mutation & Injection Framework as standard benchmarks to generate reference corpora for clone detection evaluation.
• Applies a fitness function in EvaClone that maximizes agreement between tools to identify optimal configuration parameters for clone detection tools.
• Employs a systematic approach to tune tool parameters across multiple software systems (e.g., psql, swing) to improve detection performance.
• Compares tool performance using default vs. optimized configurations to quantify the impact of configuration tuning on recall and precision.
• Analyzes the validity threats of existing evaluation methods, including over-reliance on default settings and fitness functions that favor recall over precision.
• Validates results on open-source systems, though acknowledges potential limitations in generalizability to proprietary or non-open-source systems.

Experimental results

Research questions

• RQ1 Why is tool evaluation in clone detection still considered an open challenge despite the availability of over 70 detection tools?
• RQ2 To what extent do default configuration settings in clone detection tools affect recall and precision, and how can this be mitigated?
• RQ3 How effective is the EvaClone framework in identifying optimal tool configurations for improved detection performance?
• RQ4 What are the key limitations of existing benchmarks like Bellon’s Framework and the Mutation & Injection Framework in clone detection evaluation?
• RQ5 Can a fitness function that maximizes tool agreement lead to biased evaluation results, particularly by inflating recall at the expense of precision?

Key findings

• Default configurations in clone detection tools significantly reduce evaluation validity, as shown by performance improvements of up to 21.9% in recall and 10.6% in precision when using optimized settings.
• The fitness function in EvaClone, which maximizes agreement between tools, tends to favor recall over precision, potentially inflating false positive rates.
• Only two of the 14 tools in Bellon et al.’s study were submitted with tuned configurations, highlighting the widespread use of default settings despite their suboptimal performance.
• Empirical studies using default configurations are vulnerable to threats to validity, as demonstrated by the substantial performance gains observed when using optimized configurations.
• The results from Svajlenko et al.’s study on modern tools would likely be significantly altered if evaluated with optimal configurations instead of defaults.
• Despite advances, existing benchmarks and evaluation frameworks still suffer from limitations such as reliance on open-source systems and lack of generalizability to industrial software.