[论文解读] Exploring Microtask Crowdsourcing as a Means of Fault Localization

C. Adriano, André van der Hoek|arXiv (Cornell University)|Dec 9, 2016

Mobile Crowdsensing and Crowdsourcing参考文献 83被引用 3

一句话总结

本文研究了微任务众包作为一种新型软件工程故障定位方法。通过 Amazon Mechanical Turk，作者将微任务分发给工人，由其评估特定代码片段是否包含缺陷，表明众包能够以高精度有效区分有缺陷与无缺陷的代码段，尽管在覆盖范围和成本方面存在局限。

ABSTRACT

Microtask crowdsourcing is the practice of breaking down an overarching task to be performed into numerous, small, and quick microtasks that are distributed to an unknown, large set of workers. Microtask crowdsourcing has shown potential in other disciplines, but with only a handful of approaches explored to date in software engineering, its potential in our field remains unclear. In this paper, we explore how microtask crowdsourcing might serve as a means of fault localization. We particularly take a first step in assessing whether a crowd of workers can correctly locate known faults in a few lines of code (code fragments) taken from different open source projects. Through Mechanical Turk, we collected the answers of hundreds of workers to a pre-determined set of template questions applied to the code fragments, with a replication factor of twenty answers per question. Our findings show that a crowd can correctly distinguish questions that cover lines of code that contain a fault from those that do not. We also show that various filters can be applied to identify the most effective subcrowds. Our findings also presented serious limitations in terms of the proportion of lines of code selected for inspection and the cost to collect answers. We describe the design of our experiment, discuss the results, and provide an extensive analysis of different filters and their effects in terms of speed, cost, and effectiveness. We conclude with a discussion of limitations and possible future experiments toward more full-fledged fault localization on a large scale involving more complex faults.

研究动机与目标

• 评估微任务众包是否能有效定位小型代码片段中的故障。
• 评估利用大规模匿名众包区分有缺陷与无缺陷代码行的可行性。
• 识别有效的过滤策略，以提高众包在故障定位中的准确性和效率。
• 分析在大规模故障定位中使用众包时，成本、速度与效果之间的权衡。

提出的方法

• 作者从开源项目中提取了包含已知缺陷的小型代码片段。
• 设计了标准化的模板问题，用于评估给定代码片段是否包含缺陷。
• 每个问题通过 Amazon Mechanical Turk 提交 20 次，以收集每个问题的 20 个工人响应。
• 通过在响应中采用多数投票法计算众包共识，以确定代码片段是否被视为有缺陷。
• 应用了多种过滤技术，基于响应质量和一致性识别出表现优异的子众包。
• 研究根据速度、成本和在识别正确故障位置方面的有效性来评估过滤策略。

实验结果

研究问题

• RQ1 非专家工人是否能通过微任务准确识别小型代码片段是否包含缺陷？
• RQ2 不同过滤策略在提升众包故障定位的准确性和效率方面有多有效？
• RQ3 在使用微任务众包进行故障定位时，成本、速度与准确率之间的权衡如何？
• RQ4 与无缺陷代码段相比，工人正确识别有缺陷代码段的程度如何？
• RQ5 基于共识的众包响应聚合能否可靠地区分有缺陷与无缺陷的代码段？

主要发现

• 众包在区分有缺陷与无缺陷代码片段方面表现出高准确率，共识响应在相当大比例的案例中正确识别了缺陷。
• 基于响应一致性和可靠性的工人过滤策略提高了故障定位的准确性，且未显著增加成本。
• 尽管在识别缺陷方面准确率较高，但仅有一小部分代码行被选中进行检查，表明代码覆盖范围有限。
• 由于每个问题需要 20 个响应，导致每个问题的成本居高不下，凸显了可扩展性挑战。
• 通过性能指标过滤出的子众包表现出更高的有效性，表明质量控制机制可提升结果。
• 研究发现，尽管微任务众包在故障定位中具有可行性，但在实际部署中仍受成本和低代码覆盖范围的限制。