[论文解读] Neural Network Embeddings for Test Case Prioritization

João Lousada, Miguel Ribeiro|arXiv (Cornell University)|Dec 18, 2020

Software Testing and Debugging Techniques参考文献 16被引用 6

一句话总结

该论文提出NNE-TCP，一种深度学习框架，通过从历史代码变更和测试状态转移中学习文件-测试用例关系，利用神经网络嵌入来优先排序测试用例。该方法在故障检测速度上优于随机和基于状态转移的方法，同时支持相似文件和测试用例的可视化聚类，有助于减少冗余。

ABSTRACT

In modern software engineering, Continuous Integration (CI) has become an indispensable step towards systematically managing the life cycles of software development. Large companies struggle with keeping the pipeline updated and operational, in useful time, due to the large amount of changes and addition of features, that build on top of each other and have several developers, working on different platforms. Associated with such software changes, there is always a strong component of Testing. As teams and projects grow, exhaustive testing quickly becomes inhibitive, becoming adamant to select the most relevant test cases earlier, without compromising software quality. We have developed a new tool called Neural Network Embeeding for Test Case Prioritization (NNE-TCP) is a novel Machine-Learning (ML) framework that analyses which files were modified when there was a test status transition and learns relationships between these files and tests by mapping them into multidimensional vectors and grouping them by similarity. When new changes are made, tests that are more likely to be linked to the files modified are prioritized, reducing the resources needed to find newly introduced faults. Furthermore, NNE-TCP enables entity visualization in low-dimensional space, allowing for other manners of grouping files and tests by similarity and to reduce redundancies. By applying NNE-TCP, we show for the first time that the connection between modified files and tests is relevant and competitive relative to other traditional methods.

研究动机与目标

• 为解决持续集成流水线中因全面回归测试导致的反馈延迟问题。
• 通过从历史数据中学习文件-测试用例关系，实现测试用例的自动化优先排序。
• 通过预测代码变更后最可能失败的测试，缩短反馈时间并减少资源消耗。
• 基于语义相似性，实现文件和测试用例的可视化聚类，以实现测试精简。
• 建立一种数据驱动、可扩展的替代方案，以取代手动或基于启发式的测试优先排序方法。

提出的方法

• 该框架使用神经网络嵌入，基于历史提交记录和测试状态转移数据，将文件和测试用例映射到共享的低维向量空间。
• 从版本控制系统和测试日志中学习修改文件与测试用例状态转移（失败或进展）之间的关系。
• 通过训练模型，基于文件变更预测在新代码提交后最相关的测试用例。
• 使用t-SNE将嵌入向量可视化为二维空间，以揭示结构模式并聚类相似的文件/测试用例。
• 通过识别冗余或高度相似的测试聚类，支持测试用例的精简。
• 使用APTD（平均测试用例检测百分比）作为主要评估指标对框架进行评估。

实验结果

研究问题

• RQ1 神经网络嵌入能否有效从历史软件变更数据中学习文件-测试用例关系？
• RQ2 NNE-TCP的性能与传统的随机和状态转移方法相比如何？
• RQ3 所学习的嵌入能否揭示代码和测试库中的有意义结构模式？
• RQ4 基于嵌入的聚类在多大程度上能减少测试冗余而不影响测试覆盖率？
• RQ5 该模型能否在不同软件项目和CI环境中实现良好泛化？

主要发现

• NNE-TCP在APTD指标上显著优于随机和基于状态转移的优先排序方法，表明故障检测速度更快。
• 该模型成功在嵌入空间中将相似的文件和测试用例聚类，揭示了潜在的冗余。
• 可视化结果表明文件夹结构与语义相似性之间无直接相关性，说明基于嵌入的聚类比文件系统组织更具意义。
• 该框架支持有效的测试用例聚类，可用于在保持覆盖率的前提下精简测试套件。
• 尽管性能表现强劲，NNE-TCP仍未超越更复杂的基于历史的方法，表明通过特征增强仍有改进空间。
• 该方法证明，通过深度学习学习文件-测试关联是可行且具有竞争力的，为未来研究提供了新的基线。