[论文解读] Automatic Generation of Acceptance Test Cases from Use Case Specifications: an NLP-based Approach

Chunhui Wang, Fabrizio Pastore+2 more|arXiv (Cornell University)|Jul 19, 2019

Software Engineering Research参考文献 102被引用 12

一句话总结

本文提出UMTG，一种基于NLP的方法，通过语义角色标注和约束挖掘技术，从自然语言用例规范和领域模型中自动生成可执行的系统级验收测试用例，以提取测试场景和条件。在两次工业案例研究中，UMTG在将用例步骤转化为正式约束方面实现了95%的准确率，并发现了此前遗漏的关键测试场景。

ABSTRACT

Acceptance testing is a validation activity performed to ensure the conformance of software systems with respect to their functional requirements. In safety critical systems, it plays a crucial role since it is enforced by software standards. Test engineers need to identify all the representative test execution scenarios from requirements, determine the runtime conditions that trigger these scenarios, and finally provide the input data that satisfy these conditions. Given that requirements specifications are typically large and often provided in natural language, the generation of acceptance test cases tends to be expensive and error-prone. In this paper, we present UMTG, an approach that supports the generation of executable, system-level, acceptance test cases from requirements specifications in natural language, with the goal of reducing the manual effort required to generate test cases and ensuring requirements coverage. More specifically, UMTG automates the generation of acceptance test cases based on use case specifications and a domain model for the system under test, which are commonly produced in many development environments. Unlike existing approaches, it does not impose strong restrictions on the expressiveness of use case specifications. We rely on recent advances in natural language processing to automatically identify test scenarios and to generate formal constraints that capture conditions triggering the execution of the scenarios, thus enabling the generation of test data. In two industrial case studies, UMTG automatically and correctly translated 95% of the use case specification steps into formal constraints required for test data generation; furthermore, it generated test cases that exercise not only all the test scenarios manually implemented by experts, but also some critical scenarios not previously considered.

研究动机与目标

• 通过从自然语言需求自动生��测试用例，减少安全关键系统验收测试中的手动工作量并提高测试覆盖率。
• 解决现有方法对受控自然语言、形式化模型或大量手动建模的依赖问题。
• 在无需额外行为建模的情况下，利用现有的开发工件（如用例规范和类图）自动生成可执行测试用例。
• 通过从自然语言步骤中挖掘正式约束，确保测试覆盖的可追溯性和完整性。

提出的方法

• 使用语义角色标注（SRL）分析用例步骤，提取谓词-论元结构以识别测试场景。
• 应用约束挖掘技术，将自然语言条件转换为正式约束，以支持测试数据生成。
• 构建用例测试模型（UCTM）作为有向图，表示执行路径，包括循环和分支。
• 采用覆盖驱动算法，在测试场景生成过程中最大化子类型和条件覆盖。
• 集成Alloy约束求解器，基于形式化约束生成可执行测试输入。
• 利用领域模型（类图）验证并增强用例元素的语义解释。

实验结果

研究问题

• RQ1 NLP技术能否有效用于从自然语言用例规范中提取可执行测试场景？
• RQ2 在多大程度上能自动从自然语言需求中推导出正式约束，以支持测试数据生成？
• RQ3 该方法在无需受控自然语言或额外建模的情况下，实现全面测试覆盖的有效性如何？
• RQ4 该方法能否在工业环境中发现人工测试设计所遗漏的关键测试场景？

主要发现

• 在两次工业案例研究中，UMTG成功将95%的用例规范步骤转化为测试数据生成所需的正式约束。
• 该方法生成的测试用例覆盖了专家手动实现的所有场景，以及此前被忽略的关键场景。
• 尽管场景生成的理论最坏时间复杂度为O(N³)，但由于实际用例中条件步骤和叶节点数量较少，实际性能仍具可行性。
• 导致循环的条件步骤平均仅占总步骤的18%，叶节点仅占8%，显著降低了计算开销。
• 生成的最大UCTM包含154个节点，表明即使在理论复杂度较高的情况下，实际系统仍保持可管理性。
• 该方法在真实工业系统（BodySense和HOD）中验证了可行性，表明基于NLP的测试生成在工业环境中具有可行性。