[论文解读] Large Language Models Based Automatic Synthesis of Software Specifications

Shantanu Mandal, Adhrik Chethan+6 more|arXiv (Cornell University)|Apr 18, 2023

Software Engineering Research被引用 4

一句话总结

该论文提出SpecSyn，一种新颖的框架，利用基于BERT的序列到序列学习方法，从手册、注释和论坛等非结构化自然语言源中自动合成软件规格说明。其F1得分相比先前最先进工具提升了21%，标志着首个端到端的大语言模型（LLM）方法用于此任务。

ABSTRACT

Software configurations play a crucial role in determining the behavior of software systems. In order to ensure safe and error-free operation, it is necessary to identify the correct configuration, along with their valid bounds and rules, which are commonly referred to as software specifications. As software systems grow in complexity and scale, the number of configurations and associated specifications required to ensure the correct operation can become large and prohibitively difficult to manipulate manually. Due to the fast pace of software development, it is often the case that correct software specifications are not thoroughly checked or validated within the software itself. Rather, they are frequently discussed and documented in a variety of external sources, including software manuals, code comments, and online discussion forums. Therefore, it is hard for the system administrator to know the correct specifications of configurations due to the lack of clarity, organization, and a centralized unified source to look at. To address this challenge, we propose SpecSyn a framework that leverages a state-of-the-art large language model to automatically synthesize software specifications from natural language sources. Our approach formulates software specification synthesis as a sequence-to-sequence learning problem and investigates the extraction of specifications from large contextual texts. This is the first work that uses a large language model for end-to-end specification synthesis from natural language texts. Empirical results demonstrate that our system outperforms prior the state-of-the-art specification synthesis tool by 21% in terms of F1 score and can find specifications from single as well as multiple sentences.

研究动机与目标

• 解决复杂软件系统中手动提取配置规格说明的挑战。
• 通过从非结构化源自动推导出有效的配置规则，减少因配置错误导致的系统故障。
• 开发一个端到端框架，直接从自然语言文本中合成规格说明，无需依赖基于规则或语法解析的方法。
• 通过利用大语言模型的上下文理解能力，超越现有规格说明提取工具。
• 为系统管理员提供一个可靠、集中的配置规则源，这些规则来自多样化的文档资料。

提出的方法

• 将软件规格说明合成建模为基于微调的BERT编码器-解码器架构的序列到序列学习问题。
• 在从软件手册、代码注释和在线论坛中提取的配置描述精选数据集上训练模型。
• 采用数据组合技术以增强泛化能力，将多个句子组合为单个输入序列，用于复杂规格说明的提取。
• 采用两阶段方法：单句提取用于简单规则，双句组合用于依赖上下文的复杂规格说明。
• 通过BERT的预训练权重应用迁移学习，以捕捉配置相关自然语言中的语义和句法模式。
• 使用交叉熵损失优化模型，并在保留的测试集上使用F1得分评估性能。

Figure 1 . Overview of SpecSyn

实验结果

研究问题

• RQ1 大语言模型能否有效从非结构化自然语言文本中提取软件配置规格说明？
• RQ2 基于大语言模型的方法在规格说明提取中的性能与传统基于规则或语法解析的方法相比如何？
• RQ3 序列到序列模型在手册、代码注释和讨论论坛等多样化来源上的泛化能力如何？
• RQ4 输入序列长度和组合方式对规格说明提取准确率有何影响？
• RQ5 模型能否处理从单个或多个句子中提取的简单和复杂规格说明？

主要发现

• 与先前最先进规格说明合成工具相比，SpecSyn的F1得分高出21%，显示出显著的性能提升。
• 该模型成功从单句和多句上下文中提取规格说明，其中双句输入覆盖了大多数复杂规格说明场景。
• 数据组合技术提升了模型的泛化能力，使其在训练数据有限的情况下表现更优。
• 基于BERT的微调使用有效提升了对配置相关自然语言的理解能力，即使在模糊或依赖上下文的表达中亦然。
• 当输入序列超过两句话时，模型性能显著下降，表明未来工作需引入长上下文建模能力。
• 实证结果证实，基于大语言模型的方法在规格说明提取准确率上优于传统语法和基于规则的方法。

Figure 2 . SpecSyn model architecture