[论文解读] Traceability Transformed: Generating more Accurate Links with Pre-Trained BERT Models

Jinfeng Lin, Yalin Liu+3 more|arXiv (Cornell University)|Feb 8, 2021

Software Engineering Research被引用 7

一句话总结

本文提出 T-BERT，一种新颖的框架，利用预训练的 BERT 模型在开源项目中更准确地生成软件工件（如缺陷和提交记录）之间的追溯链接。通过从相关代码搜索任务迁移知识，T-BERT 克服了数据稀疏性问题，在与传统 VSM 模型相比时，平均 MAP 提升达 60.31%。Single-BERT 架构表现最佳，而 Siamese-BERT 提供了一种快速且接近等效的替代方案。

ABSTRACT

Software traceability establishes and leverages associations between diverse development artifacts. Researchers have proposed the use of deep learning trace models to link natural language artifacts, such as requirements and issue descriptions, to source code; however, their effectiveness has been restricted by availability of labeled data and efficiency at runtime. In this study, we propose a novel framework called Trace BERT (T-BERT) to generate trace links between source code and natural language artifacts. To address data sparsity, we leverage a three-step training strategy to enable trace models to transfer knowledge from a closely related Software Engineering challenge, which has a rich dataset, to produce trace links with much higher accuracy than has previously been achieved. We then apply the T-BERT framework to recover links between issues and commits in Open Source Projects. We comparatively evaluated accuracy and efficiency of three BERT architectures. Results show that a Single-BERT architecture generated the most accurate links, while a Siamese-BERT architecture produced comparable results with significantly less execution time. Furthermore, by learning and transferring knowledge, all three models in the framework outperform classical IR trace models. On the three evaluated real-word OSS projects, the best T-BERT stably outperformed the VSM model with average improvements of 60.31% measured using Mean Average Precision (MAP). RNN severely underperformed on these projects due to insufficient training data, while T-BERT overcame this problem by using pretrained language models and transfer learning.

研究动机与目标

• 解决使用深度学习进行自动化软件追溯时存在的准确率低和数据稀疏性问题。
• 实现从相关且数据丰富的任务（代码搜索）到低数据场景下追溯链接生成的有效迁移学习。
• 开发一种可扩展且高效的框架，用于工业规模的追溯，其性能优于传统的信息检索（IR）和基于 RNN 的模型。
• 评估不同 BERT 架构（Single、Siamese、Twin）在问题-提交追溯上下文中的性能与效率权衡。

提出的方法

• 在大规模代码搜索数据集上微调预训练的 BERT 模型，以学习通用的代码与自然语言表示，随后将其迁移至追溯任务。
• 采用三步训练策略：在代码搜索任务上预训练，在追溯数据上微调，然后通过极少的再训练适应新项目。
• 使用 Single-BERT 架构分别编码问题描述和代码变更集，再通过点积或余弦相似度计算相似度。
• 实现 Siamese-BERT 架构，通过共享权重对两类工件进行编码，从而在保持高准确率的同时实现更快的推理速度。
• 采用对比学习结合负样本采样（ONS）策略，在不显著增加计算开销的前提下提升模型收敛速度与性能。
• 在分类头中使用余弦嵌入损失（Cosine Embedding Loss），以对齐相关工件的语义表示，提升链接预测效果。

Figure 1 : An example commit message and code change set, where green lines have been added and red ones removed. The commit was tagged by the committer to the depicted issue.

实验结果

研究问题

• RQ1 从相关且数据丰富的任务（代码搜索）进行迁移学习，是否能显著提升低数据场景下软件追溯链接的准确性？
• RQ2 在问题-提交追溯任务中，不同 BERT 架构（Single、Siamese、Twin）在准确率与推理效率方面如何比较？
• RQ3 使用预训练语言模型并结合迁移学习，是否能有效克服限制 RNN 基于模型和传统 IR 模型性能的数据稀疏性问题？
• RQ4 经过中间训练的 T-BERT 模型是否能有效适应新开源项目而无需完全重新训练？

主要发现

• Single-BERT 架构在准确率上表现最佳，在三个真实开源项目中，平均 MAP 相较于经典 VSM 模型提升了 60.31%。
• Siamese-BERT 架构在准确率上与 Single-BERT 接近，但执行时间显著减少，使其在大规模部署中更具实用性。
• 通过迁移学习训练的 T-BERT 模型在相同数据集上优于 VSM 和基于 RNN 的模型（TraceNN），证明了预训练语言模型在弥合语义鸿沟方面的有效性。
• ONS（负样本采样）训练策略在不显著增加计算开销的前提下，提升了模型收敛速度与性能。
• 尽管训练数据有限，T-BERT 仍实现了稳定且高水平的性能，表明迁移学习能有效缓解追溯任务中的数据稀疏性问题。
• 该框架展现出强大的泛化能力，中间训练的 T-BERT 模型可无需完全重新训练即高效适配新项目，显著降低时间和资源成本。

Figure 2 : A three step workflow applies T-BERT to NLA-PLA traceability. 1) Pre-training data are functions collected from Github projects 2) A BERT is trained as a language model for code with these functions and composed with a relation classifier as the T-BERT model 3) Functions are split as spec