[论文解读] CGEMs: A Metric Model for Automatic Code Generation using GPT-3

Aishwarya Narasimhan, Krishna Prasad Agara Venkatesha Rao+1 more|arXiv (Cornell University)|Aug 23, 2021

Explainable Artificial Intelligence (XAI)参考文献 7被引用 6

一句话总结

该论文提出CGEMs，一种通过编译检查、自然语言到逻辑的转换、编辑次数、静态代码度量以及基于自然语言处理的特征组合来评估GPT-3生成代码的度量模型。基于这些度量训练的神经网络分类器在区分可接受与不可接受代码时，达到76.92%的准确率和55.56%的F1分数，且通过XAI技术增强了可解释性。

ABSTRACT

Today, AI technology is showing its strengths in almost every industry and walks of life. From text generation, text summarization, chatbots, NLP is being used widely. One such paradigm is automatic code generation. An AI could be generating anything; hence the output space is unconstrained. A self-driving car is driven for 100 million miles to validate its safety, but tests cannot be written to monitor and cover an unconstrained space. One of the solutions to validate AI-generated content is to constrain the problem and convert it from abstract to realistic, and this can be accomplished by either validating the unconstrained algorithm using theoretical proofs or by using Monte-Carlo simulation methods. In this case, we use the latter approach to test/validate a statistically significant number of samples. This hypothesis of validating the AI-generated code is the main motive of this work and to know if AI-generated code is reliable, a metric model CGEMs is proposed. This is an extremely challenging task as programs can have different logic with different naming conventions, but the metrics must capture the structure and logic of the program. This is similar to the importance grammar carries in AI-based text generation, Q&A, translations, etc. The various metrics that are garnered in this work to support the evaluation of generated code are as follows: Compilation, NL description to logic conversion, number of edits needed, some of the commonly used static-code metrics and NLP metrics. These metrics are applied to 80 codes generated using OpenAI's GPT-3. Post which a Neural network is designed for binary classification (acceptable/not acceptable quality of the generated code). The inputs to this network are the values of the features obtained from the metrics. The model achieves a classification accuracy of 76.92% and an F1 score of 55.56%. XAI is augmented for model interpretability.

研究动机与目标

• 为解决在非约束输出空间中验证AI生成代码的挑战。
• 开发一种可靠、自动化的基于度量的评估框架，用于评估类似GPT-3的大语言模型生成的代码。
• 识别并量化反映代码质量的关键特征，如编译成功、逻辑保真度、编辑次数和代码结构。
• 设计一个二分类模型，基于可度量的特征预测GPT-3生成的代码是否可接受。
• 通过可解释人工智能（XAI）技术增强模型的可解释性，以提高代码质量评估的透明度。

提出的方法

• 该框架收集了80个GPT-3生成的代码样本，并应用一系列评估度量：编译成功率、自然语言到逻辑转换的准确性、所需编辑次数、标准静态代码度量（如圈复杂度），以及基于自然语言处理的度量。
• 每个代码样本在这些度量上进行评分，形成模型输入的特征向量。
• 使用基于度量特征的输入，训练一个前馈神经网络进行二分类（可接受与不可接受代码）。
• 通过优化分类性能来训练模型，评估指标包括准确率和F1分数。
• 应用XAI技术以解释模型的预测结果，并识别对分类结果影响最大的特征。

实验结果

研究问题

• RQ1 编译、静态代码和基于自然语言处理的度量组合是否能有效评估GPT-3生成代码的质量？
• RQ2 基于这些度量，神经网络分类器在区分可接受与不可接受代码方面表现如何？
• RQ3 哪些单个度量对代码质量分类的贡献最为显著？
• RQ4 XAI技术在多大程度上能提升代码质量评估模型的可解释性？
• RQ5 所提出的度量模型是否足够稳健，能够应对生成代码中逻辑和命名规范的差异？

主要发现

• CGEMs模型在80个GPT-3生成代码样本的测试集上实现了76.92%的分类准确率。
• 二分类任务的F1得分为55.56%，表明精确率与召回率之间保持了中等平衡。
• 编译成功和编辑次数成为代码质量的强指标，分别反映了功能正确性和可维护性。
• 基于自然语言处理的度量，特别是衡量自然语言提示与代码逻辑对齐程度的度量，显著提升了模型的预测能力。
• XAI分析表明，逻辑保真度和代码结构度量是模型决策过程中最具影响力的特征。
• 结果表明，基于度量和数据驱动的方法可为AI生成代码提供统计上有效且可解释的评估方式。