[Paper Review] Automatic Programming: Large Language Models and Beyond

Michael R. Lyu, Baishakhi Ray+3 more|arXiv (Cornell University)|May 3, 2024

Topic Modeling4 citations

TL;DR

This paper investigates the challenges and future of automatic programming using Large Language Models (LLMs), proposing that automated program repair and program analysis can enhance trust and correctness in LLM-generated code. It advocates for LLM agents augmented with verification and repair capabilities to produce higher-assurance code, shifting the programmer's role toward ensuring trust and autonomy in software development by 2030.

ABSTRACT

Automatic programming has seen increasing popularity due to the emergence of tools like GitHub Copilot which rely on Large Language Models (LLMs). At the same time, automatically generated code faces challenges during deployment due to concerns around quality and trust. In this article, we study automated coding in a general sense and study the concerns around code quality, security and related issues of programmer responsibility. These are key issues for organizations while deciding on the usage of automatically generated code. We discuss how advances in software engineering such as program repair and analysis can enable automatic programming. We conclude with a forward looking view, focusing on the programming environment of the near future, where programmers may need to switch to different roles to fully utilize the power of automatic programming. Automated repair of automatically generated programs from LLMs, can help produce higher assurance code from LLMs, along with evidence of assurance

Motivation & Objective

• To examine the trust, correctness, and security challenges in LLM-generated code for real-world software projects.
• To investigate how program repair and analysis can improve the reliability and quality of automatically generated code.
• To explore the evolving role of software engineers in a future where LLMs automate much of the coding process.
• To propose a vision for autonomous, self-improving code generation using LLMs combined with formal verification and testing.
• To examine the feasibility of extending automated repair to probabilistic programs and other non-traditional code types.

Proposed method

• Proposes using automated program repair techniques to fix LLM-generated code, guided by curated test suites.
• Introduces the concept of generating additional test inputs $T'$ from natural language descriptions to serve as evidence of correctness for repaired code.
• Envisions LLM agents augmented with program analysis and repair capabilities to autonomously improve code quality and generate verifiable commits.
• Suggests integrating LLM-generated code via sanitizer code or containment mechanisms to limit potential harm in legacy systems.
• Proposes generating both code and formal proofs from natural language specifications, leveraging recent advances in code-to-specification and proof generation.
• Explores the extension of automated repair to probabilistic programs using symbolic execution and semantics-aware repair.

Figure 2. Inferring Specifications for Program Repair (ack. unpublished article ( Gao et al . , 2023a ) )

Experimental results

Research questions

• RQ1 What criteria should determine the acceptability of LLM-generated code in production software systems?
• RQ2 How can automated program repair improve the correctness and trustworthiness of LLM-generated code?
• RQ3 In what ways can LLM agents with program analysis capabilities achieve autonomous improvement of generated code?
• RQ4 How can formal verification and test generation be integrated into the LLM code generation pipeline to ensure higher assurance?
• RQ5 What are the implications of integrating LLM-generated code with manually written code, particularly regarding trust boundaries and component compatibility?

Key findings

• Automated repair of LLM-generated code using curated test suites can serve as concrete evidence of correctness, increasing confidence in integration.
• LLM agents augmented with program analysis and repair capabilities can autonomously improve code quality, moving toward self-improving software generation.
• The integration of LLM-generated code into existing systems requires safeguards such as sanitizers or containment to limit potential failure propagation.
• Formal verification and proof generation from natural language descriptions are emerging as viable paths toward higher-assurance AI-generated code.
• The feasibility of repairing probabilistic programs using symbolic execution opens new avenues for automated improvement in machine learning workloads.
• The future role of software engineers is expected to shift from large-scale coding to ensuring trust, correctness, and autonomy of AI-generated artifacts.

Figure 5. Evolution of programmer roles captured by DALL-E; programmer role (a) as code composer and designer instead of code writer, (b) as quality assurance specialist instead of code writer.