[Paper Review] CodePlan: Repository-level Coding using LLMs and Planning

Ramakrishna Bairi, Atharv Sonwane+7 more|arXiv (Cornell University)|Sep 21, 2023

Software Engineering ResearchComputer Science3 citations

TL;DR

CodePlan is a planning-based framework that automates complex, repository-wide code modifications using LLMs by orchestrating multi-step edits through incremental dependency and impact analysis. It achieves 5 out of 6 repositories passing validity checks—surpassing baselines that fail on all—by dynamically identifying and propagating necessary changes across interdependent files.

ABSTRACT

Software engineering activities such as package migration, fixing errors reports from static analysis or testing, and adding type annotations or other specifications to a codebase, involve pervasively editing the entire repository of code. We formulate these activities as repository-level coding tasks. Recent tools like GitHub Copilot, which are powered by Large Language Models (LLMs), have succeeded in offering high-quality solutions to localized coding problems. Repository-level coding tasks are more involved and cannot be solved directly using LLMs, since code within a repository is inter-dependent and the entire repository may be too large to fit into the prompt. We frame repository-level coding as a planning problem and present a task-agnostic framework, called CodePlan to solve it. CodePlan synthesizes a multi-step chain of edits (plan), where each step results in a call to an LLM on a code location with context derived from the entire repository, previous code changes and task-specific instructions. CodePlan is based on a novel combination of an incremental dependency analysis, a change may-impact analysis and an adaptive planning algorithm. We evaluate the effectiveness of CodePlan on two repository-level tasks: package migration (C#) and temporal code edits (Python). Each task is evaluated on multiple code repositories, each of which requires inter-dependent changes to many files (between 2-97 files). Coding tasks of this level of complexity have not been automated using LLMs before. Our results show that CodePlan has better match with the ground truth compared to baselines. CodePlan is able to get 5/6 repositories to pass the validity checks (e.g., to build without errors and make correct code edits) whereas the baselines (without planning but with the same type of contextual information as CodePlan) cannot get any of the repositories to pass them.

Motivation & Objective

• To address the challenge of automating large-scale, inter-dependent code changes across entire repositories, which are beyond the scope of current LLM-based tools.
• To overcome the limitations of LLMs in handling repository-wide edits due to context window constraints and code interdependence.
• To design a task-agnostic framework that generates a chain of edits guided by dynamic analysis of code dependencies and change impacts.
• To improve accuracy and reliability in automated code migration and refactoring by integrating planning with LLM generation.

Proposed method

• CodePlan frames repository-level coding as a planning problem, decomposing edits into a sequence of atomic, context-aware LLM calls.
• It performs incremental dependency analysis to track syntactic and semantic relationships across files in the codebase.
• It applies change may-impact analysis to predict which code regions are affected by each edit, generating 'change obligations' for subsequent steps.
• It uses an adaptive planning algorithm to prioritize and sequence edits based on dependency order and impact propagation.
• Each LLM call receives a curated context including the full repository, prior edits, and task-specific instructions to ensure correctness.
• The framework validates each edit incrementally, ensuring syntactic and semantic consistency before proceeding to the next step.

Experimental results

Research questions

• RQ1 Can a planning-based framework effectively automate repository-level coding tasks that require pervasive, inter-dependent changes across multiple files?
• RQ2 How can LLMs be guided to make correct, consistent edits in large codebases when context is too large for a single prompt?
• RQ3 What role does dynamic impact analysis play in reducing errors and improving correctness in multi-step code transformations?
• RQ4 How does adaptive planning improve success rates compared to direct LLM prompting on monolithic code contexts?
• RQ5 To what extent can a task-agnostic framework handle diverse repository-level tasks like API migration and temporal edits?

Key findings

• CodePlan achieved 5 out of 6 repositories passing validity checks (e.g., compilation and correct semantic changes), while baselines with identical contextual information failed on all.
• The framework demonstrated superior alignment with ground truth edits compared to baseline methods, especially in handling transitive dependencies across file boundaries.
• Impact analysis significantly reduced the number of incorrect or redundant edits by identifying affected code regions before LLM intervention.
• The use of incremental dependency tracking enabled accurate propagation of changes through complex call graphs and data dependencies.
• The adaptive planning strategy improved convergence and reduced the number of failed edit sequences compared to non-planning baselines.
• The framework successfully handled both C# and Python repositories with 2 to 97 files requiring coordinated edits, proving scalability and robustness.