[Paper Review] CoLadder: Supporting Programmers with Hierarchical Code Generation in Multi-Level Abstraction

Ryan Yen, Jiawen Zhu+3 more|arXiv (Cornell University)|Oct 12, 2023

Software Engineering Research4 citations

TL;DR

CoLadder is an interactive system that supports programmers in hierarchical code generation by enabling structured task decomposition, modular prompt blocks, and direct code manipulation during prompt authoring. A user study with 12 experienced programmers showed it improves intention externalization and reduces code editing, especially in early stages, by aligning prompts with mental models across abstraction levels.

ABSTRACT

Programmers increasingly rely on Large Language Models (LLMs) for code generation. However, misalignment between programmers' goals and generated code complicates the code evaluation process and demands frequent switching between prompt authoring and code evaluation. Yet, current LLM-driven code assistants lack sufficient scaffolding to help programmers format intentions from their overarching goals, a crucial step before translating these intentions into natural language prompts. To address this gap, we adopted an iterative design process to gain insights into programmers' strategies when using LLMs for programming. Building on our findings, we created CoLadder, a system that supports programmers by facilitating hierarchical task decomposition, direct code segment manipulation, and result evaluation during prompt authoring. A user study with 12 experienced programmers showed that CoLadder is effective in helping programmers externalize their problem-solving intentions flexibly, improving their ability to evaluate and modify code across various abstraction levels, from goal to final code implementation.

Motivation & Objective

• To address the gap in scaffolding for intention formation and externalization when using LLMs for programming.
• To reduce the cognitive burden caused by misalignment between programmer goals and generated code.
• To support flexible, multi-level abstraction in prompt authoring for improved code generation and evaluation.
• To investigate how hierarchical task decomposition supports better control and comprehension in LLM-assisted programming workflows.

Proposed method

• CoLadder employs a tree-based editor to allow programmers to hierarchically decompose tasks into modular prompt blocks corresponding to specific code segments.
• Each prompt block is generated via LLMs and maps directly to a code segment, enabling structured, multi-level abstraction.
• The system supports block-based code operations, allowing direct manipulation of generated code segments without full re-prompting.
• It integrates real-time result evaluation during prompt authoring to support iterative refinement and alignment with intentions.
• An iterative design process informed the system based on observations of programmer strategies with LLMs.
• User studies with 12 experienced programmers evaluated CoLadder’s impact on intention externalization, code control, and comprehension.

Figure 1. CoLadder enables programmers to flexibly decompose tasks, aligning with their mental models for solving programming tasks using LLM-driven code assistants (A). The system provides a tree-based editor that allows programmers to hierarchically express their intent through smaller, modular-ba

Experimental results

Research questions

• RQ1 RQ1: How do programmers form and externalize their intentions when using LLMs for code generation?
• RQ2 RQ2: To what extent does CoLadder improve programmers’ control over the prompt-to-code generation process?
• RQ3 RQ3: How does CoLadder affect programmers’ ability to evaluate and modify code across abstraction levels?
• RQ4 RQ4: What is the impact of CoLadder on programmer comprehension and code maintainability?

Key findings

• Participants required significantly less code editing when using CoLadder, particularly in early stages, due to block-based code manipulation features.
• CoLadder enhanced programmers’ ability to externalize and refine intentions hierarchically, aligning with their mental models of problem-solving.
• The system improved control over the intention scaffolding process (RQ2), though not necessarily over the final program code itself.
• Participants demonstrated stronger program comprehension in recall tests, suggesting deeper understanding of code structure when using CoLadder.
• Over-reliance on LLM suggestions was reduced, as participants engaged in deliberate, step-by-step program construction.
• Task familiarity had only a weak to moderate correlation with perceived workload and satisfaction, indicating that experience outweighs prior task-specific knowledge in system effectiveness.

Figure 2. We have adapted Norman’s seven stages of action to illustrate cognitive processes when interacting with LLM-driven systems. This model covers programmer intention formation (E) and externalization (F) within the gulf of execution , while the gulf of evaluation evaluates alignment between g