[Paper Review] Synthesizing Imperative Programs from Examples Guided by Static Analysis

Sunbeom So, Hakjoo Oh|arXiv (Cornell University)|Feb 21, 2017

Teaching and Learning Programming8 references6 citations

TL;DR

This paper presents a novel algorithm that synthesizes imperative programs from input-output examples and partial code by combining enumerative program synthesis with static analysis to drastically reduce search space. The approach, implemented in the open-source tool Simpl, achieves an average synthesis time of 6.6 seconds across 30 introductory programming problems, outperforming a baseline by 25x with pruning, and successfully handles complex control structures like nested loops.

ABSTRACT

We present a novel algorithm that synthesizes imperative programs for introductory programming courses. Given a set of input-output examples and a partial program, our algorithm generates a complete program that is consistent with every example. Our key idea is to combine enumerative program synthesis and static analysis, which aggressively prunes out a large search space while guaranteeing to find, if any, a correct solution. We have implemented our algorithm in a tool, called SIMPL, and evaluated it on 30 problems used in introductory programming courses. The results show that SIMPL is able to solve the benchmark problems in 6.6 seconds on average.

Motivation & Objective

• To develop an automated system that helps introductory programming students by generating complete programs from incomplete ones and input-output examples.
• To address the scalability and interactivity limitations of existing program synthesis techniques in educational settings.
• To eliminate the need for instructor-provided feedback rules or correct implementations by relying solely on examples and partial code.
• To enable synthesis of complex imperative programs with loops and conditionals from minimal input.
• To design a system that is both efficient and safe, guaranteeing correct solutions when they exist.

Proposed method

• The algorithm performs enumerative search over candidate programs in increasing size, starting from a partial program with holes.
• It integrates static analysis to prune intermediate programs that cannot satisfy any input-output example, based on semantic invariants.
• The method uses state normalization to reduce equivalent program states and avoid redundant exploration.
• It applies a safety-preserving pruning technique grounded in formal semantics, ensuring no correct solution is lost during optimization.
• The system dynamically selects from given resources (variables, constants, arrays) to complete the partial program.
• The implementation, Simpl, is open-sourced and publicly available for educational use.

Experimental results

Research questions

• RQ1 Can static analysis be effectively combined with enumerative synthesis to accelerate program generation in an educational context?
• RQ2 How effective is pruning via static analysis in reducing the search space for imperative program synthesis?
• RQ3 Can the system synthesize complex control structures like nested loops from minimal examples and partial code?
• RQ4 Does the approach scale to real-world introductory programming problems with practical performance?
• RQ5 Can the system generate correct programs without requiring instructor-provided rules or complete solutions?

Key findings

• Simpl synthesizes all 30 benchmark problems in an average of 6.6 seconds, demonstrating high efficiency and interactivity for educational use.
• The static analysis-guided pruning technique reduces average runtime by 25x compared to a baseline with optimization, from 165.5 seconds to 6.6 seconds.
• Without pruning, the baseline algorithm timed out on three problems within the 1-hour limit, highlighting the necessity of pruning.
• The system successfully generates programs with complex structures such as nested loops, even when absent in the initial partial program.
• Simpl requires only a few input-output examples (up to four) and minimal resources, making it accessible for beginners.
• The tool is publicly available and open-sourced, supporting reproducibility and integration into educational platforms.