[Paper Review] Domain Knowledge Matters: Improving Prompts with Fix Templates for Repairing Python Type Errors

Yun Peng, Shuzheng Gao+3 more|arXiv (Cornell University)|Jun 2, 2023

Software Engineering Research4 citations

TL;DR

This paper proposes TypeFix, a prompt-based approach that enhances code pre-trained models for repairing Python type errors by integrating domain-specific fix templates mined from real-world patches. By adaptively locating masks using generalized edit patterns, TypeFix improves prompt effectiveness, achieving 26 and 55 successful repairs on BugsInPy and TypeBugs—outperforming the best baseline by 9 and 14 fixes, respectively, with 75% coverage of developer patches.

ABSTRACT

Although the dynamic type system of Python facilitates the developers in writing Python programs, it also brings type errors at run-time. There exist rule-based approaches for automatically repairing Python type errors. The approaches can generate accurate patches but they require domain experts to design patch synthesis rules and suffer from low template coverage of real-world type errors. Learning-based approaches alleviate the manual efforts in designing patch synthesis rules. Among the learning-based approaches, the prompt-based approach which leverages the knowledge base of code pre-trained models via pre-defined prompts, obtains state-of-the-art performance in general program repair tasks. However, such prompts are manually defined and do not involve any specific clues for repairing Python type errors, resulting in limited effectiveness. How to automatically improve prompts with the domain knowledge for type error repair is challenging yet under-explored. In this paper, we present TypeFix, a novel prompt-based approach with fix templates incorporated for repairing Python type errors. TypeFix first mines generalized fix templates via a novel hierarchical clustering algorithm. The identified fix templates indicate the common edit patterns and contexts of existing type error fixes. TypeFix then generates code prompts for code pre-trained models by employing the generalized fix templates as domain knowledge, in which the masks are adaptively located for each type error instead of being pre-determined. Experiments on two benchmarks, including BugsInPy and TypeBugs, show that TypeFix successfully repairs 26 and 55 type errors, outperforming the best baseline approach by 9 and 14, respectively. Besides, the proposed fix template mining approach can cover 75% of developers' patches in both benchmarks, increasing the best rule-based approach PyTER by more than 30%.

Motivation & Objective

• To address the limited effectiveness of manually defined prompts in repairing Python type errors.
• To reduce reliance on expert-designed patch synthesis rules in rule-based approaches.
• To improve prompt-based program repair by incorporating domain knowledge from real-world type error fixes.
• To increase template coverage of real-world type errors beyond existing rule-based methods.
• To develop a method that automatically generates context-aware, adaptive prompts using mined fix templates.

Proposed method

• TypeFix first mines generalized fix templates using a novel hierarchical clustering algorithm on real-world type error fixes.
• The mined templates capture common edit patterns and syntactic contexts of type error repairs.
• The approach dynamically identifies optimal mask positions in buggy code based on the fix templates, replacing static prompt design.
• It integrates these templates as domain knowledge into prompts for code pre-trained models, enhancing their repair accuracy.
• The model generates patches by predicting masked tokens using the enriched, context-aware prompts.
• The method is evaluated on two benchmarks: BugsInPy and TypeBugs, using standard APR evaluation metrics.

Experimental results

Research questions

• RQ1 Can fix templates mined from real-world patches improve the performance of prompt-based program repair for Python type errors?
• RQ2 How does the adaptive mask placement strategy based on fix templates compare to fixed, pre-defined prompts in repairing complex type errors?
• RQ3 To what extent can the proposed fix template mining approach improve coverage over existing rule-based methods?
• RQ4 Does incorporating domain-specific knowledge via templates lead to higher repair success rates than generic prompts?
• RQ5 How robust is the approach across diverse type error patterns in real-world Python code?

Key findings

• TypeFix successfully repairs 26 type errors on the BugsInPy benchmark, outperforming the best baseline by 9 fixes.
• On the TypeBugs benchmark, TypeFix achieves 55 successful repairs, surpassing the best baseline by 14 fixes.
• The fix template mining approach covers 75% of developer patches in both benchmarks, increasing coverage by over 30% compared to the best rule-based method, PyTER.
• The adaptive mask placement strategy significantly improves repair effectiveness compared to static, manually defined prompts.
• The integration of domain-specific fix templates into prompts leads to a measurable performance gain in both repair accuracy and generalization.
• The approach demonstrates strong generalization across diverse type error patterns, indicating robustness in real-world scenarios.