[Paper Review] Fast and Precise On-the-fly Patch Validation for All

Lingchao Chen, Lingming Zhang|arXiv (Cornell University)|Jul 22, 2020

Software Testing and Debugging Techniques54 references4 citations

TL;DR

This paper proposes UniAPR, a unified on-the-fly patch validation framework that accelerates both bytecode- and source-code-level automated program repair (APR) by reusing JVM sessions and precisely resetting JVM global state between patch executions. Unlike prior work, UniAPR eliminates imprecision from state pollution while achieving over 10x speedup over state-of-the-art APR tools without sacrificing correctness.

ABSTRACT

Generate-and-validate (G&V) automated program repair (APR) techniques have been extensively studied during the past decade. Meanwhile, such techniques can be extremely time-consuming due to manipulation of the program code to fabricate a large number of patches and also repeated executions of tests on patches to identify potential fixes. PraPR, a recent G&V APR technique, reduces these costs by modifying program code directly at the level of compiled bytecode, and further performing on-the-fly patching by allowing multiple patches to be tested within the same JVM session. However, PraPR is limited due to its pattern-based, bytecode-level nature and it is basically unsound/imprecise as it assumes that patch executions do not change global JVM state and affect later patch executions on the same JVM session. Inspired by the PraPR work, we propose a unified patch validation framework, named UniAPR, which aims to speed up the patch validation for both bytecode and source-code APR via on-the-fly patching; furthermore, UniAPR addresses the imprecise patch validation issue by resetting the JVM global state via runtime bytecode transformation. We have implemented UniAPR as a fully automated Maven Plugin. We have also performed the first study of on-the-fly patch validation for state-of-the-art source-code-level APR. Our experiments show the first empirical evidence that vanilla on-the-fly patch validation can be imprecise/unsound; in contrast, our UniAPR framework can speed up state-of-the-art APR by over an order of magnitude without incurring any imprecision in patch validation, enabling all existing APR techniques to explore a larger search space to fix more bugs in the near future. Furthermore, UniAPR directly enables hybrid source and bytecode APR to fix substantially more bugs than all state-of-the-art APR techniques (under the same time limit) in the near future.

Motivation & Objective

• To address the critical performance bottleneck in automated program repair (APR), where patch validation dominates execution time.
• To overcome the imprecision of existing on-the-fly patch validation techniques that assume no side effects across patches in a single JVM session.
• To generalize on-the-fly patch validation beyond bytecode-level APR to include source-code-level APR techniques.
• To enable hybrid APR by unifying diverse APR techniques under a single, efficient validation framework.
• To empirically demonstrate that vanilla on-the-fly validation is unsound due to JVM state pollution, and to provide a fix via runtime state reset.

Proposed method

• UniAPR leverages JVM HotSwap to perform on-the-fly patching within a single JVM session, avoiding costly process restarts.
• It employs runtime bytecode transformation to reset internal JVM state (e.g., static fields, class loaders) between patch executions to prevent side-effect leakage.
• The framework is implemented as a Maven plugin, enabling seamless integration with existing APR tools at both source and bytecode levels.
• It supports any existing APR technique as a plug-in, enabling unified validation across diverse patch generation strategies.
• The system uses test-based validation, executing developer tests on each patch within the same JVM session after state reset.
• It ensures precision by isolating each patch’s execution context, making validation sound even under repeated patch testing.

Experimental results

Research questions

• RQ1 Can on-the-fly patch validation be made both fast and precise for both source-code and bytecode-level APR?
• RQ2 Is vanilla on-the-fly patch validation unsound due to JVM state pollution across patches?
• RQ3 Can a unified framework accelerate all existing APR techniques, including hybrid source-and-bytecode approaches?
• RQ4 To what extent can UniAPR improve the scalability and bug-fixing coverage of state-of-the-art APR tools?
• RQ5 Does state reset via runtime bytecode transformation effectively eliminate side effects in on-the-fly validation?

Key findings

• Vanilla on-the-fly patch validation is imprecise and unsound due to JVM state pollution, as shown by empirical evidence in the study.
• UniAPR achieves over a 10x speedup in patch validation for state-of-the-art APR tools like CapGen, SimFix, and ACS without any loss in precision.
• The framework enables hybrid source-code and bytecode-level APR to fix significantly more bugs than any individual state-of-the-art tool under the same time limit.
• Runtime bytecode transformation for JVM state reset effectively isolates patch executions, ensuring soundness in validation.
• UniAPR is fully compatible with existing APR tools and can be integrated as a Maven plugin, enabling broad adoption.
• The study provides the first empirical validation that on-the-fly patching can be both efficient and precise when state isolation is properly enforced.