[Paper Review] Invalidator: Automated Patch Correctness Assessment via Semantic and Syntactic Reasoning

Thanh Le-Cong, Duc-Minh Luong+5 more|arXiv (Cornell University)|Jan 3, 2023

Software Testing and Debugging Techniques4 citations

TL;DR

Invalidator is a novel automated patch correctness assessment technique that combines semantic reasoning via program invariants and syntactic reasoning using CodeBERT embeddings to detect overfitting in APR-generated patches. It outperforms state-of-the-art baselines by detecting 79% of overfitting patches—23% more than the best baseline—while requiring no test case generation and operating fully automatically.

ABSTRACT

Automated program repair (APR) faces the challenge of test overfitting, where generated patches pass validation tests but fail to generalize. Existing methods for patch assessment involve generating new tests or manual inspection, which can be time-consuming or biased. In this paper, we propose a novel technique, INVALIDATOR, to automatically assess the correctness of APR-generated patches via semantic and syntactic reasoning. INVALIDATOR leverages program invariants to reason about program semantics while also capturing program syntax through language semantics learned from a large code corpus using a pre-trained language model. Given a buggy program and the developer-patched program, INVALIDATOR infers likely invariants on both programs. Then, INVALIDATOR determines that an APR-generated patch overfits if: (1) it violates correct specifications or (2) maintains erroneous behaviors from the original buggy program. In case our approach fails to determine an overfitting patch based on invariants, INVALIDATOR utilizes a trained model from labeled patches to assess patch correctness based on program syntax. The benefit of INVALIDATOR is threefold. First, INVALIDATOR leverages both semantic and syntactic reasoning to enhance its discriminative capability. Second, INVALIDATOR does not require new test cases to be generated, but instead only relies on the current test suite and uses invariant inference to generalize program behaviors. Third, INVALIDATOR is fully automated. Experimental results demonstrate that INVALIDATOR outperforms existing methods in terms of Accuracy and F-measure, correctly identifying 79% of overfitting patches and detecting 23% more overfitting patches than the best baseline.

Motivation & Objective

• Address the persistent challenge of test overfitting in automated program repair (APR), where patches pass test suites but fail to generalize.
• Overcome the limitations of existing approaches that rely on expensive manual inspections or hard-to-generate test cases to detect behavioral differences.
• Develop an automated, scalable, and accurate method to assess patch correctness without requiring new test case generation.
• Leverage both semantic (program invariants) and syntactic (pre-trained code embeddings) reasoning to improve detection of overfitting patches.
• Provide a fully automated solution that enhances the reliability and practical adoption of APR tools in real-world software development.

Proposed method

• Infers program invariants from both the buggy and developer-patched versions of a program to capture semantic specifications.
• Determines that an APR-generated patch is overfitting if it violates correct invariants or preserves erroneous behaviors from the original buggy program.
• When invariant-based reasoning fails, employs a trained machine learning model to assess patch correctness based on syntactic similarity.
• Uses CodeBERT to generate dense vector representations of program syntax, capturing language semantics from a large code corpus.
• Measures syntactic differences between the APR-generated patch, the buggy program, and the ground-truth developer patch to estimate overfitting likelihood.
• Combines invariant-based reasoning and syntax-based classification in a two-stage framework to improve discriminative power and robustness.

(a) An overfitting patch generated by Kali [ 35 ]

Experimental results

Research questions

• RQ1 Can program invariants effectively capture semantic specifications to detect overfitting in APR-generated patches?
• RQ2 Can syntactic similarity measured via pre-trained code models improve the detection of overfitting patches when semantic reasoning fails?
• RQ3 How does combining semantic and syntactic reasoning outperform existing automated patch correctness assessment techniques?
• RQ4 To what extent can Invalidator detect overfitting patches without generating additional test cases?
• RQ5 How does Invalidator compare in performance to state-of-the-art baselines on real-world APR datasets?

Key findings

• Invalidator successfully detects 79% of overfitting patches on a dataset of 885 APR-generated patches from Defects4J, representing a 23% improvement over the best-performing baseline.
• Invalidator achieves 14% higher accuracy and 19% higher F-measure compared to the best baseline, demonstrating superior overall performance.
• The integration of program invariants enables Invalidator to generalize beyond test suite coverage, identifying patches that pass tests but are semantically incorrect.
• The syntactic-based classifier using CodeBERT features provides a flexible, adaptive threshold for patch correctness by comparing similarity to ground truth and buggy versions.
• Invalidator operates fully automatically without requiring any new test case generation, reducing dependency on complex test generation tools.
• The approach demonstrates that combining semantic and syntactic reasoning significantly enhances the detection of overfitting patches compared to methods relying on only one modality.

(b) The correct patch written by human developers