[Paper Review] Automatically Recommend Code Updates: Are We There Yet?

Yue Liu, Chakkrit Tantithamthavorn+3 more|arXiv (Cornell University)|Sep 15, 2022

Mobile and Web Applications5 citations

TL;DR

This paper presents AutoUpdate, the first comprehensive evaluation of pre-trained Code Language Models (CodeLMs) like CodeT5, CodeBERT, and UniXcoder for recommending Android code updates. Using a novel dataset of 209k updated methods from 3,195 Android apps, it shows CodeLMs outperform traditional methods by 190–385% in realistic time-wise evaluation, with CodeT5 achieving the best performance across update types.

ABSTRACT

In recent years, large pre-trained Language Models of Code (CodeLMs) have shown promising results on various software engineering tasks. One such task is automatic code update recommendation, which transforms outdated code snippets into their approved and revised counterparts. Although many CodeLM-based approaches have been proposed, claiming high accuracy, their effectiveness and reliability on real-world code update tasks remain questionable. In this paper, we present the first extensive evaluation of state-of-the-art CodeLMs for automatically recommending code updates. We assess their performance on two diverse datasets of paired updated methods, considering factors such as temporal evolution, project specificity, method size, and update complexity. Our results reveal that while CodeLMs perform well in settings that ignore temporal information, they struggle in more realistic time-wise scenarios and generalize poorly to new projects. Furthermore, CodeLM performance decreases significantly for larger methods and more complex updates. Furthermore, we observe that many CodeLM-generated "updates" are actually null, especially in time-wise settings, and meaningful edits remain challenging. Our findings highlight the significant gap between the perceived and actual effectiveness of CodeLMs for real-world code update recommendation and emphasize the need for more research on improving their practicality, robustness, and generalizability.

Motivation & Objective

• To investigate the feasibility and effectiveness of pre-trained CodeLMs in recommending automated code updates for Android applications.
• To address the gap in existing methods that are not tailored for Android-specific code updates, such as API deprecations and compatibility fixes.
• To evaluate the performance of CodeLMs under realistic time-wise evaluation scenarios, avoiding overly optimistic results from random data splits.
• To analyze the impact of update types, method size, and update size on CodeLM performance for Android code updates.
• To provide a foundation for future research by identifying limitations in temporal adaptability and generalization of CodeLMs in evolving Android ecosystems.

Proposed method

• Curated a unique dataset of 209,000 paired updated methods from 3,195 open-source Android apps hosted on GitHub and published on Google Play between 2008 and 2022.
• Evaluated state-of-the-art CodeLMs—CodeT5, CodeBERT, CodeGPT, UniXcoder, and T5—on the task of recommending code updates using a sequence-to-sequence generation framework.
• Applied beam search decoding with varying beam sizes to assess generation quality and robustness across different inference strategies.
• Implemented a time-wise evaluation scenario, where models were trained on older code versions and tested on future updates, simulating real-world deployment conditions.
• Conducted ablation studies to analyze the influence of update type, method size, and update size on model performance.
• Used standard metrics such as exact match and BLEU to evaluate prediction accuracy and fluency of generated code updates.

Experimental results

Research questions

• RQ1 How do pre-trained CodeLMs compare to traditional deep learning models (e.g., Tufano et al., AutoTransform) in recommending Android code updates?
• RQ2 What is the impact of using a time-wise evaluation scenario versus random data splitting on the perceived performance of CodeLMs?
• RQ3 Which CodeLM architecture (e.g., CodeT5, CodeBERT) performs best across different types of code updates in Android apps?
• RQ4 How do update type, method size, and update size affect the performance of CodeLMs in code update recommendation?
• RQ5 To what extent do CodeLMs generalize across different Android app codebases and evolving API changes?

Key findings

• Pre-trained CodeLMs significantly outperform traditional baselines, achieving a 190% to 385% improvement in perfect prediction accuracy under realistic time-wise evaluation.
• CodeT5 consistently achieves the highest performance across all update types, demonstrating superior generalization and adaptability in Android code update recommendation.
• Time-wise evaluation reveals that random data splits lead to overly optimistic performance estimates, highlighting the need for temporal validation in future SE model evaluations.
• The dataset contains diverse update types, with refactoring and bug fixes being the most frequent, indicating the need for models to generalize across multiple change semantics.
• Model performance degrades with larger method sizes and larger update sizes, suggesting challenges in handling complex or extensive code transformations.
• CodeLMs show strong potential for automating Android code updates, but their temporal adaptability and robustness to evolving APIs remain key challenges for future work.