[Paper Review] Adaptive Intellect Unleashed: The Feasibility of Knowledge Transfer in Large Language Models

Qing Huang, Yishun Wu+4 more|arXiv (Cornell University)|Aug 9, 2023

Topic ModelingComputer Science3 citations

TL;DR

This paper proposes a knowledge transfer framework to enhance large language models' (LLMs) generalization in software engineering tasks by guiding them to transfer knowledge from familiar, similar APIs or code snippets to unseen ones. The approach improves performance across API inference, code example generation, and FQN inference, with hierarchical transfer and AI-Chain prompting outperforming direct transfer and chain-of-thought.

ABSTRACT

We conduct the first empirical study on using knowledge transfer to improve the generalization ability of large language models (LLMs) in software engineering tasks, which often require LLMs to generalize beyond their training data. Our proposed general knowledge transfer approach guides the LLM towards a similar and familiar API or code snippet it has encountered before, improving the model's generalization ability for unseen knowledge. We apply this approach to three software engineering tasks: API inference, code example generation, and FQN inference, and find transfer span, transfer strategy, and transfer architecture as key factors affecting the method. Our findings demonstrate the feasibility of knowledge transfer and its potential to enhance LLMs' performance in various software engineering tasks. The effectiveness of knowledge transfer varies depending on the target domain and task, with the hierarchical strategy being more effective than direct transfer, and AI-Chain outperforming CoT in prompt design. The implications of these findings extend beyond software engineering tasks and suggest that knowledge transfer can enhance LLMs' ability to handle unknowns in any natural language task.

Motivation & Objective

• Address the limited generalization ability of LLMs in software engineering tasks due to insufficient training data coverage.
• Investigate whether knowledge transfer can improve LLMs' performance on unseen software engineering tasks.
• Identify key factors—transfer span, strategy, and architecture—that influence knowledge transfer effectiveness.
• Demonstrate the feasibility of knowledge transfer across three distinct software engineering tasks: API inference, code example generation, and FQN inference.
• Provide a generalizable framework for enhancing LLMs' ability to handle unknown or rare code constructs through transfer from familiar, semantically similar examples.

Proposed method

• Propose a general knowledge transfer approach that maps unknown code constructs to functionally similar, familiar APIs or code snippets.
• Use in-context prompting to guide LLMs to first identify a known, analogous source (e.g., a Java API) before transferring knowledge to the target (e.g., a Kotlin API).
• Implement two transfer strategies: direct transfer (one-step mapping) and hierarchical transfer (multi-step reasoning via intermediate familiar constructs).
• Apply two prompt design architectures: Chain-of-Thought (CoT) and AI-Chain, to evaluate their impact on knowledge transfer performance.
• Use GPT-3.5 as the base LLM and collect new APIs and code snippets from post-2021 documentation to test generalization on unseen data.
• Evaluate transfer effectiveness across three tasks using zero-shot and few-shot settings, measuring accuracy and generalization gains.

Experimental results

Research questions

• RQ1 To what extent can knowledge transfer improve LLM generalization in software engineering tasks with unseen APIs or code snippets?
• RQ2 How do different transfer spans (e.g., source-target distance) affect knowledge transfer performance?
• RQ3 Which transfer strategy—direct or hierarchical—yields better results in improving LLM generalization?
• RQ4 How do different prompt architectures (e.g., CoT vs. AI-Chain) influence the effectiveness of knowledge transfer?
• RQ5 Does the effectiveness of knowledge transfer vary across different software engineering tasks and domains?

Key findings

• Knowledge transfer is feasible and significantly improves LLM generalization in software engineering tasks, especially when the target task involves unseen or rare code constructs.
• The hierarchical transfer strategy outperforms direct transfer, achieving higher accuracy by enabling multi-step reasoning through intermediate, familiar code constructs.
• AI-Chain outperforms CoT in prompt design, demonstrating greater effectiveness in guiding LLMs to transfer knowledge accurately across domains.
• Transfer performance varies by task and domain, with FQN inference showing strong gains due to structural similarity in package names and method signatures.
• The method successfully generalizes to new APIs released after the training data cutoff (post-September 2021), confirming its robustness to data distribution shifts.
• The approach is transferable across programming languages, with successful mapping from Java to Kotlin examples, suggesting potential for broader language support.