[Paper Review] A Survey on LLM-based Code Generation for Low-Resource and Domain-Specific Programming Languages

Sathvik Joel, Jie JW Wu+1 more|arXiv (Cornell University)|Oct 4, 2024

Embedded Systems Design Techniques4 citations

TL;DR

This survey conducts a systematic literature review of 111 papers (2020–2024) to analyze LLM-based code generation for low-resource (LRPLs) and domain-specific (DSLs) programming languages. It identifies key challenges such as data scarcity and specialized semantics, evaluates existing benchmarks, metrics, and model improvement strategies, and highlights the lack of standardized evaluation frameworks—offering a foundation for future research in this underexplored domain.

ABSTRACT

Large Language Models (LLMs) have shown impressive capabilities in code generation for popular programming languages. However, their performance on Low-Resource Programming Languages (LRPLs) and Domain-Specific Languages (DSLs) remains a significant challenge, affecting millions of developers-3.5 million users in Rust alone-who cannot fully utilize LLM capabilities. LRPLs and DSLs encounter unique obstacles, including data scarcity and, for DSLs, specialized syntax that is poorly represented in general-purpose datasets. Addressing these challenges is crucial, as LRPLs and DSLs enhance development efficiency in specialized domains, such as finance and science. While several surveys discuss LLMs in software engineering, none focus specifically on the challenges and opportunities associated with LRPLs and DSLs. Our survey fills this gap by systematically reviewing the current state, methodologies, and challenges in leveraging LLMs for code generation in these languages. We filtered 111 papers from over 27,000 published studies between 2020 and 2024 to evaluate the capabilities and limitations of LLMs in LRPLs and DSLs. We report the LLMs used, benchmarks, and metrics for evaluation, strategies for enhancing performance, and methods for dataset collection and curation. We identified four main evaluation techniques and several metrics for assessing code generation in LRPLs and DSLs. Our analysis categorizes improvement methods into six groups and summarizes novel architectures proposed by researchers. Despite various techniques and metrics, a standard approach and benchmark dataset for evaluating code generation in LRPLs and DSLs are lacking. This survey serves as a resource for researchers and practitioners at the intersection of LLMs, software engineering, and specialized programming languages, laying the groundwork for future advancements in code generation for LRPLs and DSLs.

Motivation & Objective

• Address the critical gap in existing literature that overlooks low-resource and domain-specific programming languages (LRPLs and DSLs) in LLM-based code generation research.
• Systematically analyze the current state of LLMs for code generation in LRPLs and DSLs, focusing on evaluation metrics, benchmarks, and model improvement techniques.
• Identify and categorize data collection and curation strategies specific to LRPLs and DSLs, which face unique challenges due to limited training data and specialized semantics.
• Highlight the lack of standardized benchmarks and evaluation metrics for LRPLs and DSLs, contrasting their challenges with those in high-resource programming languages (HRPLs).
• Provide actionable research opportunities to advance code generation in specialized domains, including cross-lingual transfer, intermediate representations, and formal verification techniques.

Proposed method

• Conducted a systematic literature review (SLR) filtering 111 papers from over 27,000 studies (2020–2024), with additional snowballing and extended search (2024–2025) to ensure comprehensive coverage.
• Categorized evaluation techniques into four main types and analyzed 14+ metrics used to assess code generation performance in LRPLs and DSLs, including syntactic and semantic correctness.
• Classified model improvement strategies into six main groups: fine-tuning, prompt engineering, retrieval-augmented generation (RAG), distillation, chain-of-thought prompting, and fine-tuning with synthetic data.
• Mapped and analyzed datasets used in the literature, focusing on curation methods such as web scraping, mining from repositories, and manual annotation, especially for low-resource languages.
• Explored novel architectures and techniques such as intermediate language representations and symbolic execution for improving code generation and evaluation in DSLs.
• Used a structured framework to compare approaches in LRPLs and DSLs, identifying distinct challenges—especially for DSLs—due to domain-specific syntax, semantics, and expertise requirements.

Experimental results

Research questions

• RQ1 What benchmarks, metrics, and evaluation techniques are currently used to assess LLM-based code generation in low-resource and domain-specific programming languages?
• RQ2 What methodologies and model improvement strategies are proposed in the literature to enhance LLM performance for LRPLs and DSLs?
• RQ3 How are datasets collected, curated, and pre-processed for training and evaluating LLMs in LRPL and DSL contexts?
• RQ4 What are the key differences in challenges and approaches between code generation for LRPLs versus DSLs, particularly regarding data scarcity and domain specificity?
• RQ5 What research opportunities exist to address the lack of standardization in evaluation and the absence of unified benchmarks for LRPLs and DSLs?

Key findings

• Despite growing interest, most LRPLs and DSLs lack standardized benchmarks and evaluation metrics, unlike high-resource languages such as Python.
• The MultiPL-E benchmark shows a stark performance gap, with LLMs achieving significantly lower scores on LRPLs like Rust and R compared to Python.
• Six main categories of model improvement strategies were identified: fine-tuning, prompt engineering, RAG, distillation, chain-of-thought prompting, and synthetic data generation.
• Data collection for LRPLs and DSLs relies heavily on web scraping, GitHub mining, and manual curation, but these methods often result in limited, noisy, or non-representative datasets.
• DSLs face more pronounced challenges than LRPLs due to highly specialized syntax, semantics, and the need for expert knowledge or domain-specific tools.
• Research opportunities include developing formal equivalence checkers, cross-lingual transfer learning from HRPLs, intermediate language representations, and integrating symbolic execution or theorem proving for semantic validation.