[Paper Review] The potential of LLMs for coding with low-resource and domain-specific programming languages

Artur Tarassow|arXiv (Cornell University)|Jul 24, 2023

Topic Modeling4 citations

TL;DR

This study investigates the feasibility of using large language models (LLMs) for coding in low-resource and domain-specific programming languages, focusing on gretl’s hansl scripting language. Despite limited public code, the LLM—based on GPT-3.5—successfully generated, documented, and refactored gretl code, provided accurate econometric explanations, and corrected syntax through iterative prompting, demonstrating strong generalization to low-resource domain-specific languages.

ABSTRACT

This paper presents a study on the feasibility of using large language models (LLM) for coding with low-resource and domain-specific programming languages that typically lack the amount of data required for effective LLM processing techniques. This study focuses on the econometric scripting language named hansl of the open-source software gretl and employs a proprietary LLM based on GPT-3.5. Our findings suggest that LLMs can be a useful tool for writing, understanding, improving, and documenting gretl code, which includes generating descriptive docstrings for functions and providing precise explanations for abstract and poorly documented econometric code. While the LLM showcased promoting docstring-to-code translation capability, we also identify some limitations, such as its inability to improve certain sections of code and to write accurate unit tests. This study is a step towards leveraging the power of LLMs to facilitate software development in low-resource programming languages and ultimately to lower barriers to entry for their adoption.

Motivation & Objective

• To assess the feasibility of using large language models (LLMs) for programming in low-resource and domain-specific languages, such as gretl’s hansl.
• To investigate whether publicly available LLMs can generalize effectively to programming languages with limited training data, like hansl.
• To evaluate the LLM’s ability to assist with common econometric coding tasks, including code documentation, explanation, refactoring, and error correction.
• To explore the potential of LLMs as programming assistants in applied econometrics, where domain-specific and low-resource languages are prevalent.

Proposed method

• Employed a proprietary LLM based on GPT-3.5 to perform coding tasks in the hansl scripting language used by gretl.
• Conducted a series of coding tasks including code generation from natural language prompts, docstring generation, code explanation, and refactoring.
• Used iterative prompt development to correct syntactic errors in generated scripts, such as incorrect smpl and gnuplot command syntax.
• Evaluated the LLM’s output on real econometric tasks, including time-series differencing, OLS regression, and fitted value plotting.
• Compared generated code against correct gretl syntax and assessed correctness, readability, and technical accuracy of explanations.
• Focused on domain-specific tasks involving linear algebra and econometrics to test the LLM’s understanding of technical content.

Experimental results

Research questions

• RQ1 Can LLMs effectively generate and understand code in low-resource, domain-specific programming languages like hansl?
• RQ2 To what extent can LLMs produce accurate and syntactically correct gretl scripts from natural language descriptions?
• RQ3 How well can LLMs generate descriptive docstrings and explain complex econometric code in hansl?
• RQ4 Can iterative prompting correct syntactic errors in LLM-generated gretl code, and how effective is this process?
• RQ5 What are the limitations of current LLMs in improving code quality and generating unit tests for low-resource languages?

Key findings

• The LLM successfully generated a functional gretl script for an introductory econometrics task, including first differencing, OLS regression, and plotting fitted values.
• Despite minor syntactic errors—such as incorrect smpl and gnuplot command syntax—the LLM-generated script was largely correct and usable after iterative prompting.
• The LLM demonstrated strong capability in generating descriptive, technical docstrings for hansl functions and translating them back into code.
• The model provided precise, accurate explanations of abstract and poorly documented econometric code, enhancing code readability and maintainability.
• The LLM improved code readability by suggesting better variable and function names, particularly in complex linear algebra and statistical operations.
• The LLM failed to generate correct unit tests for gretl code, indicating a key limitation in test generation for low-resource languages.