[Paper Review] Efficient and Green Large Language Models for Software Engineering: Literature Review, Vision, and the Road Ahead

Jieke Shi, Zhou Yang+1 more|arXiv (Cornell University)|Apr 6, 2024

Software System Performance and Reliability4 citations

TL;DR

This paper advocates for efficient and green Large Language Models (LLMs) in software engineering, proposing a research roadmap to reduce computational costs and environmental impact. It emphasizes model compression, inference acceleration, program optimization, and fine-tuning on efficient code to enable accessible, high-performance, and sustainable LLM4SE solutions.

ABSTRACT

Large Language Models (LLMs) have recently shown remarkable capabilities in various software engineering tasks, spurring the rapid growth of the Large Language Models for Software Engineering (LLM4SE) area. However, limited attention has been paid to developing efficient LLM4SE techniques that demand minimal computational cost, time, and memory resources, as well as green LLM4SE solutions that reduce energy consumption, water usage, and carbon emissions. This paper aims to redirect the focus of the research community towards the efficiency and greenness of LLM4SE, while also sharing potential research directions to achieve this goal. It commences with a brief overview of the significance of LLM4SE and highlights the need for efficient and green LLM4SE solutions. Subsequently, the paper presents a vision for a future where efficient and green LLM4SE revolutionizes the LLM-based software engineering tool landscape, benefiting various stakeholders, including industry, individual practitioners, and society. The paper then delineates a roadmap for future research, outlining specific research paths and potential solutions for the research community to pursue. While not intended to be a definitive guide, the paper aims to inspire further progress, with the ultimate goal of establishing efficient and green LLM4SE as a central element in the future of software engineering.

Motivation & Objective

• Address the high computational and environmental costs of current LLM4SE solutions, which limit accessibility for individual developers and startups.
• Overcome challenges in LLM4SE efficiency, including long training times, high memory usage, and slow inference due to latency and resource demands.
• Reduce the environmental footprint of LLM4SE by minimizing energy consumption and carbon emissions during training and inference.
• Improve the efficiency and sustainability of LLM-generated code, which often underperforms compared to human-written code in execution time and memory usage.
• Enable broader adoption of LLM4SE by making solutions feasible for deployment on personal devices and low-resource environments.

Proposed method

• Propose model compression techniques such as knowledge distillation, quantization, and pruning to reduce model size and inference cost without sacrificing performance.
• Explore dynamic and cascade inference strategies that adaptively select smaller or faster models based on input complexity, reducing reliance on large, slow LLMs.
• Investigate speculative and non-autoregressive decoding methods to accelerate LLM inference, improving response time and reducing energy use.
• Integrate LLMs with existing program analysis tools (e.g., code completion engines) to filter infeasible outputs early, enhancing inference efficiency.
• Develop program transformation tools that automatically optimize LLM inference code for hardware-specific features like CPU/GPU parallelism and backend libraries.
• Apply compiler-level optimizations such as dead code elimination and tuning of optimization flags to reduce execution overhead in LLM inference pipelines.

Experimental results

Research questions

• RQ1 How can LLM4SE solutions be made more efficient in terms of time, memory, and energy consumption during both training and inference?
• RQ2 What techniques can be used to accelerate LLM inference in software engineering tasks without compromising accuracy or functionality?
• RQ3 How can LLM-generated code be optimized to match or exceed the efficiency of human-written code in execution time and memory usage?
• RQ4 What role can integration with static analysis and program verification tools play in improving the efficiency and sustainability of LLM4SE pipelines?
• RQ5 How can LLMs be fine-tuned on human-written efficient code to generate high-performance, low-energy code by default?

Key findings

• Current LLM4SE solutions are prohibitively expensive and resource-intensive, with GPT-3 training costing over $4 million and cloud inference reaching $200,000/month.
• LLaMA model training consumes 2,638 MWh of energy and emits approximately 1,300 tons of CO₂, highlighting the environmental cost of LLMs.
• LLM inference on personal devices remains impractical due to high memory demands, limiting offline and low-latency deployment.
• LLM-generated code often exhibits worse performance than human-written code in execution time and memory usage, reducing software competitiveness.
• Techniques like speculative decoding and non-autoregressive generation can accelerate inference, but their application in LLM4SE remains underexplored.
• Optimizing LLM inference code via dead code removal, compiler flags, and hardware-aware compilation can significantly reduce energy consumption and improve efficiency.