[Paper Review] Dcc --help: Generating Context-Aware Compiler Error Explanations with Large Language Models

Andrew Taylor, Alexandra Vassar+2 more|arXiv (Cornell University)|Aug 23, 2023

Software Engineering ResearchComputer Science3 citations

TL;DR

This paper introduces dcc --help, an open-source tool that integrates a Large Language Model (LLM) into the Debugging C Compiler (DCC) to generate context-aware, novice-focused explanations for both compile-time and run-time errors. By prompting the LLM with source code, error location, and standard error messages, the system produces actionable, natural language guidance—achieving 90% conceptual accuracy for compile-time errors and 75% for run-time errors, with high student adoption in a large-scale CS1/CS2 course.

ABSTRACT

In the challenging field of introductory programming, high enrollments and failure rates drive us to explore tools and systems to enhance student outcomes, especially automated tools that scale to large cohorts. This paper presents and evaluates the dcc --help tool, an integration of a Large Language Model (LLM) into the Debugging C Compiler (DCC) to generate unique, novice-focused explanations tailored to each error. dcc --help prompts an LLM with contextual information of compile- and run-time error occurrences, including the source code, error location and standard compiler error message. The LLM is instructed to generate novice-focused, actionable error explanations and guidance, designed to help students understand and resolve problems without providing solutions. dcc --help was deployed to our CS1 and CS2 courses, with 2,565 students using the tool over 64,000 times in ten weeks. We analysed a subset of these error/explanation pairs to evaluate their properties, including conceptual correctness, relevancy, and overall quality. We found that the LLM-generated explanations were conceptually accurate in 90% of compile-time and 75% of run-time cases, but often disregarded the instruction not to provide solutions in code. Our findings, observations and reflections following deployment indicate that dcc-help provides novel opportunities for scaffolding students' introduction to programming.

Motivation & Objective

• Address the high failure and attrition rates in introductory programming (CS1) courses by improving student access to meaningful error feedback.
• Overcome the limitations of cryptic, unhelpful compiler error messages that hinder novice learners in understanding and resolving programming errors.
• Scale personalized, on-demand debugging assistance to large student cohorts where traditional teaching support is insufficient.
• Transform the compiler from a passive error reporter into an active, context-aware learning guide using generative AI.
• Evaluate the feasibility, quality, and impact of LLM-generated explanations in real-world educational settings with thousands of students.

Proposed method

• Integrate a Large Language Model (GPT-3.5-turbo-0301) directly into the DCC compiler pipeline to generate real-time explanations.
• Prompt the LLM with contextual information: source code, error location, and the original compiler error message.
• Instruct the LLM to produce novice-focused, conceptual, and actionable explanations without providing direct code solutions.
• Implement rate-limiting and warnings to prevent over-reliance and promote critical thinking.
• Deploy the tool in CS1 and CS2 courses at a large Australian university, enabling students to access explanations on-demand.
• Collect and analyze 64,000 error/explanation pairs from 2,565 students over ten weeks to evaluate explanation quality and usage patterns.

Experimental results

Research questions

• RQ1 To what extent are LLM-generated explanations conceptually accurate for compile-time and run-time C programming errors?
• RQ2 How do students use dcc --help in practice, and does usage correlate with academic deadlines or difficulty levels?
• RQ3 Can LLM-generated explanations effectively scaffold novice programmers without providing direct code solutions?
• RQ4 What are the perceived and measurable impacts of such a tool on student learning outcomes and debugging confidence?
• RQ5 How does the integration of generative AI into the compiler change the role of the compiler from error reporting to active learning support?

Key findings

• The LLM-generated explanations were conceptually correct in 90% of compile-time error cases and 75% of run-time error cases.
• Despite instructions to avoid code solutions, the LLM frequently provided direct code fixes, indicating a need for stronger prompt constraints.
• The tool was used over 64,000 times by 2,565 students in ten weeks, with usage spiking before major assessment deadlines.
• Students showed strong engagement with the tool, suggesting it was perceived as a valuable, accessible learning aid.
• No significant performance decline was observed in the final exam—indicating that the tool did not undermine learning outcomes.
• The integration of generative AI into the compiler transformed its role into a proactive, context-aware learning companion, enhancing student support at scale.