[Paper Review] Smart tutor to provide feedback in programming courses

David Roldán‐Álvarez|arXiv (Cornell University)|Jan 24, 2023

E-Learning and Knowledge ManagementComputer Science3 citations

TL;DR

This paper presents an AI-based intelligent tutor designed to provide real-time, personalized feedback to university students learning programming, using natural language processing to understand and respond to their coding questions. Tested over a full course at URJC, the system demonstrated effectiveness in delivering accurate answers and examples, with students finding it easy to use and potentially valuable for other courses.

ABSTRACT

Artificial Intelligence (AI) is becoming more and more popular as time passes, allowing to perform tasks that were difficult to do in the past. From predictions to customization, AI is being used in many areas, not being educational environments outside this situation. AI is being used in educational settings to customize contents or to provide personalized feedback to the students, among others. In this scenario, AI in programming teaching is something that still has to be explored, since in this area we usually find assessment tools that allow grading the students work, but we can not find many tools aimed towards providing feedback to the students in the process of creating their program. In this work we present an AI based intelligent tutor that answers students programming questions. The tool has been tested by university students at the URJC along a whole course. Even if the tool is still in its preliminary phase, it helped the students with their questions, providing accurate answers and examples. The students were able to use the intelligent tutor easily and they thought that it could be a useful tool to use in other courses.

Motivation & Objective

• To address the lack of AI tools that provide formative feedback during the programming development process, beyond automated grading.
• To design an intelligent tutor that understands students' programming questions and delivers accurate, context-aware responses.
• To evaluate the usability and perceived usefulness of the tutor in a real university programming course setting.
• To explore the potential of AI in enhancing student learning by offering immediate, example-based feedback.

Proposed method

• The system employs natural language processing (NLP) techniques to interpret students' textual questions about their code.
• It retrieves relevant feedback and code examples from a knowledge base tailored to common programming errors and concepts.
• The tutor generates responses by matching student queries to pre-defined patterns and solutions in the knowledge base.
• The tool was integrated into a university programming course at URJC for full-term deployment and evaluation.
• Student interactions were logged to assess response accuracy and user experience.
• Feedback was evaluated based on student perceptions and system performance in answering questions.

Experimental results

Research questions

• RQ1 Can an AI-based tutor effectively understand and respond to students’ natural language questions about their programming code?
• RQ2 How do students perceive the usability and helpfulness of the intelligent tutor in a real course setting?
• RQ3 To what extent does the tutor improve student learning through timely, accurate feedback?
• RQ4 Can such a system be practically deployed and adopted in university-level programming education?

Key findings

• The intelligent tutor successfully provided accurate and relevant feedback to students' programming questions during a full academic course.
• Students reported that the tool was easy to use and perceived it as a potentially useful aid for future courses.
• The system demonstrated feasibility in delivering example-based responses that supported student understanding.
• Although still in a preliminary phase, the tool showed promise in enhancing formative feedback in programming education.