[Paper Review] TaskTracker-tool: a Toolkit for Tracking of Code Snapshots and Activity Data During Solution of Programming Tasks

Elena Lyulina, Anastasiia Birillo+2 more|arXiv (Cornell University)|Dec 9, 2020

Teaching and Learning Programming21 references4 citations

TL;DR

This paper presents TaskTracker-tool, a toolkit for collecting and analyzing code snapshots and IDE interaction data during programming task solutions in IntelliJ-based IDEs. It enables fine-grained tracking of student activity across multiple languages (Python, Java, Kotlin, C++), supports server-side data aggregation, and includes post-processing tools for analysis. The key contribution is a publicly available dataset of 148 participants’ solution processes, enabling deep insights into programming behavior, error patterns, and learning dynamics.

ABSTRACT

The process of writing code and use of features in an integrated development environment (IDE) is a fruitful source of data in computing education research. Existing studies use records of students' actions in the IDE, consecutive code snapshots, compilation events, and others, to gain deep insight into the process of student programming. In this paper, we present a set of tools for collecting and processing data of student activity during problem-solving. The first tool is a plugin for IntelliJ-based IDEs (PyCharm, IntelliJ IDEA, CLion). By capturing snapshots of code and IDE interaction data, it allows to analyze the process of writing code in different languages -- Python, Java, Kotlin, and C++. The second tool is designed for the post-processing of data collected by the plugin and is capable of basic analysis and visualization. To validate and showcase the toolkit, we present a dataset collected by our tools. It consists of records of activity and IDE interaction events during solution of programming tasks by 148 participants of different ages and levels of programming experience. We propose several directions for further exploration of the dataset.

Motivation & Objective

• To address the lack of comprehensive, multi-language tools for capturing detailed programming activity during task solving in educational settings.
• To enable researchers and educators to study the full process of student problem-solving, including code evolution and IDE interactions, across diverse programming languages.
• To provide a flexible, extensible toolkit that supports data collection, remote storage, and basic analysis without disrupting the natural programming environment.
• To generate a public, high-fidelity dataset of student programming behavior for future research in computing education.
• To support the development of personalized learning interventions, such as context-aware hints, based on real student solution patterns.

Proposed method

• A plugin for IntelliJ-based IDEs (e.g., PyCharm, IntelliJ IDEA) captures sequential code snapshots and low-level IDE interaction events (e.g., typing, running, refactoring) in real time.
• The plugin sends data to a remote server for centralized storage and management, ensuring data integrity and scalability.
• A dedicated post-processing tool enables basic analysis, such as score dynamics, action frequency, and code change patterns, with visualization support.
• The toolkit supports tracking per-task context, preserving the relationship between actions and specific programming problems.
• The system is designed to be extensible, with modular components for plugin, server, and analysis tool, allowing customization for different research needs.
• The dataset was collected from 148 participants across varying ages and experience levels, with activity logged during solution of standardized programming tasks.

Experimental results

Research questions

• RQ1 How do students' coding patterns and IDE interaction behaviors differ across programming languages and experience levels?
• RQ2 What are the common error patterns and solution trajectories in programming tasks, and how do they vary with student experience?
• RQ3 How do students modify their code after achieving a correct solution, and what actions are most frequent during refactoring?
• RQ4 Can code change dynamics and score evolution be used to infer student intent and problem-solving strategies?
• RQ5 To what extent can solution process data be used to generate personalized hints or feedback in real time?

Key findings

• The dataset reveals that students often paste large code blocks from previous solutions, suggesting reliance on prior work or copy-paste patterns.
• Many students perform frequent code runs during development, indicating a preference for validating incomplete code early and often.
• A significant number of students use the IDE as a diff tool by pasting input and output for character-by-character comparison.
• Solution processes can be segmented into three distinct phases: initial implementation, correction, and refactoring, as shown by score and code change dynamics.
• There is a notable gap in post-solution activity: students frequently refactor code after achieving a correct score, indicating a focus on code quality beyond correctness.
• Age and experience levels correlate with distinct patterns in feature usage and error types, suggesting developmental trajectories in programming behavior.