[Paper Review] Empirical Characterization of Logging Smells in Machine Learning Code

Patrick Loic Foalem, Léuson Da Silva+3 more|arXiv (Cornell University)|Mar 24, 2026

Scientific Computing and Data Management0 citations

TL;DR

This paper empirically identifies 12 ML-specific logging smells from 444 open-source ML repos, validating them with a practitioner survey and releasing a labeled dataset of 2,448 smell instances.

ABSTRACT

Logging plays a central role in ensuring reproducibility, observability, and reliability in machine learning (ML) systems. While logging is generally considered a good engineering practice, poorly designed logging can negatively affect experiment tracking, security, debugging, and system performance. In this paper, we present an empirical study of logging smells in ML projects and propose a taxonomy of ML-specific logging smell types. We conducted a large-scale analysis of 444 ML repositories and manually labeled 2,448 instances of logging smells. Based on this analysis, we identified 12 categories of logging smells spanning security, metric management, configuration, verbosity, and context-related issues. Our results show that logging smells are widespread in ML systems and vary in frequency and manifestation across projects. To assess practical relevance, we conducted a survey with 27 ML practitioners. Most respondents agreed with the identified smells and reported that several types, including Logging Sensitive Data, Metric Overwrite, Missing Hyperparameter Logging, and Log Without Context, have a strong impact on reproducibility, maintainability, and trustworthiness. Other smells, such as Heavy Data Logging and Print-based Logging, were perceived as more context-dependent. We publicly release our labeled dataset to support future research. Our findings highlight logging quality as a critical and underexplored aspect of ML system engineering and open opportunities for automated detection and repair of logging issues.

Motivation & Objective

• Motivate the study by showing logging is crucial for ML observability but often poorly designed in ML codebases.
• Build a data-driven taxonomy of ML-specific logging smells through large-scale labeling.
• Validate the taxonomy with ML practitioners to assess relevance and severity.
• Provide a replication package with a labeled dataset to support future research.

Proposed method

• Use an existing dataset of ML logging practices from Foalem et al. (2024) to identify logging statements in Python files across 444 active ML repositories.
• Cluser and filter logging-related code to exclude purely configuration statements, yielding 15,911 logging statements across 4,528 functions.
• Apply a human-in-the-loop, iterative taxonomy construction with LLM-assisted coding (GPT-5-mini) and manual validation to derive 12 smells.
• Split 2,448 function-level samples into phases for seed generation, iterative refinement, saturation, and reliability assessment with inter-rater agreement (κ = 0.817).
• Produce a structured JSON dataset documenting usage contexts, snippets, and metadata, and publish it for replication.

Experimental results

Research questions

• RQ1 RQ1: What logging smells exist in open-source ML-based systems?
• RQ2 RQ2: How do ML practitioners perceive and experience these logging smells in practice?

Key findings

• 12 ML-specific logging smells were identified, spanning security, metric management, configuration, verbosity, and context-related issues.
• Smells are widespread and vary in frequency and manifestation across projects.
• Practitioner responses show strong agreement with the taxonomy and higher perceived relevance and severity for smells impacting security, reproducibility, and experiment correctness.
• Some smells (e.g., Heavy Data Logging, Print-based Logging) are context-dependent, reflecting performance vs observability trade-offs.
• A publicly released dataset of 2,448 logging smell instances supports automated detection and AI-assisted repair research.