[Paper Review] A Grammar of Machine Learning Workflows

Roth, Simon|arXiv (Cornell University)|Mar 11, 2026

Scientific Computing and Data Management0 citations

TL;DR

A grammar-based, typestate framework embeds leakage-prevention in the supervised learning lifecycle by enforcing four call-time constraints on eight kernel primitives, enabling reject-at-call workflows to prevent data leakage.

ABSTRACT

Data leakage affected 294 published papers across 17 scientific fields (Kapoor & Narayanan, 2023); a living survey has since grown that count to 648 across 30 fields. The dominant response has been documentation: checklists, linters, best-practice guides. Documentation reduces errors but does not close structural failures. This paper proposes a structural remedy: a grammar that decomposes the supervised learning lifecycle into 8 kernel primitives connected by a typed directed acyclic graph (DAG), with four hard constraints that reject the two most damaging leakage classes at call time. The grammar's core contribution is the terminal assess constraint: a runtime-enforced evaluate/assess boundary where repeated test-set assessment is rejected by a guard on a nominally distinct Evidence type. A companion study across 2,047 experimental instances quantifies why this matters: selection leakage inflates performance by d_z = 0.93 and memorization leakage by d_z = 0.53-1.11. Two maintained implementations (Python, R) demonstrate the claims. The appendix specification lets anyone build a conforming version.

Motivation & Objective

• Motivate data leakage as a structural problem that documentation cannot fix.
• Propose a compositional grammar that decomposes the ML lifecycle into typed primitives with guards.
• Embed leakage-prevention rules into a call-time enforcement mechanism to reject invalid workflows.
• Demonstrate the grammar with implementations in Python and R and provide a portability appendix.

Proposed method

• Define eight kernel primitives that govern the core ML lifecycle (split, cv, prepare, fit, predict, evaluate, explain, assess).
• Construct a typed directed acyclic graph (type DAG) connecting primitives to model data flow and enforce a terminal assess boundary.
• Introduce four hard, empirically grounded constraints that reject leakage at call time (assess-once, per-fold prepare after split, type-safe transitions, no unregistered data into fit).
• Explain the terminal boundary between evaluate (formative) and assess (summative) and how this separation is enforced by guards.
• Provide two language implementations (Python and R) and a Julia internal proof-of-concept to show portability and conformance to the specification.
• Offer conformance conditions to verify implementations against the type DAG and four constraints.

Experimental results

Research questions

• RQ1 Can data leakage be rejected at the structural level by a grammar rather than detected post hoc?
• RQ2 What are the minimal primitives and constraints needed to cover the supervised learning lifecycle without enabling leakage?
• RQ3 How do guard-based checks at call time compare to post-hoc leakage detectors in preventing invalid workflows?
• RQ4 Do terminal assess-once constraints effectively prevent repeated test-set evaluation across models?
• RQ5 How do implementations in Python and R reflect the formal specification and remain portable?

Key findings

• The grammar enforces four hard constraints that prevent major leakage classes at call time and makes repeated test-set assessment detectable as a terminal boundary.
• Preliminary studies show Class II leakage effects (selection bias) are large, supporting the need for structural prevention; some effects (e.g., stacking leakage) were falsified in the companion study.
• The terminal assess-once constraint is effective in practice, with evidence of inflation when seeds and selection strategies are used, and with two of three predictions confirmed.
• Two concrete implementations (Python and R) conform to the eight primitives and four constraints, and a third (Julia) serves as portability proof.
• The appendix specification enables independent reimplementation and verification of the grammar and guards.