[Paper Review] Data Science and Digital Systems: The 3Ds of Machine Learning Systems Design

Neil D. Lawrence|arXiv (Cornell University)|Mar 26, 2019

Machine Learning and Data Classification7 references4 citations

TL;DR

This paper proposes a systematic 3D framework—Decomposition, Data, and Deployment—for designing machine learning systems, emphasizing data-first architecture and continuous model monitoring to ensure robust, scalable AI deployment. The approach shifts focus from model-centric development to end-to-end system design that prioritizes data quality, task modularity, and production resilience through continuous testing and streaming infrastructure.

ABSTRACT

Machine learning solutions, in particular those based on deep learning methods, form an underpinning of the current revolution in "artificial intelligence" that has dominated popular press headlines and is having a significant influence on the wider tech agenda. Here we give an overview of the 3Ds of ML systems design: Data, Design and Deployment. By considering the 3Ds we can move towards \emph{data first} design.

Motivation & Objective

• Address the challenge of deploying machine learning systems effectively in both greenfield and brownfield environments.
• Overcome the 'Solow paradox' in AI by ensuring machine learning innovations translate into measurable productivity gains.
• Propose a structured design methodology to shift from model-centric to data-first system development.
• Enable continuous model deployment and monitoring through progression testing and hypervisor systems.
• Promote organizational change toward data-as-a-service culture with shared data pipelines and standardized data readiness levels.

Proposed method

• Apply a 3D framework: Decomposition (breaking down complex tasks into automatable sub-tasks), Data (ensuring data quality and readiness), and Deployment (continuous integration and monitoring).
• Use task decomposition to identify repetitive, well-defined sub-tasks suitable for supervised learning, with emphasis on feature engineering and model interpretability.
• Implement data curation at source to create consumable data streams, avoiding repeated cleaning per task—advocating for data-first architecture.
• Introduce 'progression testing' as a form of continual regression testing to detect concept drift and performance degradation in deployed models.
• Employ hypervisor systems—such as surrogate models or emulators—to monitor model behavior and trigger retraining when system conditions change.
• Adopt streaming architectures (e.g., Apache Kafka) to enable stateless, asynchronous, and persistent data processing for real-time model evaluation and observability.

Experimental results

Research questions

• RQ1 How can machine learning systems be systematically designed to ensure scalability and reliability in production environments?
• RQ2 What role does data quality and data readiness play in the success of machine learning deployments?
• RQ3 How can continuous monitoring and testing be integrated into ML deployment pipelines to maintain model performance over time?
• RQ4 What architectural patterns support a data-first approach that enables reuse across multiple teams and services?
• RQ5 How can organizations transition from model-centric to data-as-a-service culture to improve deployment efficiency and maintainability?

Key findings

• The 3D framework—Decomposition, Data, and Deployment—provides a structured approach to machine learning systems design that improves system robustness and scalability.
• Data should be cleaned at the output stage rather than at input, enabling reusable, high-quality data streams across teams.
• Progression testing is essential for detecting concept drift and performance degradation in deployed models, ensuring long-term reliability.
• Hypervisor systems using emulation or surrogate modeling can monitor model behavior and trigger retraining when environmental changes invalidate assumptions.
• Streaming architectures like Apache Kafka support stateless, persistent data pipelines that enable real-time dashboards, anomaly detection, and efficient system observability.
• Organizations benefit most when they re-architect systems around data-as-a-service principles, with shared data pipelines and standardized data readiness levels.