[Paper Review] NSML: Meet the MLaaS platform with a real-world case study

Hanjoo Kim, Minkyu Kim+10 more|arXiv (Cornell University)|Oct 8, 2018

Scientific Computing and Data ManagementDecision Sciences22 references109 citations

TL;DR

NSML is an MLaaS platform designed to simplify environment setup, resource management, collaboration, and deployment for enterprise-scale ML tasks; it demonstrates usefulness via real-world case studies and competitions.

ABSTRACT

The boom of deep learning induced many industries and academies to introduce machine learning based approaches into their concern, competitively. However, existing machine learning frameworks are limited to sufficiently fulfill the collaboration and management for both data and models. We proposed NSML, a machine learning as a service (MLaaS) platform, to meet these demands. NSML helps machine learning work be easily launched on a NSML cluster and provides a collaborative environment which can afford development at enterprise scale. Finally, NSML users can deploy their own commercial services with NSML cluster. In addition, NSML furnishes convenient visualization tools which assist the users in analyzing their work. To verify the usefulness and accessibility of NSML, we performed some experiments with common examples. Furthermore, we examined the collaborative advantages of NSML through three competitions with real-world use cases.

Motivation & Objective

• Motivate the need for an integrated ML platform that handles environment setup, dependency management, collaboration, and scalable resource utilization.
• Propose NSML as a practical MLaaS solution that supports automatic resource allocation, scheduling, sharing of data/models, monitoring, hyperparameter tuning, and leaderboards.
• Demonstrate NSML through real-world case studies and competitions to validate usability and robustness.

Proposed method

• Introduce NSML as a two-module system: resource management (scheduler, virtualization) and user interaction (CLI/web UI).
• Implement locality-aware scheduling and residual resource defragmentation to optimize data placement and GPU usage.
• Provide session and resource monitoring with alerts and dashboards for users and admins.
• Offer datasets management with on-demand dataset loading and private/public sharing for collaboration.
• Provide a web-based visualization interface (Visdom/TensorBoard) for comparing sessions and tracking progress.
• Support hyperparameter tuning in parallel (grid/random search and advanced methods like population-based training).

Experimental results

Research questions

• RQ1 How can a unified MLaaS platform improve environment setup, collaboration, and resource utilization for enterprise-scale ML tasks?
• RQ2 Can NSML's scheduling, monitoring, and visualization tools enable efficient experimentation and reproducibility across teams?
• RQ3 Do competitions and real-world case studies on NSML demonstrate usability, robustness, and potential for commercialization of ML models?
• RQ4 What are the limitations of NSML, and what future enhancements are needed (datasets versioning, multi-dataset tasks, advanced visualizations, distributed learning)?

Key findings

• NSML enables efficient resource management with a cluster of GPUs, achieving high utilization and enabling scalable experimentation.
• The platform supports collaborative work through dataset sharing, team workspaces, and reproducible sessions.
• NSML provides multiple interfaces (CLI and web) and visualization tools (Visdom/TensorBoard) for analyzing and comparing models and hyperparameters.
• Three ML competitions on NSML demonstrated usability and showed that best models could improve upon baselines in real customer services.
• Monitoring tools improved GPU utilization awareness and helped users optimize code for better resource efficiency.