[Paper Review] NSML: A Machine Learning Platform That Enables You to Focus on Your Models

Nako Sung, Minkyu Kim+9 more|arXiv (Cornell University)|Dec 16, 2017

Machine Learning and Data Classification13 references70 citations

TL;DR

NSML automates resource management, data handling, experiment tracking, and leaderboard comparisons to let researchers focus on model development; a prototype on 80 GPUs showed efficiency gains in alpha tests.

ABSTRACT

Machine learning libraries such as TensorFlow and PyTorch simplify model implementation. However, researchers are still required to perform a non-trivial amount of manual tasks such as GPU allocation, training status tracking, and comparison of models with different hyperparameter settings. We propose a system to handle these tasks and help researchers focus on models. We present the requirements of the system based on a collection of discussions from an online study group comprising 25k members. These include automatic GPU allocation, learning status visualization, handling model parameter snapshots as well as hyperparameter modification during learning, and comparison of performance metrics between models via a leaderboard. We describe the system architecture that fulfills these requirements and present a proof-of-concept implementation, NAVER Smart Machine Learning (NSML). We test the system and confirm substantial efficiency improvements for model development.

Motivation & Objective

• Define practical requirements for efficient deep learning research from a large online ML community.
• Design and implement a deep learning research platform (NSML) that automates ancillary tasks.
• Demonstrate NSML on real-world tasks to show efficiency gains and usability.

Proposed method

• Centralized scheduler with master-slave architecture for resource allocation and a leader election mechanism to avoid SPOF.
• Containerized storage and ML containers to separate data/storage from model environments and enable library isolation.
• Web-based UI and NSML-CLI for user interaction, with automatic environment packaging and dataset mounting.
• Support for multiple deep learning libraries (TensorFlow, PyTorch, MXNet) via Python APIs and containerized environments.
• Hyperparameter tuning and model evaluation during training using a REPL-like workflow and a Kaggle-like leaderboard for comparisons.
• Alpha-test demonstrations on MNIST, GAN-based face generation, BiLSTM movie rating prediction, and CNN-based facial emotion recognition.

Experimental results

Research questions

• RQ1 How can ML research workflows be streamlined to reduce manual tasks like GPU allocation and experiment tracking?
• RQ2 Can a platform support multiple libraries and serverless usage while maintaining reproducibility?
• RQ3 What impact does automated resource management and leaderboard-driven comparison have on research efficiency?
• RQ4 How effective is NSML in real-world tasks across different ML domains?

Key findings

• NSML provides automatic GPU allocation, dataset management, and experiment visualization to reduce manual labor.
• The platform supports multiple libraries (TensorFlow, PyTorch, MXNet) and both single GPUs and GPU clusters.
• A built-in leaderboard enables straightforward comparison of models and hyperparameters across datasets.
• Alpha testers reported improved usability and research efficiency, with some system instability noted during early testing.
• NSML enables live visualization and replication of experiments, aiding reproducibility and rapid iteration.
• Demonstrations on MNIST, GAN-based face generation, movie rating prediction, and facial emotion recognition showcased practical applicability.