[Paper Review] Neural Network Distiller: A Python Package For DNN Compression Research

Neta Zmora, Guy Jacob+3 more|arXiv (Cornell University)|Oct 27, 2019

Computational Physics and Python ApplicationsComputer Science39 references52 citations

TL;DR

This paper introduces Neural Network Distiller, an open-source Python package in PyTorch that provides a library of DNN compression algorithms, scheduling tools, and tutorials to enable easy experimentation and extension in model compression research.

ABSTRACT

This paper presents the philosophy, design and feature-set of Neural Network Distiller, an open-source Python package for DNN compression research. Distiller is a library of DNN compression algorithms implementations, with tools, tutorials and sample applications for various learning tasks. Its target users are both engineers and researchers, and the rich content is complemented by a design-for-extensibility to facilitate new research. Distiller is open-source and is available on Github at https://github.com/NervanaSystems/distiller.

Motivation & Objective

• Motivate the need for standardized, repeatable DNN compression research for edge-device inference.
• Provide an extensible, easy-to-use library of compression algorithms and tooling for researchers and engineers.
• Enable rapid experimentation by interleaving training and compression via a scheduling system and YAML configurations.

Proposed method

• Implement core compression building blocks (pruning, quantization, regularization) within a Python/PyTorch framework.
• Provide additional techniques (low-rank decomposition, conditional computation, knowledge distillation, automated compression) and integration with existing models/datasets.
• Introduce a compression-scheduling subsystem with a YAML-based recipe parser to interleave training and compression according to user-defined schedules.
• Offer utilities for logging, checkpoints, data loading, model summaries, and export to ONNX to support experimentation and deployment workflows.
• Demonstrate applicability across tasks (image classification, recommender systems, NLP, object detection) using models from Torchvision and Cadene, with easy extensibility for new models/tasks.

Experimental results

Research questions

• RQ1 How can researchers test and compare DNN compression methods under the same experimental conditions (datasets, preprocessing, hyper-parameters, execution environment)?
• RQ2 How can compression methods be effectively interleaved with training to achieve iterative pruning, quantization, or other compressions?
• RQ3 Can Distiller support mixing compression methods (e.g., quantization and pruning) and provide a flexible framework to prototype new ideas?
• RQ4 What is the scope and practicality of Distiller’s scheduling and utilities for rapid research prototyping and knowledge sharing?

Key findings

• Distiller aggregates a broad set of compression methods (pruning, quantization, regularization) and related techniques (low-rank, distillation) into a single library.
• It provides a compression-scheduling subsystem and YAML-based configuration to control training and compression iterations programmatically or via config files.
• The toolkit is designed for easy integration with PyTorch, Torchvision models, and Cadene models, and includes tutorials, notebooks, and documentation to accelerate research.
• Distiller supports multiple tasks, including image classification, recommender systems, NLP, and object detection, demonstrating its versatility across domains.
• The authors acknowledge current limitations, such as simulating INT8 with FP32 operations and the absence of ONNX/Glow export in the released version, with plans to align with native PyTorch quantization.