[Paper Review] GluonCV and GluonNLP: Deep Learning in Computer Vision and Natural Language Processing

Jian Guo, He He+14 more|arXiv (Cornell University)|Jul 9, 2019

Multimodal Machine Learning ApplicationsComputer Science18 references90 citations

TL;DR

GluonCV and GluonNLP are MXNet-based toolkits that provide modular building blocks, pre-trained models, and a model zoo for rapid prototyping in computer vision and natural language processing, with cross-language deployment.

ABSTRACT

We present GluonCV and GluonNLP, the deep learning toolkits for computer vision and natural language processing based on Apache MXNet (incubating). These toolkits provide state-of-the-art pre-trained models, training scripts, and training logs, to facilitate rapid prototyping and promote reproducible research. We also provide modular APIs with flexible building blocks to enable efficient customization. Leveraging the MXNet ecosystem, the deep learning models in GluonCV and GluonNLP can be deployed onto a variety of platforms with different programming languages. The Apache 2.0 license has been adopted by GluonCV and GluonNLP to allow for software distribution, modification, and usage.

Motivation & Objective

• Provide modular APIs to reuse building blocks across CV and NLP models.
• Offer pre-trained state-of-the-art models, training scripts, and training logs for rapid prototyping and reproducibility.
• Leverage the MXNet ecosystem to enable deployment across multiple languages and platforms.
• Promote open-source development under Apache 2.0 license and foster an active community.

Proposed method

• Explain modular API design with reusable components such as data utilities, models, initializers, and losses.
• Describe data pipeline construction using bucketed batching and padding utilities to create efficient mini-batches.
• Maintain a model zoo with hundreds of pre-trained models across CV and NLP tasks.
• Demonstrate performance relative to other open-source frameworks on standard benchmarks.
• Highlight deployment flexibility across CPU, GPU, and multiple languages via MXNet bindings.

Experimental results

Research questions

• RQ1 How do modular APIs in GluonCV/NLP enable efficient prototyping and customization across CV and NLP tasks?
• RQ2 What performance gains do GluonCV/NLP models achieve compared with other open-source implementations on standard benchmarks?
• RQ3 How does cross-language deployment via MXNet affect usability and portability of CV/NLP models?

Key findings

• GluonCV/NLP match or exceed performance of other open-source implementations on major benchmarks across CV and NLP tasks.
• On ImageNet with ResNet-50, GluonCV/NLP shows notably higher top-1 accuracy (79.2%) than cited baselines.
• The model zoo provides over 200 ready-to-use models for diverse tasks.
• Evidence of practical deployment benefits from MXNet’s multi-language bindings and efficient low-level operators.
• Int8 inference examples show substantial latency reductions for models like BERT_BASE.