[Paper Review] Intel nGraph: An Intermediate Representation, Compiler, and Executor for Deep Learning

Scott Cyphers, Arjun K. Bansal+19 more|arXiv (Cornell University)|Jan 24, 2018

Parallel Computing and Optimization Techniques2 references105 citations

TL;DR

The paper presents Intel nGraph, a framework-bridging intermediate representation and compiler-executor stack to optimize deep learning performance across frameworks and hardware backends.

ABSTRACT

The Deep Learning (DL) community sees many novel topologies published each year. Achieving high performance on each new topology remains challenging, as each requires some level of manual effort. This issue is compounded by the proliferation of frameworks and hardware platforms. The current approach, which we call "direct optimization", requires deep changes within each framework to improve the training performance for each hardware backend (CPUs, GPUs, FPGAs, ASICs) and requires $\mathcal{O}(fp)$ effort; where $f$ is the number of frameworks and $p$ is the number of platforms. While optimized kernels for deep-learning primitives are provided via libraries like Intel Math Kernel Library for Deep Neural Networks (MKL-DNN), there are several compiler-inspired ways in which performance can be further optimized. Building on our experience creating neon (a fast deep learning library on GPUs), we developed Intel nGraph, a soon to be open-sourced C++ library to simplify the realization of optimized deep learning performance across frameworks and hardware platforms. Initially-supported frameworks include TensorFlow, MXNet, and Intel neon framework. Initial backends are Intel Architecture CPUs (CPU), the Intel(R) Nervana Neural Network Processor(R) (NNP), and NVIDIA GPUs. Currently supported compiler optimizations include efficient memory management and data layout abstraction. In this paper, we describe our overall architecture and its core components. In the future, we envision extending nGraph API support to a wider range of frameworks, hardware (including FPGAs and ASICs), and compiler optimizations (training versus inference optimizations, multi-node and multi-device scaling via efficient sub-graph partitioning, and HW-specific compounding of operations).

Motivation & Objective

• Motivate the need for a framework- and backend-agnostic path to accelerate deep learning workloads.
• Describe the nGraph intermediate representation and its graph-based structure.
• Explain framework bridges that translate frontend graphs into the nGraph IR.
• Outline the transformer backends and how they generate optimized code for CPUs, NNPs, and GPUs.
• Discuss future directions for expanding framework, hardware, and optimization coverage.

Proposed method

• Define a framework-agnostic IR as a directed acyclic graph of stateless operation nodes with inputs, outputs, and attributes.
• Describe framework bridges that convert frontend computation graphs (e.g., TensorFlow, MXNet, neon) into the nGraph IR.
• Explain transformers that compile IR for specific backends and provide memory management, layout handling, and kernel selection.
• Detail backend-specific transformers for CPU (MKL-DNN), NNP, and NVIDIA GPUs (cuDNN, LLVM/PTX).
• Discuss support for collective and point-to-point communications in the graph through transformers (MPI or optimized methods).
• Propose interoperability with ONNX and plans for broader framework and hardware support in future work.

Experimental results

Research questions

• RQ1 How can a framework-independent IR enable optimized deep learning execution across multiple backends?
• RQ2 What is the role of framework bridges in translating frontend graphs to the nGraph IR?
• RQ3 How do backend transformers optimize code generation for CPU, NNP, and GPU backends?
• RQ4 What are the potential extensions to support training and multi-node/multi-device scaling?
• RQ5 How can nGraph interoperate with evolving standards and other compiler/IR efforts in deep learning?

Key findings

• nGraph provides a framework-bridged IR that enables backends to execute the same computation across CPUs, NNPs, and GPUs.
• Transformers generate backend-optimized code, integrating with libraries like MKL-DNN and cuDNN to leverage hardware capabilities.
• The IR is a directed acyclic graph of stateless operation nodes with adaptable data layouts and attributes for optimization.
• nGraph supports end-to-end compilation and execution workflows via framework bridges that map frontend graphs to IR.
• There is a vision for broader interoperability (e.g., ONNX) and expansion to additional frameworks and hardware in future work.