[Paper Review] Glow: Graph Lowering Compiler Techniques for Neural Networks

Nadav Rotem, Jordan Fix+16 more|arXiv (Cornell University)|May 2, 2018

Parallel Computing and Optimization TechniquesComputer Science13 references186 citations

TL;DR

Glow is an open-source machine learning compiler that lowers neural network graphs through multiple IR stages to target heterogeneous hardware, enabling target-independent optimizations and backend-specific code generation.

ABSTRACT

This paper presents the design of Glow, a machine learning compiler for heterogeneous hardware. It is a pragmatic approach to compilation that enables the generation of highly optimized code for multiple targets. Glow lowers the traditional neural network dataflow graph into a two-phase strongly-typed intermediate representation. The high-level intermediate representation allows the optimizer to perform domain-specific optimizations. The lower-level instruction-based address-only intermediate representation allows the compiler to perform memory-related optimizations, such as instruction scheduling, static memory allocation and copy elimination. At the lowest level, the optimizer performs machine-specific code generation to take advantage of specialized hardware features. Glow features a lowering phase which enables the compiler to support a high number of input operators as well as a large number of hardware targets by eliminating the need to implement all operators on all targets. The lowering phase is designed to reduce the input space and allow new hardware backends to focus on a small number of linear algebra primitives.

Motivation & Objective

• Motivate the need for domain-specific compilation due to limits of Moore’s Law and increasing ML compute demand.
• Propose Glow as a retargetable compiler that produces optimized code for multiple hardware backends.
• Introduce a two-phase, strongly-typed intermediate representation for high-level graph optimizations and low-level memory optimizations.
• Describe quantization and profiling approaches to enable efficient integer arithmetic on diverse devices.
• Present a runtime and partitioning mechanism to execute graphs across multiple accelerators.

Proposed method

• Introduce a two-phase strongly-typed intermediate representation (high-level dataflow IR and low-level address-only IR).
• Lower high-level operators into low-level linear algebra primitives to enable backend simplification (node lowering).
• Apply target-independent optimizations followed by target-specific optimizations and code generation (IRGen).
• Implement quantization with profile-guided ranges to convert floating-point networks to int8 representations.
• Use ClassGen to automatically generate IR instruction and node classes to ease backend development.
• Provide a CPU backend with a small, target-independent standard library and operator stacking to fuse data-parallel operations.
• Offer a Glow runtime that partitions graphs across devices, loads sub-graphs onto accelerators, and manages asynchronous inference requests.

Experimental results

Research questions

• RQ1 How can a retargetable compiler for neural networks efficiently map high-level graphs to multiple hardware backends?
• RQ2 What role do multi-level IRs and lowering play in enabling backend-specific optimizations without duplicating operator implementations?
• RQ3 How effective are quantization and profile-guided techniques in producing accurate, efficient integer computations on diverse devices?
• RQ4 What runtime strategies (partitioning, provisioning, device management) best utilize multiple accelerators for a given model?

Key findings

• Glow provides a scalable lowering approach that reduces the need to implement all operators for every backend by targeting a small set of linear algebra primitives.
• The two-level IR design enables target-independent optimizations and hardware-specific code generation, improving backend flexibility.
• Quantization with profile-guided ranges enables conversion to int8 and islands of integer computation, facilitating efficient inference.
• A CPU backend with operator stacking and a small standard library achieves fused kernels and improved memory locality, demonstrated on models like Resnet50 and VGG19.
• Glow’s runtime supports partitioning and execution across multiple accelerators, coordinating sub-graph loading and asynchronous inference.
• Evaluation compares Glow with TensorFlow and TVM on real models to assess performance, using a Kaby Lake CPU as the test platform.