[Paper Review] Jointly Optimizing Preprocessing and Inference for DNN-based Visual Analytics

Daniel Kang, Ankit Mathur+3 more|arXiv (Cornell University)|Jul 25, 2020

Visual Attention and Saliency Detection51 references4 citations

TL;DR

This paper proposes Smol, a system that jointly optimizes preprocessing and inference in DNN-based visual analytics by leveraging low-resolution input data and a hardware-aware, pipelined runtime engine. By rethinking preprocessing as a tunable optimization surface, Smol achieves up to 5.9× end-to-end throughput improvement over prior work while maintaining fixed accuracy.

ABSTRACT

While deep neural networks (DNNs) are an increasingly popular way to query large corpora of data, their significant runtime remains an active area of research. As a result, researchers have proposed systems and optimizations to reduce these costs by allowing users to trade off accuracy and speed. In this work, we examine end-to-end DNN execution in visual analytics systems on modern accelerators. Through a novel measurement study, we show that the preprocessing of data (e.g., decoding, resizing) can be the bottleneck in many visual analytics systems on modern hardware. To address the bottleneck of preprocessing, we introduce two optimizations for end-to-end visual analytics systems. First, we introduce novel methods of achieving accuracy and throughput trade-offs by using natively present, low-resolution visual data. Second, we develop a runtime engine for efficient visual DNN inference. This runtime engine a) efficiently pipelines preprocessing and DNN execution for inference, b) places preprocessing operations on the CPU or GPU in a hardware- and input-aware manner, and c) efficiently manages memory and threading for high throughput execution. We implement these optimizations in a novel system, Smol, and evaluate Smol on eight visual datasets. We show that its optimizations can achieve up to 5.9x end-to-end throughput improvements at a fixed accuracy over recent work in visual analytics.

Motivation & Objective

• To address the growing bottleneck of preprocessing in end-to-end DNN inference on modern accelerators like the NVIDIA T4.
• To challenge the long-standing assumption that DNN execution dominates inference cost, showing preprocessing often dominates instead.
• To enable new accuracy-throughput trade-offs by treating input resolution and preprocessing placement as first-class optimization parameters.
• To design a cost model that accounts for preprocessing overhead, improving query plan selection in visual analytics systems.
• To build a runtime engine that efficiently pipelines preprocessing and DNN execution across CPU and GPU, optimizing memory and threading.

Proposed method

• Introduces a preprocessing-aware cost model that estimates end-to-end inference cost by accounting for both preprocessing and DNN execution times.
• Proposes using natively available low-resolution visual data (e.g., thumbnails, partial decodes) to reduce preprocessing cost while maintaining accuracy via augmented DNN training.
• Designs a runtime engine that dynamically places preprocessing operations on CPU or GPU based on hardware characteristics and input data.
• Employs a pipelined execution model that overlaps preprocessing and DNN inference to hide latency and improve throughput.
• Uses efficient memory management and thread scheduling to maximize utilization of modern CPU and GPU resources.
• Implements a system, Smol, that integrates these optimizations into a single, end-to-end inference engine for visual analytics.

Experimental results

Research questions

• RQ1 Is preprocessing, rather than DNN execution, the dominant bottleneck in end-to-end DNN inference on modern inference-optimized accelerators?
• RQ2 Can using low-resolution or partially decoded input data enable significant throughput improvements without sacrificing accuracy?
• RQ3 How can preprocessing and DNN execution be jointly optimized through hardware-aware task placement and pipelining?
• RQ4 Can a preprocessing-aware cost model improve query plan selection in visual analytics systems compared to traditional DNN-only cost models?
• RQ5 To what extent can joint optimization of preprocessing and inference reduce end-to-end latency and improve throughput in real-world visual analytics workloads?

Key findings

• On the inference-optimized g4dn.xlarge AWS instance with a T4 GPU, preprocessing costs are 9× higher than DNN execution for ResNet-50, making it the primary bottleneck.
• Preprocessing consumes approximately 2.3× more power and costs 11× more than DNN execution on the same instance, highlighting the energy and cost inefficiency of ignoring preprocessing.
• Smol achieves up to 5.9× end-to-end throughput improvement over recent visual analytics systems at fixed accuracy across eight visual datasets.
• The use of low-resolution data combined with augmented DNN training recovers accuracy while reducing preprocessing cost, enabling new accuracy-throughput trade-offs.
• The proposed preprocessing-aware cost model is more accurate than prior models, which incorrectly assumed DNN execution dominated inference time.
• The runtime engine’s dynamic CPU/GPU placement and pipelining achieve high throughput by efficiently managing memory and threading across heterogeneous hardware.