[Paper Review] AI Tax: The Hidden Cost of AI Data Center Applications

Daniel Richins, Dharmisha Doshi+10 more|arXiv (Cornell University)|Jul 21, 2020

CCD and CMOS Imaging Sensors22 references5 citations

TL;DR

This paper introduces the concept of 'AI tax'—the hidden computational, storage, and network overheads imposed by pre- and post-processing in AI-centric edge data center applications. By deploying a real-time Face Recognition system using open-source tools and AI accelerators, the authors demonstrate that these non-inference workloads create significant bottlenecks, leading to a 15% lower total cost of ownership (TCO) when using a purpose-built, heterogeneous edge data center versus a homogeneous server setup.

ABSTRACT

Artificial intelligence and machine learning are experiencing widespread adoption in industry and academia. This has been driven by rapid advances in the applications and accuracy of AI through increasingly complex algorithms and models; this, in turn, has spurred research into specialized hardware AI accelerators. Given the rapid pace of advances, it is easy to forget that they are often developed and evaluated in a vacuum without considering the full application environment. This paper emphasizes the need for a holistic, end-to-end analysis of AI workloads and reveals the "AI tax." We deploy and characterize Face Recognition in an edge data center. The application is an AI-centric edge video analytics application built using popular open source infrastructure and ML tools. Despite using state-of-the-art AI and ML algorithms, the application relies heavily on pre-and post-processing code. As AI-centric applications benefit from the acceleration promised by accelerators, we find they impose stresses on the hardware and software infrastructure: storage and network bandwidth become major bottlenecks with increasing AI acceleration. By specializing for AI applications, we show that a purpose-built edge data center can be designed for the stresses of accelerated AI at 15% lower TCO than one derived from homogeneous servers and infrastructure.

Motivation & Objective

• To expose the often-ignored performance and cost overheads introduced by pre- and post-processing in AI workloads.
• To demonstrate that AI accelerators alone do not eliminate system-level bottlenecks, especially in storage and network I/O.
• To advocate for end-to-end benchmarking that captures the full application pipeline, not just AI kernel performance.
• To show that purpose-built, heterogeneous edge data centers can achieve 15% lower TCO than homogeneous server deployments for AI workloads.
• To highlight the role of communication frameworks like Apache Kafka in introducing latency and resource contention in streaming AI applications.

Proposed method

• Deployed a real-world, open-source Face Recognition application using the FaceNet model in an edge data center environment.
• Used state-of-the-art AI accelerators and measured end-to-end latency and system throughput across all pipeline stages.
• Characterized the full application stack, including data ingestion, pre-processing, model inference, post-processing, and inter-component communication.
• Analyzed network and storage I/O bottlenecks using Apache Kafka as the underlying communication substrate for streaming workloads.
• Designed and evaluated a purpose-built edge data center with heterogeneous hardware (CPU, GPU, and AI accelerators) to optimize for AI workloads.
• Compared TCO between homogeneous server clusters and purpose-built edge data centers under identical AI workload conditions.

Experimental results

Research questions

• RQ1 What are the primary sources of latency and resource overhead in end-to-end AI inference pipelines beyond the AI kernel computation?
• RQ2 How do pre- and post-processing stages impact system-wide performance and infrastructure costs in AI-centric applications?
• RQ3 To what extent do storage and network I/O become bottlenecks as AI acceleration increases in edge data centers?
• RQ4 Can a purpose-built, heterogeneous edge data center achieve lower total cost of ownership (TCO) than a homogeneous server-based deployment for AI workloads?
• RQ5 How does the use of standard communication frameworks like Apache Kafka affect end-to-end latency and resource utilization in streaming AI applications?

Key findings

• Pre- and post-processing stages consume a significant portion of the total request latency and rely heavily on CPU, creating a major performance bottleneck despite AI acceleration.
• Storage and network bandwidth emerge as primary bottlenecks in AI-accelerated workloads, especially under high-throughput streaming conditions.
• The use of Apache Kafka as a communication substrate introduces measurable broker waiting times and adds to system latency, particularly under high load.
• Even with high-performing AI accelerators, the overall system throughput is limited by non-AI components, validating the existence of an 'AI tax'.
• A purpose-built, heterogeneous edge data center reduces total cost of ownership (TCO) by 15% compared to a homogeneous server-based deployment for the same AI workload.
• End-to-end application performance is significantly underestimated when benchmarks focus only on AI kernel performance, highlighting the need for holistic, full-stack evaluation.