[Paper Review] A Survey on Edge Benchmarking

Blesson Varghese, Nan Wang+5 more|arXiv (Cornell University)|Apr 24, 2020

Cloud Computing and Resource Management11 citations

TL;DR

This paper presents a comprehensive survey on edge benchmarking, analyzing its evolution from traditional tightly coupled systems to modern loosely coupled edge environments. It classifies existing research by system under test, benchmarking techniques, quality metrics, and runtime, offering a systematic foundation for future edge performance evaluation and adaptive system design.

ABSTRACT

Edge computing is the next Internet frontier that will leverage computing resources located near users, sensors, and data stores for delivering more responsive services. Thus, it is envisioned that a large-scale, geographically dispersed and resource-rich distributed system will emerge and become the backbone of the future Internet. However, given the loosely coupled nature of these complex systems, their operational conditions are expected to significantly change over time. In this context, the performance of these systems will need to be captured rapidly, referred to as benchmarking, for application deployment, resource orchestration, and adaptive decision-making. Edge benchmarking is a nascent research avenue that has started gaining momentum over the last five years. This article firstly examines articles published over the last three decades to trace the history of benchmarking from tightly coupled to loosely coupled systems. Then it systematically classifies research to identify the system under test, techniques analyzed, quality metrics, and benchmark runtime in edge benchmarking.

Motivation & Objective

• To trace the historical evolution of benchmarking from tightly coupled to loosely coupled systems over the past three decades.
• To identify and classify existing research in edge benchmarking based on system under test, techniques, quality metrics, and benchmark runtime.
• To establish a structured taxonomy for edge benchmarking to support application deployment, resource orchestration, and adaptive decision-making.
• To highlight the nascent but growing research momentum in edge benchmarking over the last five years.
• To provide a foundation for future work by clarifying key dimensions and open challenges in edge performance evaluation.

Proposed method

• Systematic literature review of academic articles published over the past three decades to map the evolution of benchmarking paradigms.
• Classification of edge benchmarking research along four dimensions: system under test, benchmarking techniques, quality metrics, and benchmark runtime.
• Analysis of trends in edge benchmarking by examining the types of systems tested (e.g., edge nodes, IoT devices), techniques used (e.g., synthetic, real-world workloads), and performance metrics (e.g., latency, throughput).
• Identification of recurring patterns and gaps in benchmarking practices across different edge deployment scenarios.
• Synthesis of findings into a structured framework for understanding and guiding future edge benchmarking research.

Experimental results

Research questions

• RQ1 How has the paradigm of system benchmarking evolved from tightly coupled to loosely coupled systems over the past 30 years?
• RQ2 What are the dominant types of systems under test in current edge benchmarking research?
• RQ3 Which benchmarking techniques and quality metrics are most commonly used in edge systems?
• RQ4 How does benchmark runtime vary across different edge benchmarking studies and use cases?
• RQ5 What are the key research gaps and challenges in edge benchmarking that remain unaddressed?

Key findings

• Edge benchmarking has emerged as a nascent but rapidly growing research area over the last five years, driven by the need for responsive, distributed edge services.
• The majority of edge benchmarking studies focus on edge nodes and IoT devices as the system under test, reflecting the decentralized nature of edge computing.
• Synthetic workloads and real-world application traces are the two primary benchmarking techniques, with a growing emphasis on real-world workloads for practical relevance.
• Latency and throughput are the most frequently used quality metrics, aligning with the low-latency requirements of edge applications.
• Benchmark runtime varies significantly, with some studies executing benchmarks in seconds while others require hours, depending on workload complexity and system scale.
• A clear taxonomy of edge benchmarking is lacking in current literature, indicating a need for standardized classification and evaluation frameworks.