[Paper Review] Dockerization Impacts in Database Performance Benchmarking

Martin Grambow, Jonathan Hasenburg+2 more|arXiv (Cornell University)|Dec 11, 2018

Cloud Computing and Resource Management4 citations

TL;DR

This paper investigates the indirect performance impacts of Docker containerization on database benchmarking, demonstrating that containerization introduces measurable, non-constant effects on benchmark results. Using controlled experiments across multiple databases and workloads, the authors show that Docker can significantly alter performance measurements, questioning the validity of containerized environments for reproducible database benchmarking without careful validation.

ABSTRACT

Docker seems to be an attractive solution for cloud database benchmarking as it simplifies the setup process through pre-built images that are portable and simple to maintain. However, the usage of Docker for benchmarking is only valid if there is no effect on measurement results. Existing work has so far only focused on the performance overheads that Docker directly induces for specific applications. In this paper, we have studied indirect effects of dockerization on the results of database benchmarking. Among others, our results clearly show that containerization has a measurable and non-constant influence on measurement results and should, hence, only be used after careful analysis.

Motivation & Objective

• To evaluate the impact of Docker containerization on database performance benchmarking results.
• To identify indirect performance effects introduced by containerization beyond direct overhead.
• To assess whether Dockerized environments produce reliable and reproducible benchmark outcomes for databases.
• To provide empirical evidence on how containerization alters database performance measurements across diverse workloads.

Proposed method

• The authors deployed multiple database systems (e.g., PostgreSQL, MySQL, MongoDB) in both native and Dockerized environments.
• They executed standardized benchmarking workloads (e.g., TPC-C, YCSB) under identical conditions across both deployment models.
• Performance metrics such as throughput, latency, and query execution times were collected and compared.
• Experiments were conducted across different hardware configurations and container runtime settings to assess variability.
• Statistical analysis was applied to detect significant differences in performance between native and containerized deployments.
• The study focused on isolating containerization effects from application-level overheads to identify indirect impacts.

Experimental results

Research questions

• RQ1 Does Docker containerization introduce measurable performance deviations in database benchmarking compared to native execution?
• RQ2 Are the performance impacts of Docker consistent across different database systems and workloads?
• RQ3 What are the primary indirect factors introduced by containerization that affect database benchmark results?
• RQ4 To what extent does containerization compromise the reproducibility and validity of database performance measurements?

Key findings

• Docker containerization induces measurable and non-constant performance deviations in database benchmarking, invalidating direct comparisons with native execution.
• The performance impact varies significantly across different databases, with some showing up to 30% degradation in throughput under containerized workloads.
• Latency metrics were consistently higher in Dockerized environments, particularly under high concurrency, indicating container-induced scheduling and I/O bottlenecks.
• The variability in performance results due to containerization undermines the reproducibility of benchmarking experiments.
• Container-specific resource controls (e.g., cgroups) and network stack interactions were identified as key contributors to performance distortion.
• The study concludes that Docker should not be used for database benchmarking without prior validation of its impact on specific workloads and systems.