[Paper Review] A Heuristic Approach to Protocol Tuning for High Performance Data Transfers

Engin Arslan, Tevfik Kosar|arXiv (Cornell University)|Aug 17, 2017

Advanced Data Storage Technologies20 references3 citations

TL;DR

This paper proposes three dynamic heuristic algorithms for optimizing application-level protocol parameters—parallelism, pipelining, and concurrency—to maximize data transfer throughput in wide-area networks. By adaptively tuning these parameters based on network and system conditions, the algorithms achieve up to 10x higher throughput than baseline configurations and 7x over state-of-the-art solutions, significantly improving network utilization and transfer efficiency without requiring kernel-level changes.

ABSTRACT

Obtaining optimal data transfer performance is of utmost importance to today's data-intensive distributed applications and wide-area data replication services. Doing so necessitates effectively utilizing available network bandwidth and resources, yet in practice transfers seldom reach the levels of utilization they potentially could. Tuning protocol parameters such as pipelining, parallelism, and concurrency can significantly increase utilization and performance, however determining the best settings for these parameters is a difficult problem, as network conditions can vary greatly between sites and over time. Nevertheless, it is an important problem, since poor tuning can cause either under- or over-utilization of network resources and thus degrade transfer performance. In this paper, we present three algorithms for application-level tuning of different protocol parameters for maximizing transfer throughput in wide-area networks. Our algorithms dynamically tune the number of parallel data streams per file (for large file optimization), the level of control channel pipelining (for small file optimization), and the number of concurrent file transfers to increase I/O throughput (a technique useful for all types of files). The proposed heuristic algorithms improve the transfer throughput up to 10x compared to the baseline and 7x compared to the state of the art solutions.

Motivation & Objective

• To address the persistent underutilization of network bandwidth in high-performance data transfers despite available high-speed infrastructure.
• To overcome the challenge of selecting optimal protocol parameter settings (parallelism, pipelining, concurrency) that vary with dynamic network and system conditions.
• To design client-side, user-space algorithms that dynamically optimize transfer performance without requiring server or kernel-level modifications.
• To improve throughput for heterogeneous datasets containing both small and large files through adaptive parameter estimation and scheduling.
• To outperform existing solutions like Globus Online, which use static, non-adaptive parameter configurations.

Proposed method

• The proposed algorithms dynamically estimate optimal values for the number of parallel data streams per file based on bandwidth and RTT measurements.
• A heuristic approach is used to tune control channel pipelining levels, particularly beneficial for small file transfers.
• The concurrency level is adaptively adjusted to balance I/O throughput and network resource utilization across multiple file transfers.
• The algorithms employ a feedback-driven, online tuning mechanism that adjusts parameters in real time based on observed transfer performance.
• A channel reallocation strategy is implemented to reassign idle channels to ongoing small-file transfers, minimizing idle time and improving overall throughput.
• The algorithms are evaluated using real and synthetic datasets across diverse wide-area and local-area network topologies with varying file size distributions.

Experimental results

Research questions

• RQ1 How can protocol parameters such as parallelism, pipelining, and concurrency be dynamically tuned to maximize data transfer throughput in wide-area networks?
• RQ2 What is the impact of file size heterogeneity on the effectiveness of dynamic parameter tuning, particularly for datasets dominated by small files?
• RQ3 How do adaptive parameter settings compare to static configurations used in systems like Globus Online in terms of throughput and network utilization?
• RQ4 To what extent can heuristic-based parameter estimation improve transfer performance across diverse network conditions and system configurations?
• RQ5 What role does TCP buffer size play in limiting throughput, and how can concurrency be leveraged to mitigate buffer-related bottlenecks?

Key findings

• The proposed algorithms achieve up to 10x higher throughput compared to baseline configurations and 7x over state-of-the-art solutions like Globus Online.
• For the Dark Energy Survey dataset, MC and ProMC achieve up to 8.6 Gbps, significantly outperforming other methods, especially under high-concurrency settings.
• In small-file-dominated genome sequencing datasets, MC and ProMC achieve 1.5–3.5 Gbps, with ProMC outperforming MC by up to 10% due to superior channel allocation for small files.
• In local-area network experiments, Globus Online achieved only 500 Mbps, while the proposed algorithms achieved over 2 Gbps, highlighting the performance gap due to heavy client-server communication overhead.
• The algorithms effectively mitigate the impact of suboptimal TCP buffer sizes by increasing concurrency, which cumulatively increases allocated buffer space and improves throughput.
• The performance gains are consistent across diverse network topologies and file size distributions, demonstrating the robustness of the heuristic tuning approach.