[Paper Review] Improving Block-level Efficiency with scsi-mq

Blake Caldwell|arXiv (Cornell University)|Apr 28, 2015

Advanced Data Storage Technologies1 references3 citations

TL;DR

This paper evaluates the impact of the Linux kernel's multi-queue block layer (blk-mq) and its SCSI subsystem reimplementation (scsi-mq) on block-level I/O efficiency in a production-grade Lustre parallel filesystem. Using a 3.18 kernel with scsi-mq and caching RAID storage arrays, the study demonstrates a 13.6% reduction in SCSI write request latency and a 7x increase in CPU idle time, indicating significant improvements in I/O efficiency and scalability over the legacy single-queue block layer.

ABSTRACT

Current generation solid-state storage devices are exposing a new bottlenecks in the SCSI and block layers of the Linux kernel, where IO throughput is limited by lock contention, inefficient interrupt handling, and poor memory locality. To address these limitations, the Linux kernel block layer underwent a major rewrite with the blk-mq project to move from a single request queue to a multi-queue model. The Linux SCSI subsystem rework to make use of this new model, known as scsi-mq, has been merged into the Linux kernel and work is underway for dm-multipath support in the upcoming Linux 4.0 kernel. These pieces were necessary to make use of the multi-queue block layer in a Lustre parallel filesystem with high availability requirements. We undertook adding support of the 3.18 kernel to Lustre with scsi-mq and dm-multipath patches to evaluate the potential of these efficiency improvements. In this paper we evaluate the block-level performance of scsi-mq with backing storage hardware representative of a HPC-targerted Lustre filesystem. Our findings show that SCSI write request latency is reduced by as much as 13.6%. Additionally, when profiling the CPU usage of our prototype Lustre filesystem, we found that CPU idle time increased by a factor of 7 with Linux 3.18 and blk-mq as compared to a standard 2.6.32 Linux kernel. Our findings demonstrate increased efficiency of the multi-queue block layer even with disk-based caching storage arrays used in existing parallel filesystems.

Motivation & Objective

• To evaluate the performance benefits of the Linux multi-queue block layer (blk-mq) and its SCSI subsystem integration (scsi-mq) in a real-world HPC storage environment.
• To assess whether the efficiency improvements from blk-mq and scsi-mq translate to tangible performance gains in a production parallel filesystem like Lustre.
• To identify scalability bottlenecks in existing I/O stacks when using caching storage arrays with high IOP workloads.
• To investigate the impact of multi-queue I/O on CPU utilization and latency in a metadata- and data-intensive filesystem workload.
• To explore the potential of scsi-mq for improving latency-sensitive operations in high-availability, high-throughput storage systems.

Proposed method

• Ported the Lustre filesystem to the Linux 3.18 kernel with custom patches enabling scsi-mq and dm-multipath support.
• Conducted block-level I/O throughput and latency measurements using representative I/O patterns from a production Lustre filesystem.
• Profiled kernel CPU usage via sampled stack traces to measure idle time and identify performance bottlenecks in the I/O path.
• Used caching RAID storage arrays with rotational media to simulate real-world OST (Object Storage Target) workloads common in HPC environments.
• Compared results between the legacy single-queue block layer (Linux 2.6.32) and the new multi-queue architecture (Linux 3.18 with scsi-mq).
• Evaluated both write and read I/O workloads, with a focus on write latency and CPU efficiency improvements.

Experimental results

Research questions

• RQ1 To what extent does the multi-queue block layer (blk-mq) and its SCSI subsystem implementation (scsi-mq) reduce SCSI write request latency in a real HPC storage stack?
• RQ2 How does the use of scsi-mq affect CPU utilization and idle time in a high-I/O workload typical of a Lustre filesystem?
• RQ3 Can the performance improvements from blk-mq and scsi-mq be observed when using caching storage arrays with rotational media, rather than synthetic or SSD-based benchmarks?
• RQ4 Why does multi-channel support in the SRP driver degrade performance despite the expected benefits of increased parallelism?
• RQ5 To what extent are metadata operations, such as journal replay during recovery, limited by I/O throughput and latency in the current block layer?

Key findings

• SCSI write request latency was reduced by up to 13.6% when using scsi-mq with the Linux 3.18 kernel compared to the legacy block layer in 2.6.32.
• CPU idle time increased by a factor of 7 in the 3.18 kernel with scsi-mq, indicating significantly improved I/O efficiency and reduced CPU contention.
• The multi-queue block layer demonstrated measurable performance gains even with caching storage arrays based on rotational media, which are typically I/O-bound by hardware limits.
• Despite the theoretical benefits of multiple hardware queues, using multiple channels in the SRP driver degraded performance, suggesting a mismatch between driver design and hardware parallelism.
• The study confirms that the legacy block layer’s scalability is limited by lock contention, interrupt handling, and cache coherency issues, even under realistic HPC storage workloads.
• The results suggest that metadata target operations—such as journal replay during filesystem recovery—could benefit significantly from low-latency, high-throughput I/O stacks like scsi-mq.