[Paper Review] BPF for storage: an exokernel-inspired approach

Yu Jian Wu, Hongyi Wang+5 more|arXiv (Cornell University)|Feb 25, 2021

Advanced Data Storage Technologies20 references4 citations

TL;DR

This paper proposes using eBPF to accelerate storage I/O by injecting user-defined functions deep in the kernel’s I/O stack, bypassing the file system and block layers to reduce latency and increase IOPS. By enabling BPF-based traversal of on-disk data structures like B-trees, the approach achieves over 2.5× higher IOPS and 50% lower latency on NVMe devices while preserving file system safety through exokernel-inspired mechanisms.

ABSTRACT

The overhead of the kernel storage path accounts for half of the access latency for new NVMe storage devices. We explore using BPF to reduce this overhead, by injecting user-defined functions deep in the kernel's I/O processing stack. When issuing a series of dependent I/O requests, this approach can increase IOPS by over 2.5$ imes$ and cut latency by half, by bypassing kernel layers and avoiding user-kernel boundary crossings. However, we must avoid losing important properties when bypassing the file system and block layer such as the safety guarantees of the file system and translation between physical blocks addresses and file offsets. We sketch potential solutions to these problems, inspired by exokernel file systems from the late 90s, whose time, we believe, has finally come!

Motivation & Objective

• To reduce the kernel's software overhead in the storage stack, which accounts for ~50% of I/O latency on modern NVMe devices.
• To enable application-specific, low-level I/O processing via eBPF without requiring kernel modifications or application-level rewrites.
• To maintain file system safety and access control guarantees while bypassing traditional kernel layers.
• To support efficient, high-throughput I/O chaining for dependent operations like index traversals.
• To enable concurrency and fairness in BPF-based storage operations without relying on traditional kernel scheduling.

Proposed method

• Injecting eBPF programs deep in the kernel’s I/O path to process I/O requests directly in the kernel, avoiding user-kernel transitions.
• Using BPF to traverse on-disk data structures (e.g., B-trees, LSM trees) by chaining multiple I/O operations without returning to userspace.
• Implementing a hook at the NVMe layer to track and invalidate BPF-issued I/Os when file system extents are unmapped, ensuring consistency.
• Introducing a per-process counter in the NVMe layer to bound chained I/O submissions and prevent unbounded loops or resource exhaustion.
• Leveraging exokernel principles to allow user-defined understanding of file system metadata while preserving isolation and safety.
• Designing a caching model where BPF functions return application-level objects rather than pages, aligning with modern application-level cache management.

Experimental results

Research questions

• RQ1 How can eBPF be used to reduce kernel I/O overhead in fast storage stacks without sacrificing safety or portability?
• RQ2 What mechanisms are needed to enforce file system access control and data integrity when bypassing the file system and block layers?
• RQ3 How can BPF-based I/O chaining be made safe and efficient for stateful, dependent operations like index traversal?
• RQ4 What mechanisms ensure fairness and prevent resource exhaustion in concurrent BPF-based I/O workloads?
• RQ5 How can BPF-based storage operations interoperate with existing Linux file systems and I/O scheduling policies?

Key findings

• BPF-based I/O chaining reduces I/O latency by 50% and increases IOPS by over 2.5× on modern NVMe storage devices.
• The approach achieves significant performance gains by eliminating user-kernel transitions and bypassing the file system and block layers.
• A hook at the NVMe layer enables invalidation of BPF-issued I/Os when file extents are unmapped, preserving consistency with minimal overhead.
• A per-process I/O chain counter prevents unbounded I/O loops and supports fairness in multi-process environments.
• The design supports read-only BPF traversals of immutable data structures like LSM SSTables and on-disk B-trees, which are common in modern storage systems.
• The method integrates well with application-level caching and avoids interference with the kernel’s buffer cache, enabling efficient, object-granularity data management.