[Paper Review] From WiscKey to Bourbon: A Learned Index for Log-Structured Merge Trees

Yifan Dai, Yien Xu+5 more|arXiv (Cornell University)|May 28, 2020

Data Quality and ManagementDecision Sciences34 references34 citations

TL;DR

The paper introduces Bourbon, a learned-index extension for WiscKey-style LSM trees, using piecewise linear regression to speed up lookups with a cost-benefit mechanism that decides when learning is worthwhile, achieving 1.23×–1.78× faster lookups across datasets and workloads.

ABSTRACT

We introduce BOURBON, a log-structured merge (LSM) tree that utilizes machine learning to provide fast lookups. We base the design and implementation of BOURBON on empirically-grounded principles that we derive through careful analysis of LSM design. BOURBON employs greedy piecewise linear regression to learn key distributions, enabling fast lookup with minimal computation, and applies a cost-benefit strategy to decide when learning will be worthwhile. Through a series of experiments on both synthetic and real-world datasets, we show that BOURBON improves lookup performance by 1.23x-1.78x as compared to state-of-the-art production LSMs.

Motivation & Objective

• Motivate applying learned indexes to log-structured merge trees (LSMs) to improve read lookups without incurring excessive write overheads.
• Derive guidelines for when and where learning should be applied inside an LSM (file vs. level granularity, workload awareness).
• Design and implement Bourbon, a production-quality learned-index extension for WiscKey, and evaluate its performance across synthetic and real datasets.

Proposed method

• Analyze WiscKey to identify where learned indexes can reduce indexing overhead without destabilizing writes.
• Adopt greedy piecewise linear regression (PLR) to learn key distributions within SSTables or levels.
• Introduce a cost-benefit analyzer to decide, online, whether learning a given file or level is worthwhile.
• Store values separately (key-value separation) and learn over fixed-size keys with offsets for variable-sized values.
• Evaluate file-level learning (default) vs level-level learning, demonstrating when each is beneficial.
• Present Bourbon’s integration into an optimized LSM system with ~5K additional LOC.

Experimental results

Research questions

• RQ1 Can learned indexes improve lookup performance in LSM-based systems like WiscKey without excessive re-learning due to writes?
• RQ2 What granularities (file vs level) and workload conditions maximize the benefits of a learned index in an LSM?
• RQ3 How should a system decide online when to learn a given file or level to maximize net benefit?
• RQ4 How does the learned-index approach perform across memory-resident and fast-storage scenarios (e.g., Optane) for LSMs?

Key findings

• Bourbon yields 1.23×–1.78× faster lookups across a range of datasets and workloads compared to production LSMs.
• Learning gains are most pronounced when indexing contributes a sizable portion of lookup latency, such as in memory-resident or fast-storage scenarios where data-access costs are low or dominated by indexing.
• File-level learning outperforms level-level learning under write-heavy workloads, while level learning can be beneficial for read-only settings.
• A cost-benefit analyzer (online) prevents learning for short-lived files, reducing wasted learning effort.
• Learning guidelines indicate: (i) favor learning lower, stable levels/files; (ii) delay learning for short-lived files; (iii) do not neglect higher levels due to potential negative lookups; (iv) be workload-aware; (v) avoid level learning under heavy writes.