[Paper Review] Re-enabling high-speed caching for LSM-trees

Lei Guo, Dejun Teng+4 more|arXiv (Cornell University)|Jun 7, 2016

Advanced Data Storage Technologies12 references3 citations

TL;DR

This paper proposes dLSM, a modified LSM-tree that uses an on-disk compaction buffer to reduce cache invalidation during intensive writes. By maintaining stable disk addresses for frequently compacted data, dLSM enables persistent caching, achieving 5–8x higher query performance than standard LSM-tree with the same DRAM cache on HDD storage.

ABSTRACT

LSM-tree has been widely used in cloud computing systems by Google, Facebook, and Amazon, to achieve high performance for write-intensive workloads. However, in LSM-tree, random key-value queries can experience long latency and low throughput due to the interference from the compaction, a basic operation in the algorithm, to caching. LSM-tree relies on frequent compaction operations to merge data into a sorted structure. After a compaction, the original data are reorganized and written to other locations on the disk. As a result, the cached data are invalidated since their referencing addresses are changed, causing serious performance degradations. We propose dLSM in order to re-enable high-speed caching during intensive writes. dLSM is an LSM-tree with a compaction buffer on the disk, working as a cushion to minimize the cache invalidation caused by compactions. The compaction buffer maintains a series of snapshots of the frequently compacted data, which represent a consistent view of the corresponding data in the underlying LSM-tree. Being updated in a much lower rate than that of compactions, data in the compaction buffer are almost stationary. In dLSM, an object is referenced by the disk address of the corresponding block either in the compaction buffer for frequently compacted data, or in the underlying LSM-tree for infrequently compacted data. Thus, hot objects can be effectively kept in the cache without harmful invalidations. With the help of a small on-disk compaction buffer, dLSM achieves a high query performance by enabling effective caching, while retaining all merits of LSM-tree for write-intensive data processing. We have implemented dLSM based on LevelDB. Our evaluations show that with a standard DRAM cache, dLSM can achieve 5--8x performance improvement over LSM with the same cache on HDD storage.

Motivation & Objective

• Address the performance degradation in LSM-trees caused by cache invalidation during compaction operations.
• Overcome the fundamental limitation that compaction reorganizes data and invalidates cached blocks, even when data content remains unchanged.
• Enable effective caching for random key-value queries in write-intensive workloads without relying on SSDs or additional key-value caches.
• Preserve the write efficiency and range query performance of LSM-trees while significantly improving random read throughput.
• Provide a low-overhead, general-purpose solution applicable to dynamic and mixed workloads without requiring system-wide architectural changes.

Proposed method

• Introduce a persistent on-disk compaction buffer that stores snapshots of frequently compacted data, maintaining stable disk block addresses.
• Use the compaction buffer as a staging area where data are updated at a much lower rate than in the main LSM-tree, minimizing address changes.
• Reference hot objects via the compaction buffer’s stable block addresses for frequently accessed data, while infrequently accessed data remain in the main LSM-tree.
• Decouple the cache from the physical location of data by ensuring cached blocks in the compaction buffer remain valid across compactions.
• Leverage existing LSM-tree compaction scheduling and merge logic (as in LevelDB) to maintain write efficiency and balance I/O load.
• Use large Bloom filters in the compaction buffer to enable fast key-value lookups with minimal performance overhead.

Experimental results

Research questions

• RQ1 Can a persistent on-disk buffer reduce cache invalidation in LSM-trees during intensive compactions?
• RQ2 To what extent can a compaction buffer improve random read performance without degrading write throughput or increasing memory usage?
• RQ3 Does maintaining stable block addresses in a buffer preserve cache effectiveness in write-intensive workloads?
• RQ4 How does dLSM compare to standard LSM-tree in terms of cache hit ratio and query throughput under mixed read/write workloads?
• RQ5 Can dLSM achieve high performance on HDD storage without requiring SSDs or additional key-value caching layers?

Key findings

• With a standard DRAM cache on HDD, dLSM achieves 5–8x higher query performance than standard LSM-tree under the same write workload.
• The average cache hit ratio in dLSM remains stable at around 90% during compaction, while standard LSM-tree drops to an average of 47%.
• Random read throughput in dLSM stays consistently high (around 3,400 QPS), whereas standard LSM-tree’s throughput fluctuates between 50 and 850 QPS during compaction.
• During full compaction, dLSM maintains high performance, while standard LSM-tree suffers significant performance degradation due to cache churn.
• The compaction buffer reduces cache invalidation bursts by decoupling the update rate of frequently accessed data from the main compaction process.
• dLSM achieves 41% higher average cache hit ratio and 91% higher average read throughput than standard LSM-tree during compaction phases.