[Paper Review] Fractional Hitting Sets for Efficient and Lightweight Genomic Data Sketching

Rouzé, Timothé, Martayan, Igor+2 more|arXiv (Cornell University)|May 25, 2015

Genomics and Phylogenetic Studies18 references47 citations

TL;DR

This paper proposes MSPKmerCounter, a disk-based k-mer counting method that uses Minimum Substring Partitioning (MSP) to split short reads into disjoint partitions of 'super k-mers' sharing common minimum substrings. By exploiting k-mer overlaps within reads, MSP achieves high compression, reducing I/O and memory usage. The method outperforms Jellyfish and BFCounter in both speed and memory efficiency on large-scale genomic datasets, enabling scalable, memory-efficient k-mer counting on commodity hardware.

ABSTRACT

A major challenge in next-generation genome sequencing (NGS) is to assemble massive overlapping short reads that are randomly sampled from DNA fragments. To complete assembling, one needs to finish a fundamental task in many leading assembly algorithms: counting the number of occurrences of k-mers (length-k substrings in sequences). The counting results are critical for many components in assembly (e.g. variants detection and read error correction). For large genomes, the k-mer counting task can easily consume a huge amount of memory, making it impossible for large-scale parallel assembly on commodity servers. In this paper, we develop MSPKmerCounter, a disk-based approach, to efficiently perform k-mer counting for large genomes using a small amount of memory. Our approach is based on a novel technique called Minimum Substring Partitioning (MSP). MSP breaks short reads into multiple disjoint partitions such that each partition can be loaded into memory and processed individually. By leveraging the overlaps among the k-mers derived from the same short read, MSP can achieve astonishing compression ratio so that the I/O cost can be significantly reduced. For the task of k-mer counting, MSPKmerCounter offers a very fast and memory-efficient solution. Experiment results on large real-life short reads data sets demonstrate that MSPKmerCounter can achieve better overall performance than state-of-the-art k-mer counting approaches. MSPKmerCounter is available at http://www.cs.ucsb.edu/~yangli/MSPKmerCounter

Motivation & Objective

• Address the memory bottleneck in k-mer counting for large genomes, which limits de novo assembly on commodity hardware.
• Overcome the limitations of hash table-based methods that consume excessive memory, especially for high-coverage mammalian genomes.
• Develop a disk-based, scalable solution that maintains high performance and low memory usage for k-mer frequency counting.
• Enable efficient, parallelizable processing of massive short-read datasets using minimal main memory.

Proposed method

• Apply Minimum Substring Partitioning (MSP) to break short reads into disjoint partitions of 'super k-mers' that share a common minimum substring (p ≤ k).
• Leverage overlapping k-mers within each read to compress the data, reducing the number of distinct k-mers per partition and minimizing I/O overhead.
• Use a disk-based pipeline to store and process partitions sequentially, avoiding main memory overflow.
• Implement a multi-threaded version to parallelize the partitioning and counting phases, improving throughput on multi-core systems.
• Utilize a hash table for frequency counting within each partition, with results merged post-processing.
• Control memory and I/O usage via configurable parameters, enabling tunable performance across hardware constraints.

Experimental results

Research questions

• RQ1 Can a disk-based k-mer counting method achieve better memory efficiency than in-memory approaches like Jellyfish for large-scale genomics data?
• RQ2 Does Minimum Substring Partitioning (MSP) significantly reduce I/O cost by exploiting k-mer overlaps within reads?
• RQ3 How does MSPKmerCounter compare to state-of-the-art tools like Jellyfish and BFCounter in terms of running time, memory usage, and disk I/O?
• RQ4 Can MSP-based k-mer counting be effectively parallelized on multi-core systems while maintaining low memory footprint?
• RQ5 Can the method scale to very large datasets (e.g., 100x coverage) without requiring high-end memory systems?

Key findings

• MSPKmerCounter successfully performs k-mer counting on large genomic datasets using only a fraction of the memory required by in-memory tools like Jellyfish, enabling use on commodity hardware.
• The method reduces I/O cost by 10–15 times compared to naive hash-based partitioning due to effective compression via MSP.
• On the Lake Malawi cichlid dataset (k=31), MSPKmerCounter achieved faster running times than Jellyfish(Disk) and BFCounter, even with lower memory usage.
• The multi-threaded version of MSPKmerCounter (MSPKmerCounter(MT)) outperformed Jellyfish(Disk) in speed and showed good scalability up to 2 threads, limited by I/O bandwidth.
• Memory consumption, running time, and disk usage in MSPKmerCounter are fully controllable via configuration parameters, allowing tuning for diverse hardware environments.
• The method enables scalable, parallelizable k-mer counting by ensuring disjoint partitions, paving the way for distributed and local assembly pipelines.