[Paper Review] Memory Efficient De Bruijn Graph Construction

Yang Li, Pegah Kamousi+4 more|arXiv (Cornell University)|Jul 15, 2012

Algorithms and Data Compression21 references3 citations

TL;DR

This paper proposes Minimum Substring Partitioning (MSP), a disk-based method that compresses de Bruijn graph construction by leveraging k-mer overlaps to reduce partition size from Θ(kn) to Θ(n), enabling construction with under 10 GB memory. The approach achieves high compression and performance without runtime slowdown, outperforming existing algorithms on large-scale genomic datasets.

ABSTRACT

Massively parallel DNA sequencing technologies are revolutionizing genomics research. Billions of short reads generated at low costs can be assembled for reconstructing the whole genomes. Unfortunately, the large memory footprint of the existing de novo assembly algorithms makes it challenging to get the assembly done for higher eukaryotes like mammals. In this work, we investigate the memory issue of constructing de Bruijn graph, a core task in leading assembly algorithms, which often consumes several hundreds of gigabytes memory for large genomes. We propose a disk-based partition method, called Minimum Substring Partitioning (MSP), to complete the task using less than 10 gigabytes memory, without runtime slowdown. MSP breaks the short reads into multiple small disjoint partitions so that each partition can be loaded into memory, processed individually and later merged with others to form a de Bruijn graph. By leveraging the overlaps among the k-mers (substring of length k), MSP achieves astonishing compression ratio: The total size of partitions is reduced from $Θ(kn)$ to $Θ(n)$, where $n$ is the size of the short read database, and $k$ is the length of a $k$-mer. Experimental results show that our method can build de Bruijn graphs using a commodity computer for any large-volume sequence dataset.

Motivation & Objective

• Address the high memory footprint of de Bruijn graph construction in de novo genome assembly, which often exceeds hundreds of gigabytes for mammalian genomes.
• Overcome the limitations of existing disk-based partitioning methods that suffer from high I/O overhead and poor compression due to fragmented k-mer distribution.
• Develop a partitioning strategy that preserves overlapping k-mers in the same partition to minimize data size and improve efficiency.
• Enable scalable, in-memory-efficient de Bruijn graph construction on commodity hardware for billion-read datasets.
• Demonstrate that the proposed method achieves both low memory usage and high performance without runtime slowdown.

Proposed method

• Introduce Minimum Substring Partitioning (MSP), which partitions short reads based on a fixed-length minimum substring (length p ≤ k) shared among consecutive k-mers.
• Use the minimum substring as a key to group k-mers that are highly overlapping, ensuring adjacent k-mers are placed in the same partition.
• Apply a disk-based scatter-gather pipeline: partition reads into disjoint subsets using minimum substrings, process each subset in main memory, and merge results post-processing.
• Leverage the overlap structure of k-mers to compress the total partition size from Θ(kn) to Θ(n), significantly reducing I/O and memory pressure.
• Theoretical analysis under a random string model derives the expected partition size and capacity of minimum substrings, validating the compression efficiency.
• Implement a full de Bruijn graph construction pipeline using MSP, integrating with standard assembly workloads and evaluating on real short-read datasets.

Experimental results

Research questions

• RQ1 Can k-mer partitioning based on minimum substrings reduce the total size of partitions compared to traditional k-mer-based partitioning?
• RQ2 Does preserving overlapping k-mers in the same partition lead to measurable improvements in memory efficiency and I/O performance?
• RQ3 Can MSP-based de Bruijn graph construction be achieved with under 10 GB of main memory while maintaining competitive runtime?
• RQ4 How does the size of the minimum substring (p) affect the compression ratio and partitioning efficiency?
• RQ5 Is the proposed method scalable and effective on large-scale, real-world genomic datasets with billions of short reads?

Key findings

• The MSP method reduces the total partition size from Θ(kn) to Θ(n), achieving a 10-15x compression ratio compared to conventional k-mer partitioning.
• The method successfully constructs de Bruijn graphs for large datasets—such as 258.7 GB Cladonema and 137.5 GB cichlid—using less than 10 GB of main memory.
• Runtime performance is not slowed down compared to existing algorithms, despite the disk-based approach.
• The experimental results show that MSP outperforms state-of-the-art algorithms in both memory efficiency and overall performance on real genomic datasets.
• Theoretical analysis under a random string model confirms that the expected size of MSP partitions scales linearly with the dataset size n, validating the compression model.
• The approach enables de novo assembly on commodity hardware, making large-scale genome assembly feasible without high-end memory resources.