[Paper Review] Generalized Functional Pruning Optimal Partitioning (GFPOP) for Constrained Changepoint Detection in Genomic Data

Toby Dylan Hocking, Guillem Rigaill+2 more|arXiv (Cornell University)|Sep 29, 2018

Gene expression and cancer classification4 citations

TL;DR

This paper introduces GFPOP, a new algorithm for constrained changepoint detection in genomic data that achieves $O(N\log N)$ time complexity for optimal segmentation under a penalty-based model. It further proposes a sequential search method that computes optimal models with $O(\sqrt{N})$ peaks in $O(N(\log N)^2)$ time, enabling efficient, scalable peak detection in large-scale genomics data sets with tens of millions of points.

ABSTRACT

We describe a new algorithm and R package for peak detection in genomic data sets using constrained changepoint algorithms. These detect changes from background to peak regions by imposing the constraint that the mean should alternately increase then decrease. An existing algorithm for this problem exists, and gives state-of-the-art accuracy results, but it is computationally expensive when the number of changes is large. We propose the GFPOP algorithm that jointly estimates the number of peaks and their locations by minimizing a cost function which consists of a data fitting term and a penalty for each changepoint. Empirically this algorithm has a cost that is $O(N \log(N))$ for analysing data of length $N$. We also propose a sequential search algorithm that finds the best solution with $K$ segments in $O(\log(K)N \log(N))$ time, which is much faster than the previous $O(KN \log(N))$ algorithm. We show that our disk-based implementation in the PeakSegDisk R package can be used to quickly compute constrained optimal models with many changepoints, which are needed to analyze typical genomic data sets that have tens of millions of observations.

Motivation & Objective

• To develop a scalable, optimal algorithm for detecting peaks in genomic data with constrained changepoint structures, where means alternate between increasing and decreasing.
• To reduce the computational cost of optimal segmentation with many changepoints, especially for large-scale genomics data sets with $N \approx 10^7$ points.
• To enable practical computation of optimal models with $O(\sqrt{N})$ peaks—common in real genomic data—by replacing expensive $O(N\sqrt{N}\log N)$ methods with a faster sequential search.
• To provide a disk-based implementation that maintains high performance and low memory usage, making optimal peak detection feasible on standard hardware.

Proposed method

• GFPOP uses functional pruning to reduce the number of candidate changepoints from $O(N)$ to $O(\log N)$ per segment, enabling $O(N\log N)$ time complexity for a single penalty.
• It formulates the segmentation problem as minimizing a penalized negative log-likelihood under a Poisson model, with loss function $\ell(m,z) = m - z\log m$.
• The algorithm employs dynamic programming with a functional representation of the optimal cost, allowing pruning of suboptimal segment boundaries while preserving optimality.
• A sequential search algorithm is proposed that calls GFPOP $O(\log N)$ times to find the optimal model with up to $P$ peaks, reducing the number of required calls from $O(P)$ to $O(\log P)$.
• The implementation uses disk-based storage to manage large data sets, with empirical performance showing only a constant factor slowdown compared to in-memory methods.
• The method is implemented in the PeakSegDisk R package, supporting both memory and disk-based computation for scalability.

Experimental results

Research questions

• RQ1 Can a constrained changepoint algorithm achieve $O(N\log N)$ time complexity while maintaining optimality for peak detection in genomic data?
• RQ2 Is it possible to compute optimal models with $O(\sqrt{N})$ peaks in $O(N(\log N)^2)$ time, significantly faster than existing $O(N\sqrt{N}\log N)$ approaches?
• RQ3 How does disk-based storage affect the performance and memory usage of optimal changepoint detection for large-scale genomics data?
• RQ4 Can a sequential search strategy reduce the number of GFPOP calls needed to find the best model with $P$ peaks from $O(P)$ to $O(\log P)$?

Key findings

• GFPOP computes the optimal segmentation for a single penalty in $O(N\log N)$ time and $O(N\log N)$ space, with only $O(\log N)$ memory usage.
• For $N = 10^7$, the sequential search algorithm requires only 10–15 calls to GFPOP to compute a zero-error model with $O(\sqrt{N})$ peaks, compared to 2828 DP iterations in the Segment Neighborhood approach.
• The sequential search runs in $O(N\log(N)\log P)$ time, which is significantly faster than the $O(N\sqrt{N}\log N)$ time of the Segment Neighborhood method for $P > 5$ peaks.
• Empirical results show that disk-based GFPOP is only a constant factor slower than in-memory computation, enabling efficient processing of $N = 10^7$ data points in hours rather than weeks.
• The method achieves zero label errors on benchmark data when $P = O(\sqrt{N})$, confirming that this is a realistic and optimal number of peaks for typical genomic data.
• The PeakSegDisk package enables practical computation of optimal models with many peaks, reducing required storage from 220 terabytes to under 100 gigabytes and computation time from 17 weeks to under a week.