[Paper Review] A Maximum Matching Algorithm for Basis Selection in Spectral Learning

Ariadna Quattoni, Xavier Carreras+1 more|arXiv (Cornell University)|Jun 9, 2017

Blind Source Separation Techniques14 references4 citations

TL;DR

This paper proposes a maximum bipartite matching algorithm for basis selection in spectral learning, enabling efficient computation on large Hankel matrices by selecting a compact, structurally full-rank submatrix. The method achieves up to two orders of magnitude faster SVD computation with minimal performance loss, significantly scaling spectral learning for sparse sequence functions in NLP.

ABSTRACT

We present a solution to scale spectral algorithms for learning sequence functions. We are interested in the case where these functions are sparse (that is, for most sequences they return 0). Spectral algorithms reduce the learning problem to the task of computing an SVD decomposition over a special type of matrix called the Hankel matrix. This matrix is designed to capture the relevant statistics of the training sequences. What is crucial is that to capture long range dependencies we must consider very large Hankel matrices. Thus the computation of the SVD becomes a critical bottleneck. Our solution finds a subset of rows and columns of the Hankel that realizes a compact and informative Hankel submatrix. The novelty lies in the way that this subset is selected: we exploit a maximal bipartite matching combinatorial algorithm to look for a sub-block with full structural rank, and show how computation of this sub-block can be further improved by exploiting the specific structure of Hankel matrices.

Motivation & Objective

• To address the computational bottleneck in spectral learning caused by large Hankel matrix SVD decomposition.
• To develop an efficient, sample-dependent basis selection method that maintains model accuracy while reducing computational cost.
• To exploit sparsity and structural rank in Hankel matrices to enable scalable learning of long-range dependencies in sparse sequence functions.
• To provide a combinatorial alternative to greedy or exhaustive basis selection, improving efficiency without sacrificing performance.

Proposed method

• The method uses a bipartite graph representation of the Hankel matrix’s sparsity pattern, with prefixes and suffixes as nodes and non-zero entries as edges.
• It computes a maximum cardinality matching in this bipartite graph to identify a subset of prefixes and suffixes that form a submatrix of full structural rank.
• The selected submatrix is used to compute a low-rank approximation via SVD, replacing the full Hankel matrix computation.
• The algorithm leverages the specific structure of Hankel matrices to further optimize submatrix computation and factorization.
• It is designed to be optimal under non-degeneracy assumptions, ensuring minimal basis size when possible.

Experimental results

Research questions

• RQ1 Can a combinatorial maximum matching algorithm effectively identify a minimal and informative basis for spectral learning in sparse sequence functions?
• RQ2 How does the performance of spectral learning scale when using a matched submatrix instead of the full Hankel matrix?
• RQ3 To what extent can the proposed method reduce SVD computation time while preserving model accuracy on real NLP tasks?
• RQ4 Is the maximum matching approach more efficient than existing basis selection strategies like greedy selection or full prefix/suffix enumeration?

Key findings

• The proposed method selects a basis at least an order of magnitude smaller than alternative methods, drastically reducing SVD computation time.
• The SVD step becomes at least two orders of magnitude faster when using the matched submatrix instead of the full Hankel matrix.
• On the SPiCe sequence prediction benchmark, the method achieved second-best overall performance (0.6414 average ranking) and second in three out of five datasets.
• For high-rank Hankel matrices, training times remained under five minutes, demonstrating strong scalability.
• The method outperforms or matches state-of-the-art models like RNN-P and ensemble methods, even without model combination.
• The approach is robust in practice, with results suggesting the non-degeneracy assumption is nearly always satisfied for sparse sequence functions.