[Paper Review] A Canonical Form for Weighted Automata and Applications to Approximate Minimization

Borja Balle, Prakash Panangaden+1 more|arXiv (Cornell University)|Jan 27, 2015

Machine Learning and Algorithms26 references4 citations

TL;DR

This paper introduces a canonical form for weighted automata using singular value decomposition (SVD) of their infinite Hankel matrices, enabling efficient approximate minimization. The key contribution is an algorithm that truncates the SVD-based canonical form to produce a smaller automaton with provable error bounds in the ℓ² norm.

ABSTRACT

We study the problem of constructing approximations to a weighted automaton. Weighted finite automata (WFA) are closely related to the theory of rational series. A rational series is a function from strings to real numbers that can be computed by a finite WFA. Among others, this includes probability distributions generated by hidden Markov models and probabilistic automata. The relationship between rational series and WFA is analogous to the relationship between regular languages and ordinary automata. Associated with such rational series are infinite matrices called Hankel matrices which play a fundamental role in the theory of minimal WFA. Our contributions are: (1) an effective procedure for computing the singular value decomposition (SVD) of such infinite Hankel matrices based on their representation in terms of finite WFA; (2) a new canonical form for finite WFA based on this SVD decomposition; and, (3) an algorithm to construct approximate minimizations of a given WFA. The goal of our approximate minimization algorithm is to start from a minimal WFA and produce a smaller WFA that is close to the given one in a certain sense. The desired size of the approximating automaton is given as input. We give bounds describing how well the approximation emulates the behavior of the original WFA.

Motivation & Objective

• To develop a canonical form for weighted automata (WFA) based on the SVD of their infinite Hankel matrices.
• To enable approximate minimization of WFA by truncating the SVD-based canonical form, reducing state count while controlling error.
• To provide theoretical guarantees on approximation quality using the ℓ² norm between original and approximated rational series.
• To support spectral learning algorithms in machine learning by enabling smaller, more efficient models even when the true state count is overestimated.
• To extend the applicability of spectral methods beyond the realizability assumption by handling non-minimal or overparameterized models.

Proposed method

• Compute the SVD of the infinite Hankel matrix associated with a weighted automaton using its finite WFA representation.
• Define the singular value automaton (SVA) as a new canonical form derived from the SVD decomposition, ensuring uniqueness under standard SVD conditions.
• Construct an efficient algorithm to compute the SVA from any WFA realizing a given rational series, leveraging matrix algebra and WFA minimization procedures.
• Apply the SVATruncation algorithm to truncate the SVA by retaining only the top-k singular values, yielding a smaller WFA with reduced complexity.
• Use the ℓ² norm as a measure of approximation error, with bounds derived from the sum of squared discarded singular values.
• Leverage duality and co-algebraic principles to generalize Brzozowski-style minimization to approximate settings, enabling theoretical analysis.

Experimental results

Research questions

• RQ1 Can the SVD of an infinite Hankel matrix be computed efficiently from a finite WFA representation?
• RQ2 Does a canonical form for WFA exist that is uniquely determined by the SVD of its Hankel matrix?
• RQ3 Can we construct an approximate minimization algorithm for WFA that guarantees bounded error in the ℓ² norm?
• RQ4 How does the approximation error relate to the singular values of the Hankel matrix?
• RQ5 Can the proposed method be extended to other quantitative systems such as weighted tree automata or context-free grammars?

Key findings

• An efficient algorithm is presented for computing the SVD of an infinite Hankel matrix directly from a WFA, avoiding explicit matrix construction.
• The singular value automaton (SVA) provides a unique canonical form for WFA, analogous to the SVD of a matrix.
• The SVATruncation algorithm produces an approximating WFA with a user-specified number of states, with error bounded by the sum of squares of discarded singular values.
• The approximation error in the ℓ² norm is bounded by ‖f − f̂‖₂ ≤ √(∑_{i=k+1}^∞ σ_i²), where σ_i are the singular values of the Hankel matrix.
• The method enables consistent and finite-sample guarantees in PAC learning even when the true model complexity is overestimated.
• The computational cost of the SVA computation is O(n⁶), and the alternate method using WFA minimizations is O(n⁸), justifying the use of the former when possible.