[Paper Review] Tensor Reconstruction Beyond Constant Rank

Shir Peleg, Amir Shpilka+1 more|arXiv (Cornell University)|Sep 9, 2022

Complexity and Algorithms in Graphs3 citations

TL;DR

This paper presents the first efficient reconstruction algorithms for depth-3 arithmetic circuits with super-constant rank, specifically for Σ[k]V[d]Σ, multilinear Σ[k]∏[d]Σ, and set-multilinear Σ[k]∏[d]Σ circuits. It achieves this by introducing a novel technique for learning rank-preserving coordinate subspaces, avoiding the exponential blowup in k seen in prior work, and provides deterministic and randomized algorithms with improved running times of poly(n, d, c) · poly(k)kk10 and poly(n, d, c) · kkkkO(k), respectively.

ABSTRACT

We give reconstruction algorithms for subclasses of depth-3 arithmetic circuits. In particular, we obtain the first efficient algorithm for finding tensor rank, and an optimal tensor decomposition as a sum of rank-one tensors, when given black-box access to a tensor of super-constant rank. Specifically, we obtain the following results: 1) A deterministic algorithm that reconstructs polynomials computed by Σ^{[k]}⋀^{[d]}Σ circuits in time poly(n,d,c) ⋅ poly(k)^{k^{k^{10}}}, 2) A randomized algorithm that reconstructs polynomials computed by multilinear Σ^{[k]}∏^{[d]}Σ circuits in time poly(n,d,c) ⋅ k^{k^{k^{k^{O(k)}}}}, 3) A randomized algorithm that reconstructs polynomials computed by set-multilinear Σ^{[k]}∏^{[d]}Σ circuits in time poly(n,d,c) ⋅ k^{k^{k^{k^{O(k)}}}}, where c = log q if 𝔽 = 𝔽_q is a finite field, and c equals the maximum bit complexity of any coefficient of f if 𝔽 is infinite. Prior to our work, polynomial time algorithms for the case when the rank, k, is constant, were given by Bhargava, Saraf and Volkovich [Vishwas Bhargava et al., 2021]. Another contribution of this work is correcting an error from a paper of Karnin and Shpilka [Zohar Shay Karnin and Amir Shpilka, 2009] (with some loss in parameters) that also affected Theorem 1.6 of [Vishwas Bhargava et al., 2021]. Consequently, the results of [Zohar Shay Karnin and Amir Shpilka, 2009; Vishwas Bhargava et al., 2021] continue to hold, with a slightly worse setting of parameters. For fixing the error we systematically study the relation between syntactic and semantic notions of rank of Σ Π Σ circuits, and the corresponding partitions of such circuits. We obtain our improved running time by introducing a technique for learning rank preserving coordinate-subspaces. Both [Zohar Shay Karnin and Amir Shpilka, 2009] and [Vishwas Bhargava et al., 2021] tried all choices of finding the "correct" coordinates, which, due to the size of the set, led to having a fast growing function of k at the exponent of n. We manage to find these spaces in time that is still growing fast with k, yet it is only a fixed polynomial in n.

Motivation & Objective

• To develop efficient reconstruction algorithms for depth-3 arithmetic circuits when the rank k is super-constant, extending beyond the previously known constant-rank case.
• To correct an error in Karnin and Shpilka (2009) and Theorem 1.6 of Bhargava, Saraf, and Volkovich (2021), which affected parameter settings in prior results.
• To establish a systematic connection between syntactic and semantic notions of rank in ΣΠΣ circuits and their corresponding partitions.
• To design a new algorithmic technique for learning rank-preserving coordinate subspaces that avoids exhaustive search over coordinate choices, leading to improved running time dependence on k.

Proposed method

• Introduces a novel method to learn rank-preserving coordinate subspaces, replacing the prior approach of trying all possible coordinate sets.
• Applies this technique to reconstruct Σ[k]V[d]Σ circuits via a deterministic algorithm with running time poly(n, d, c) · poly(k)kk10.
• Uses a randomized approach for multilinear and set-multilinear Σ[k]∏[d]Σ circuits, achieving running time poly(n, d, c) · kkkkO(k).
• Employs black-box access to directional derivatives and uses hitting sets to identify essential variables and cluster structure.
• Applies the Berlekamp-Welch algorithm to interpolate univariate restrictions of the polynomial along lines, enabling cluster recovery.
• Leverages semantic and syntactic rank partitions to guide the reconstruction process and ensure correctness.

Experimental results

Research questions

• RQ1 Can efficient reconstruction be achieved for depth-3 circuits when the rank k is super-constant, rather than constant?
• RQ2 What is the correct relationship between syntactic and semantic rank in ΣΠΣ circuits, and how can it be exploited algorithmically?
• RQ3 Can the exponential dependence on k in prior algorithms be reduced by avoiding exhaustive search over coordinate subspaces?
• RQ4 How can errors in prior works—particularly in Karnin and Shpilka (2009) and Bhargava, Saraf, and Volkovich (2021)—be corrected while preserving the core results with adjusted parameters?
• RQ5 Is it possible to design a reconstruction algorithm that properly learns set-multilinear circuits by preserving their structural constraints?

Key findings

• The paper presents the first deterministic algorithm for reconstructing Σ[k]V[d]Σ circuits in time poly(n, d, c) · poly(k)kk10, where c is the field size or coefficient bit complexity.
• It gives the first randomized reconstruction algorithm for multilinear and set-multilinear Σ[k]∏[d]Σ circuits with running time poly(n, d, c) · kkkkO(k).
• The authors correct a critical error in Karnin and Shpilka (2009) and show that the results of that work and Bhargava, Saraf, and Volkovich (2021) still hold with slightly worse parameters.
• The new technique of learning rank-preserving subspaces reduces the running time dependence on k from exponential in k to a fixed polynomial in n, even though the exponent in k remains large.
• The algorithm successfully reconstructs minimal circuits and passes a randomized PIT check to ensure correctness with high probability.
• The method enables proper learning of set-multilinear circuits by ensuring that the learned subcircuits preserve the set-multilinear structure.