[Paper Review] Efficient Symmetric Norm Regression via Linear Sketching

Zhao Song, Ruosong Wang+3 more|arXiv (Cornell University)|Oct 4, 2019

Advanced Optimization Algorithms Research8 citations

TL;DR

This paper presents the first input-sparsity time algorithm with provable guarantees for symmetric norm regression, using a novel linear sketching framework called SymSketch. It achieves a $(1+\varepsilon)$-approximate solution for Orlicz norms and a $\sqrt{d} \cdot \mathrm{mmc}(\ell) \cdot \mathrm{polylog}\,n$-approximate solution for general symmetric norms, significantly improving over prior work with $d \cdot \mathrm{polylog}\,n$ approximation ratios.

ABSTRACT

We provide efficient algorithms for overconstrained linear regression problems with size $n imes d$ when the loss function is a symmetric norm (a norm invariant under sign-flips and coordinate-permutations). An important class of symmetric norms are Orlicz norms, where for a function $G$ and a vector $y \in \mathbb{R}^n$, the corresponding Orlicz norm $\|y\|_G$ is defined as the unique value $α$ such that $\sum_{i=1}^n G(|y_i|/α) = 1$. When the loss function is an Orlicz norm, our algorithm produces a $(1 + \varepsilon)$-approximate solution for an arbitrarily small constant $\varepsilon > 0$ in input-sparsity time, improving over the previously best-known algorithm which produces a $d \cdot \mathrm{polylog} n$-approximate solution. When the loss function is a general symmetric norm, our algorithm produces a $\sqrt{d} \cdot \mathrm{polylog} n \cdot \mathrm{mmc}(\ell)$-approximate solution in input-sparsity time, where $\mathrm{mmc}(\ell)$ is a quantity related to the symmetric norm under consideration. To the best of our knowledge, this is the first input-sparsity time algorithm with provable guarantees for the general class of symmetric norm regression problem. Our results shed light on resolving the universal sketching problem for linear regression, and the techniques might be of independent interest to numerical linear algebra problems more broadly.

Motivation & Objective

• To develop a universal sketching-based algorithm for symmetric norm regression that works across a broad class of loss functions, including Orlicz norms and other symmetric norms.
• To close the gap between theoretical sketching techniques and practical applicability by achieving $(1+\varepsilon)$-approximation for Orlicz norms in input-sparsity time.
• To extend the scope of linear sketching beyond $\ell_p$ norms and $M$-estimators to include complex symmetric norms such as top-$k$, sum-mix, and $k$-support norms.
• To provide provable approximation guarantees for general symmetric norms, introducing a new quantity $\mathrm{mmc}(\ell)$ that captures the norm's structural complexity.
• To resolve the universal sketching problem for linear regression by designing a single framework applicable to diverse symmetric norm losses.

Proposed method

• Introduces a new sketching framework called SymSketch, which is a structured linear sketch tailored for symmetric norms.
• Employs a two-stage sketching process: first, a randomized embedding maps the residual vector into a lower-dimensional space while preserving symmetric norm structure.
• Uses a subspace embedding via a Fast Johnson-Lindenstrauss Transform to preserve $\ell_2$-norms of the transformed residuals, which are then related back to the original symmetric norm via norm equivalence arguments.
• Leverages the symmetry of the norm (invariant under sign flips and permutations) to design a sketch that depends only on the singular values and structure of the data matrix.
• Applies a novel analysis of the $\ell_2$-embedding properties of the sketch matrix to bound the approximation ratio in terms of $\mathrm{mmc}(\ell)$, a norm-specific complexity measure.
• Combines the sketch with standard least-squares regression on the sketched problem to compute an approximate solution in $O(\mathrm{nnz}(A) + \mathrm{poly}(d))$ time.

Experimental results

Research questions

• RQ1 Can we design a single sketching-based algorithm that achieves $(1+\varepsilon)$-approximation for symmetric norm regression in input-sparsity time, without requiring a new design per loss function?
• RQ2 Is it possible to extend the success of linear sketching from $\ell_p$ norms and $M$-estimators to Orlicz norms with optimal approximation ratio?
• RQ3 How can we handle general symmetric norms—such as top-$k$, sum-mix, and $k$-support norms—that are not Orlicz norms but are widely used in machine learning?
• RQ4 What is the minimal sketch size and what structural properties of the norm determine the approximation quality in such a framework?
• RQ5 Can we unify the analysis of sketching for symmetric norms by introducing a single complexity parameter $\mathrm{mmc}(\ell)$ that captures the norm's behavior under sketching?

Key findings

• The proposed SymSketch algorithm achieves a $(1+\varepsilon)$-approximate solution for Orlicz norm regression in $O(\mathrm{nnz}(A) + \mathrm{poly}(d/\varepsilon))$ time, improving over the prior $d \cdot \mathrm{polylog}\,n$-approximation.
• For general symmetric norms, the algorithm provides a $\sqrt{d} \cdot \mathrm{mmc}(\ell) \cdot \mathrm{polylog}\,n$-approximate solution in input-sparsity time, where $\mathrm{mmc}(\ell)$ is a norm-specific complexity measure.
• The algorithm is the first to achieve provable input-sparsity time guarantees for the general symmetric norm regression problem, resolving a key open problem in universal sketching.
• Experiments on U.S. Census and YearPredictionMSD datasets show that SymSketch outperforms uniform sampling and prior sketching methods in both Orlicz and symmetric norm settings.
• With embedding dimension $5d$, SymSketch achieves an approximation ratio of $1.25$ for the sum-mix $\ell_1 + \ell_2$ norm, demonstrating strong practical performance.
• Theoretical analysis shows that the sketch preserves $\ell_2$-norms of transformed residuals up to a factor of $O(\sqrt{d} \log^3 n)$, which is then used to bound the symmetric norm approximation.