[Paper Review] Iterative Methods Based on Soft Thresholding of Hierarchical Tensors

Markus Bachmayr, Reinhold Schneider|arXiv (Cornell University)|Jan 30, 2015

Tensor decomposition and applications8 references4 citations

TL;DR

This paper proposes an iterative method for solving high-dimensional elliptic PDEs using hierarchical tensor formats, where soft thresholding of hierarchical singular values controls tensor ranks during iterations. The method ensures quasi-optimal ranks for all iterates without prior knowledge of solution structure, achieving linear convergence with guaranteed low-rank approximation via a posteriori adaptive thresholding parameters.

ABSTRACT

We construct a soft thresholding operation for rank reduction of hierarchical tensors and subsequently consider its use in iterative thresholding methods, in particular for the solution of discretized high-dimensional elliptic problems. The proposed method for the latter case automatically adjusts the thresholding parameters, based only on bounds on the spectrum of the operator, such that the arising tensor ranks of the resulting iterates remain quasi-optimal with respect to the algebraic or exponential-type decay of the hierarchical singular values of the true solution. In addition, we give a modified algorithm using inexactly evaluated residuals that retains these features. The practical efficiency of the scheme is demonstrated in numerical experiments.

Motivation & Objective

• Address the challenge of rank explosion in iterative solvers for high-dimensional elliptic problems when using tensor formats.
• Develop a rank-reduction strategy that maintains low tensor ranks across iterates despite rank-increasing operations.
• Ensure convergence to the true solution while preserving quasi-optimality of tensor ranks without requiring prior knowledge of the solution's low-rank structure.
• Design a scheme that adaptively adjusts thresholding parameters based on spectral bounds and residual behavior, enabling robust performance in practice.

Proposed method

• Introduce a soft thresholding operation on hierarchical singular values via the hierarchical singular value decomposition (HSVD), replacing hard thresholding used in prior methods.
• Use the non-expansive property of soft thresholding to allow aggressive rank reduction without compromising convergence of the iterative scheme.
• Construct an adaptive algorithm that adjusts the thresholding parameter $\alpha_k$ at each step based on an a posteriori criterion, ensuring quasi-optimality of iterates.
• Integrate the soft thresholding into a Richardson-type iterative solver for $\mathcal{A}\mathbf{u} = \mathbf{f}$, where $\mathbf{u}_{k+1} = \mathbf{S}_{\alpha_k}(\mathbf{u}_k - \mu(\mathcal{A}\mathbf{u}_k - \mathbf{f}))$.
• Develop a perturbed variant of the scheme that allows inexact residual evaluations while retaining convergence and rank control.
• Leverage bounds on the spectrum of $\mathcal{A}$ and the norm of $\mathbf{f}$ to define the adaptive $\alpha_k$ sequence without requiring knowledge of the solution's decay behavior.

Experimental results

Research questions

• RQ1 Can soft thresholding in hierarchical tensor formats be used to control tensor ranks during iterative solution of high-dimensional PDEs?
• RQ2 Does the non-expansive nature of soft thresholding enable stable and convergent iteration with guaranteed rank quasi-optimality?
• RQ3 Can thresholding parameters be adaptively chosen a posteriori using only spectral bounds and residual norms, without prior knowledge of solution structure?
• RQ4 How does the method perform numerically when residuals are computed inexactly, and does it still maintain convergence and rank control?
• RQ5 To what extent does the method reduce rank accumulation compared to prior approaches relying on periodic truncation?

Key findings

• The proposed soft thresholding method ensures that the hierarchical tensor ranks of all iterates remain quasi-optimal with respect to the best low-rank approximation of the true solution.
• The adaptive choice of $\alpha_k$ based on an a posteriori criterion maintains convergence and rank control using only spectral bounds of $\mathcal{A}$ and $\|\mathbf{f}\|$, without prior knowledge of the solution’s decay properties.
• Numerical experiments show monotonic decrease in residual norms and stable rank growth, with maximum ranks of iterates increasing only slightly with dimension $d$ (e.g., $d=16, 32, 64$), far below theoretical worst-case bounds.
• The method exhibits linear convergence to the solution, with the error ratio $\text{err}_k / \|\mathbf{r}_k\|$ remaining uniformly proportional to the $H^1$-norm error, confirming reliable error monitoring.
• The perturbed version of the algorithm retains convergence and rank control even when residuals are computed inexactly, with adaptive thresholding adjusting to changes in iterate differences.
• The method avoids rank accumulation between truncation steps, unlike prior schemes that rely on periodic hard thresholding, thus enabling stable performance even with high-rank operator actions.