[Paper Review] Secant acceleration of sequential residual methods for solving large-scale nonlinear systems of equations

Ernesto G. Birgin, J. M. Martı́nez|arXiv (Cornell University)|Dec 24, 2020

Advanced Optimization Algorithms Research51 references4 citations

TL;DR

This paper proposes an accelerated variant of the DF-SANE algorithm for solving large-scale nonlinear systems of equations by incorporating a secant-based acceleration technique into sequential residual methods. The method reduces iteration count and computational time—especially for difficult problems—by leveraging past iterates to improve convergence, demonstrating up to a tenfold speedup over NITSOL on challenging 3D Bratu problems with high condition numbers.

ABSTRACT

Sequential Residual Methods try to solve nonlinear systems of equations $F(x)=0$ by iteratively updating the current approximate solution along a residual-related direction. Therefore, memory requirements are minimal and, consequently, these methods are attractive for solving large-scale nonlinear systems. However, the convergence of these algorithms may be slow in critical cases; therefore, acceleration procedures are welcome. In this paper, we suggest to employ a variation of the Sequential Secant Method in order to accelerate Sequential Residual Methods. The performance of the resulting algorithm is illustrated by applying it to the solution of very large problems coming from the discretization of partial differential equations.

Motivation & Objective

• To address slow convergence of sequential residual methods (SRM) in large-scale nonlinear systems, especially when backtracking is frequent.
• To develop an efficient acceleration technique that minimizes residual evaluations and memory usage while improving convergence speed.
• To extend the global convergence theory of DF-SANE to include secant-based acceleration using residual-related directions.
• To evaluate performance on large-scale PDE discretizations, particularly in high-condition-number regimes where standard methods struggle.
• To compare the accelerated method against NITSOL and standard DF-SANE in terms of iterations, functional evaluations, and CPU time.

Proposed method

• Introduces a generalized sequential residual method that allows non-residual but residual-related search directions, improving flexibility and convergence.
• Applies a multipoint secant method to approximate the inverse Jacobian using past iterates and residual differences, reducing the need for exact Jacobian computation.
• Employs a variable-memory secant update strategy with parameter $ p $, controlling the number of past iterates used in the approximation.
• Uses a line search with sufficient descent condition to ensure global convergence, while minimizing functional evaluations.
• Implements a dynamic step size selection based on spectral (Barzilai-Borwein-like) principles to enhance convergence in early iterations.
• Integrates the acceleration into the DF-SANE framework, forming Accelerated DF-SANE, with theoretical convergence guarantees extended to the new variant.

Experimental results

Research questions

• RQ1 Can secant-based acceleration significantly reduce the number of iterations and functional evaluations in sequential residual methods for large-scale nonlinear systems?
• RQ2 How does the performance of the accelerated method compare to NITSOL and standard DF-SANE on difficult PDE discretizations?
• RQ3 What is the impact of the memory parameter $ p $ on the convergence speed and computational cost of the accelerated method?
• RQ4 Does the acceleration technique maintain global convergence while improving local convergence rates?
• RQ5 How does the method perform on ill-conditioned problems, particularly when $ \theta $ in the Bratu problem is negative and large in magnitude?

Key findings

• On the 3D Bratu problem with $ n_p = 40 $ and $ \theta = -100 $, Accelerated DF-SANE outperformed NITSOL by approximately an order of magnitude in CPU time, reaching convergence in under 1000 seconds versus over 10,000 seconds for NITSOL.
• For $ \theta \geq -20 $, both methods performed similarly, with NITSOL slightly faster; however, for $ \theta < -20 $, Accelerated DF-SANE showed a clear advantage, especially in high-condition-number regimes.
• The number of iterations of Accelerated DF-SANE remained nearly constant as $ p $ increased from 3 to 17, varying by at most 20% across the tested range.
• CPU time per iteration increased with $ p $, with a measured linear fit $ L(p) \approx 4.08 + 1.01p $, indicating a 300% increase in time cost from $ p=3 $ to $ p=17 $.
• DF-SANE failed to converge within a reasonable time (hours) on the most difficult instances ($ \theta = -100 $), while Accelerated DF-SANE reached the stopping criterion in under 1000 seconds.
• The number of functional evaluations was consistently lower for Accelerated DF-SANE than for NITSOL on hard problems, confirming reduced evaluation overhead despite higher per-iteration cost.