[Paper Review] Black-box optimization on Hyper-rectangle using Recursive Modified Pattern Search and Application to Matrix Completion Problem with Non-convex Regularization

Priyam Das|arXiv (Cornell University)|Apr 28, 2016

Advanced Optimization Algorithms Research3 citations

TL;DR

This paper proposes a parallel, pattern search-based black-box optimization algorithm for hyper-rectangular domains, using coordinate-wise step-size adaptation and 2n-direction evaluations per iteration to accelerate convergence. It outperforms GA and SA in computation time on benchmark functions and enables efficient GPU-accelerated matrix completion with non-convex regularization.

ABSTRACT

In this paper, a pattern search based optimization technique is developed to optimize any black-box function on a hyper-rectangle. This algorithm consists of a series of `runs' and inside each `run' iterations are performed until a convergence criteria is satisfied following the principle which is similar to that of Generalized Pattern Search. During an iteration, jumps are made along the co-ordinates of the parameter one at a time with varying step-sizes within the restricted parameter space to search for the best direction to move. While solving a problem on $n$-dimensional hyper-rectangle, inside each iteration the objective function is evaluated at $2n$ independent directions. Hereby parallel computing can be easily incorporated using up to $2n$ (i.e., in the order of $n$) threads which is very convenient for GPU computing. Unlike other existing black-box optimization techniques (e.g., Genetic Algorithm (GA), Simulated Annealing (SA)), the prior knowledge of convexity about the objective function can be exploited and in that case it can be solved in lesser time. The comparative study of the performances of the proposed algorithm, GA and SA have been provided for several low-dimensional, a few moderate and high-dimensional benchmark functions with corresponding computation times. Simulation study has also been performed for moderate and high-dimensional cases where the solution is a boundary point in the domain. This black-box optimization technique has been used to solve matrix completion problem with non-convex regularization incorporating parallel computing.

Motivation & Objective

• To develop a scalable, parallelizable black-box optimization method for functions defined on n-dimensional hyper-rectangles.
• To exploit prior knowledge of convexity to improve convergence speed when applicable.
• To enable efficient GPU computing by supporting up to 2n parallel evaluations per iteration.
• To solve high-dimensional matrix completion problems with non-convex regularization using a derivative-free approach.
• To compare performance against GA and SA on low-, moderate-, and high-dimensional benchmark functions.

Proposed method

• The algorithm performs a series of 'runs', each involving iterative coordinate-wise search with adaptive step-sizes within the parameter bounds.
• At each iteration, the objective function is evaluated at 2n independent directions—positive and negative along each coordinate.
• The method uses a recursive modified pattern search strategy similar to Generalized Pattern Search to guide search direction selection.
• Parallelization is natively supported via up to 2n threads, enabling efficient GPU implementation.
• The algorithm dynamically adjusts step-sizes based on function evaluation outcomes to improve convergence.
• It integrates well with non-convex regularization in matrix completion by treating the objective as a black-box function.

Experimental results

Research questions

• RQ1 How does the proposed pattern search algorithm compare in convergence speed and accuracy to GA and SA on standard benchmark functions?
• RQ2 To what extent can parallelization via 2n evaluations per iteration improve computational efficiency on GPU architectures?
• RQ3 Can the algorithm efficiently locate boundary solutions in moderate- and high-dimensional optimization problems?
• RQ4 How effective is the method in solving matrix completion with non-convex regularization when the objective is black-box?
• RQ5 Does prior knowledge of convexity significantly reduce computation time in the proposed framework?

Key findings

• The proposed algorithm achieves faster convergence than GA and SA on all tested benchmark functions, with significant reductions in computation time.
• The algorithm demonstrates strong scalability in high-dimensional problems due to its native support for 2n-parallel evaluations.
• For problems where the optimal solution lies on the boundary of the domain, the algorithm successfully locates the solution with high accuracy.
• The integration of the method with non-convex regularization in matrix completion yields competitive results, validated through simulation.
• When convexity is known a priori, the algorithm converges in fewer iterations and less time than in non-convex cases.
• The method shows clear advantages in GPU-accelerated settings due to its regular, data-parallel evaluation pattern.