[Paper Review] optimParallel: an R Package Providing Parallel Versions of the Gradient-Based Optimization Methods of optim()

Florian Gerber, Reinhard Furrer|arXiv (Cornell University)|Apr 30, 2018

Machine Learning and Data Classification3 references3 citations

TL;DR

This paper introduces optimParallel, an R package that provides parallel implementations of gradient-based optimization methods from R's optim() function. By leveraging R's lexical scoping to cache and reuse function evaluations, it enables concurrent computation of the objective function and gradient, reducing optimization time by up to a factor of 1+2p for functions with slow evaluation times (>0.05 seconds), especially when analytic gradients are used.

ABSTRACT

The R package optimParallel provides a parallel version of the gradient-based optimization methods of optim(). The main function of the package is optimParallel(), which has the same usage and output as optim(). Using optimParallel() can significantly reduce optimization times. We introduce the R package and illustrate its implementation, which takes advantage of the lexical scoping mechanism of R.

Motivation & Objective

• To address long optimization times in R when using gradient-based methods like L-BFGS-B, BFGS, and CG on computationally intensive functions.
• To develop a drop-in replacement for optim() that enables parallel evaluation of the objective function and gradient without altering the interface.
• To reduce optimization time significantly by exploiting parallel computing resources through efficient function evaluation caching.
• To ensure feature and result compatibility with the original optim() function while enabling new features like log information capture and forward difference gradient approximation.

Proposed method

• The package implements a wrapper function, optimParallel(), that mirrors the interface and output of the base R function optim().
• It uses R's lexical scoping to cache the results of objective function (fn) and gradient (gr) evaluations, avoiding redundant computations when the same parameter vector is reused.
• For each iteration, fn() and gr() are evaluated in parallel using multiple processor cores, reducing the effective time per iteration to max(T_fn, T_gr) when both are computed.
• When no analytic gradient is provided, the package uses a forward difference approximation to reduce the number of fn() evaluations from 1+2p to 1+p per iteration.
• The implementation reuses the stable C code of optim() to guarantee identical results, ensuring correctness and reproducibility.
• It integrates with the R parallel package via makeCluster and setDefaultCluster to manage parallel execution environments.

Experimental results

Research questions

• RQ1 Can parallel evaluation of the objective and gradient functions in R's optim() significantly reduce optimization time for computationally intensive functions?
• RQ2 How does the performance of optimParallel() compare to the original optim() function in terms of speedup across different function evaluation times and parameter dimensions?
• RQ3 What is the impact of using analytic versus numeric gradients on the performance gains of the parallel optimization approach?
• RQ4 Can the lexical scoping mechanism in R be effectively exploited to cache and reuse function evaluations in a parallel optimization context without introducing overhead or errors?
• RQ5 What is the minimum function evaluation time threshold for which the parallel overhead of optimParallel() becomes justified?

Key findings

• For functions with evaluation times exceeding 0.05 seconds, optimParallel() achieves a speedup of approximately 2× when an analytic gradient is provided.
• Without an analytic gradient, the speedup scales with the number of parameters, achieving a theoretical speedup of 1+2p due to parallel evaluation of 1+2p objective function calls.
• The elapsed time per iteration in optimParallel() is consistently close to T_fn, the time to evaluate the objective function alone, regardless of the number of parameters or gradient specification.
• The benchmark results show that optimParallel() maintains feature parity with optim(), producing identical optimization results across all test cases.
• The use of forward difference approximation reduces the number of fn() evaluations from 1+2p to 1+p per iteration, improving efficiency when fewer than 1+2p cores are available.
• The implementation introduces only a small computational overhead, making it beneficial for functions with evaluation times above 0.05 seconds.