[Paper Review] Gradient Descent-based D-optimal Design for the Least-Squares Polynomial Approximation

Vitaly Zankin, Gleb Ryzhakov+1 more|arXiv (Cornell University)|Jun 18, 2018

Probabilistic and Robust Engineering DesignDecision Sciences6 references4 citations

TL;DR

This paper proposes a gradient descent-based method for D-optimal experimental design in least-squares polynomial approximation, optimizing the determinant of the information matrix to minimize approximation error. The method achieves superior accuracy with fewer samples compared to LHS, Sobol’ sequences, and Maxvol sampling, while exhibiting slow Lebesgue constant growth.

ABSTRACT

In this work, we propose a novel sampling method for Design of Experiments. This method allows to sample such input values of the parameters of a computational model for which the constructed surrogate model will have the least possible approximation error. High efficiency of the proposed method is demonstrated by its comparison with other sampling techniques (LHS, Sobol' sequence sampling, and Maxvol sampling) on the problem of least-squares polynomial approximation. Also, numerical experiments for the Lebesgue constant growth for the points sampled by the proposed method are carried out.

Motivation & Objective

• To develop a sampling method that minimizes approximation error in least-squares polynomial approximation by optimizing experimental design.
• To address the limitation of traditional space-filling designs (e.g., LHS, Sobol’ sequences) in achieving high accuracy with minimal samples.
• To improve surrogate modeling efficiency in uncertainty quantification by minimizing the condition number of the design matrix.
• To demonstrate the effectiveness of gradient-based optimization for D-optimal design in multivariate polynomial approximation.

Proposed method

• Optimizes the D-optimality criterion by maximizing the determinant of the information matrix $ A^T A $, where $ A $ is the model matrix of basis polynomial evaluations.
• Derives an analytical gradient of the log-determinant objective function with respect to each design point, enabling efficient optimization.
• Uses gradient descent to iteratively update design points to maximize $ ext{det}(A^T A) $, improving the conditioning of the least-squares problem.
• Applies the IPOP-optimization algorithm to handle non-rectangular domains and ensure robust convergence.
• Incorporates analytical derivatives of univariate orthogonal polynomials to compute the gradient efficiently.
• Validates the method on multivariate polynomial approximation tasks with varying domain shapes and polynomial degrees.

Experimental results

Research questions

• RQ1 Can gradient-based optimization of the D-optimality criterion produce more accurate least-squares polynomial approximations than standard sampling methods?
• RQ2 How does the proposed method compare to LHS, Sobol’ sequences, and Maxvol sampling in terms of approximation error and condition number?
• RQ3 What is the growth behavior of the Lebesgue constant for points selected by the proposed method across different domain geometries?
• RQ4 Can the method be effectively applied to non-rectangular domains while maintaining high design quality?

Key findings

• The proposed gradient descent-based D-optimal design significantly reduces approximation error compared to LHS, Sobol’ sequences, and Maxvol sampling across multiple test functions.
• The method achieves a lower condition number of the design matrix, indicating better numerical stability in the least-squares solution.
• Lebesgue constant growth for the proposed design is slow, indicating good interpolation properties and stability in polynomial approximation.
• The method effectively samples non-rectangular domains (e.g., circular, diamond-shaped) with good spatial coverage and minimal clustering.
• Numerical experiments confirm that the proposed method outperforms standard sampling techniques even with a reduced number of samples.
• The analytical gradient computation enables efficient optimization, making the method scalable to higher-dimensional problems.