[Paper Review] Random generation of optimal saturated designs

Roberto Fontana|arXiv (Cornell University)|Mar 26, 2013

Optimal Experimental Design Methods1 references3 citations

TL;DR

This paper proposes a Bayesian nonparametric method using discovery probability estimation to determine when to stop searching for optimal saturated designs in experimental design. By modeling new optimal designs as 'new species' in a population, it uses a penalized likelihood approach to estimate the probability of discovering a better design, enabling data-driven stopping rules for algorithms like SAS Proc Optex.

ABSTRACT

Efficient algorithms for searching for optimal saturated designs are widely available. They maximize a given efficiency measure (such as D-optimality) and provide an optimum design. Nevertheless, they do not guarantee a \emph{global} optimal design. Indeed, they start from an initial random design and find a local optimal design. If the initial design is changed the optimum found will, in general, be different. A natural question arises. Should we stop at the design found or should we run the algorithm again in search of a better design? This paper uses very recent methods and software for discovery probability to support the decision to continue or stop the sampling. A software tool written in SAS has been developed.

Motivation & Objective

• To address the challenge of determining when to stop searching for optimal saturated designs in experimental design, especially when algorithms may converge to local optima.
• To develop a decision-support methodology that determines whether additional search iterations are likely to yield better designs.
• To apply recent advances in discovery probability estimation—originally from species richness estimation—to the problem of optimal design generation.
• To provide a practical, software-based solution for researchers to avoid unnecessary computation while maximizing the chance of finding globally optimal designs.
• To extend the utility of existing optimization tools like SAS Proc Optex by integrating a principled stopping criterion based on statistical learning theory.

Proposed method

• Models the discovery of new optimal designs as a species discovery problem, where each unique D-optimal design is a 'species'.
• Employs a Bayesian nonparametric estimator of discovery probability based on Favaro et al. (2012), adapted to the discrete optimization context of design generation.
• Uses a penalized log-likelihood function to handle parameter space constraints, particularly when the gamma function terms become undefined.
• Applies a distance-based penalty to log-likelihood components when parameters fall outside the feasible region (e.g., θ + iσ ≤ 0), ensuring numerical stability.
• Implements a recursive stopping rule: if the estimated probability of discovering a new optimal design in the next iteration is below a threshold p⋆, the algorithm halts.
• Integrates the method with SAS Proc Optex, enabling users to run multiple search iterations and apply the stopping rule automatically.

Experimental results

Research questions

• RQ1 How can we determine when to stop searching for optimal saturated designs without guaranteeing global optimality?
• RQ2 To what extent can discovery probability estimation from species richness models be adapted to discrete optimization problems in experimental design?
• RQ3 What is the optimal stopping rule for iterative design search algorithms that balances computational cost and the likelihood of finding a better design?
• RQ4 How can penalized likelihood estimation improve the robustness of discovery probability estimation in constrained parameter spaces?
• RQ5 Can a software tool be developed to automate the stopping decision in optimal design generation using real-time discovery probability updates?

Key findings

• The discovery probability estimator provides a statistically grounded method to assess whether further search iterations are likely to yield better designs.
• The penalized likelihood approach ensures numerical stability even when parameter values approach or enter non-feasible regions of the parameter space.
• The method successfully identifies when the probability of discovering a new optimal design falls below a threshold (p⋆), justifying algorithm termination.
• The developed SAS software tool enables users to apply the stopping rule to any design problem with customizable factors, levels, and models.
• Empirical results show that the method avoids unnecessary computation by halting once the likelihood of finding a better design diminishes, improving search efficiency.
• The approach is generalizable beyond D-optimality and can be adapted to other criteria such as A-optimality or G-optimality.