[Paper Review] Up-and-Down and the Percentile-Finding Problem

Assaf P. Oron|ArXiv.org|Aug 21, 2008

Statistical Methods in Clinical Trials57 references10 citations

TL;DR

This paper advances Up-and-Down (U&D) designs for percentile estimation by proving the k-in-a-row (KR) variant has a unimodal stationary distribution and outperforms others in convergence and precision. It introduces centered isotonic regression (CIR) and the auto-detect estimator for nonparametric estimation, develops interval estimation, and proposes a hybrid Bayesian Up-and-Down (BUD) design that improves robustness over CRM while matching or exceeding U&D and CRM performance in simulation studies.

ABSTRACT

Up-and-Down (U&D) is a popular sequential design for estimating threshold percentiles in binary experiments. However, U&D application practices have stagnated, and significant gaps in understanding its properties persist. The first part of my work aims to fill gaps in U&D theory. New results concerning stationary distribution properties are proven. A second focus of this study is nonparametric U&D estimation. An improvement to isotonic regression called "centered isotonic regression" (CIR), and a new averaging estimator called "auto-detect" are introduced and their properties studied. Bayesian percentile-finding designs, most notably the continual reassessment method (CRM) developed for Phase I clinical trials, are also studied. In general, CRM convergence depends upon random run-time conditions -- meaning that convergence is not always assured. Small-sample behavior is studied as well. It is shown that CRM is quite sensitive to outlier sub-sequences of thresholds, resulting in highly variable small-sample behavior between runs under identical conditions. Nonparametric CRM variants exhibit a similar sensitivity. Ideas to combine the advantages of U&D and Bayesian designs are examined. A new approach is developed, using a hybrid framework, that evaluates the evidence for overriding the U&D allocation with a Bayesian one.

Motivation & Objective

• To close critical theoretical gaps in Up-and-Down (U&D) design theory, particularly regarding stationary distribution properties and convergence behavior.
• To improve nonparametric estimation in U&D by introducing centered isotonic regression (CIR) and the auto-detect estimator, with associated interval estimation.
• To rigorously analyze the convergence and small-sample behavior of Bayesian continual reassessment method (CRM) designs, challenging assumptions of guaranteed convergence.
• To develop a hybrid design, Bayesian Up-and-Down (BUD), that combines U&D and CRM advantages, enhancing robustness and estimation performance.
• To provide practical guidance for selecting optimal U&D and hybrid designs under real-world constraints like boundary effects and experimental variability.

Proposed method

• Proves that the k-in-a-row (KR) U&D variant has a single-mode stationary distribution, contradicting prior literature claims.
• Introduces centered isotonic regression (CIR), a modified isotonic regression that pools estimates around a central point to improve precision and reduce bias.
• Develops the auto-detect estimator, which adaptively averages reversals in U&D sequences to improve estimation efficiency.
• Derives interval estimation procedures for both CIR and auto-detect estimators using asymptotic and simulation-based methods.
• Proposes the Bayesian Up-and-Down (BUD) design, which uses a hybrid framework to override U&D allocations with Bayesian rules when evidence accumulates.
• Employs simulation studies and real data from an anesthesiology experiment to evaluate performance across diverse threshold distributions and sample sizes.

Experimental results

Research questions

• RQ1 Does the k-in-a-row (KR) U&D design have a unimodal stationary distribution, and how does it compare to other U&D variants in convergence rate and estimation precision?
• RQ2 Can centered isotonic regression (CIR) significantly improve nonparametric estimation in U&D designs, and what are its theoretical properties?
• RQ3 What is the true small-sample behavior of the continual reassessment method (CRM), and under what conditions does it converge to the optimal dose?
• RQ4 How does the performance of the proposed BUD hybrid design compare to standalone U&D and CRM designs in terms of estimation accuracy and robustness?
• RQ5 To what extent do boundary effects and outlier sequences impact the reliability of CRM and nonparametric CRM variants in small samples?

Key findings

• The k-in-a-row (KR) U&D variant has a unimodal stationary distribution and is proven to be the fastest-converging and most precise U&D design for estimating below-median percentiles such as Q₀.₃ and Q₀.₂.
• Centered isotonic regression (CIR) reduces bias and mean squared error (MSE) compared to standard isotonic regression, especially in small samples and under non-uniform threshold distributions.
• The auto-detect estimator achieves better estimation efficiency than both isotonic regression and reversal-averaging methods, particularly in non-normal threshold settings.
• CRM does not always converge to the optimal dose; its convergence is sensitive to random run-time conditions, and small-sample behavior is highly variable due to outlier sequences.
• The BUD hybrid design outperforms both standalone U&D and CRM in estimation precision and is significantly more robust to small-sample variability and boundary effects.
• In the anesthesiology experiment, BUD maintained stable performance across runs, while CRM and CCD showed high run-to-run variability, confirming the benefit of hybridization.