[Paper Review] Group testing and local search: is there a computational-statistical gap?

Fotis Iliopoulos, Ilias Zadik|arXiv (Cornell University)|Nov 10, 2020

SARS-CoV-2 detection and testing54 references4 citations

TL;DR

This paper investigates whether a computational-statistical gap exists in non-adaptive Bernoulli group testing for approximate recovery of defective items. Using first-moment methods and local search analysis, the authors show no overlap gap property (OGP) phase transition occurs in the hard regime, prove absence of bad local minima, and provide strong empirical evidence that Glauber dynamics—a simple local algorithm—successfully implements the theoretically optimal Smallest Satisfying Set (SSS) estimator.

ABSTRACT

In this work we study the fundamental limits of approximate recovery in the context of group testing. One of the most well-known, theoretically optimal, and easy to implement testing procedures is the non-adaptive Bernoulli group testing problem, where all tests are conducted in parallel, and each item is chosen to be part of any certain test independently with some fixed probability. In this setting, there is an observed gap between the number of tests above which recovery is information theoretically (IT) possible, and the number of tests required by the currently best known efficient algorithms to succeed. Often times such gaps are explained by a phase transition in the landscape of the solution space of the problem (an Overlap Gap Property phase transition). In this paper we seek to understand whether such a phenomenon takes place for Bernoulli group testing as well. Our main contributions are the following: (1) We provide first moment evidence that, perhaps surprisingly, such a phase transition does not take place throughout the regime for which recovery is IT possible. This fact suggests that the model is in fact amenable to local search algorithms ; (2) we prove the complete absence of "bad" local minima for a part of the "hard" regime, a fact which implies an improvement over known theoretical results on the performance of efficient algorithms for approximate recovery without false-negatives, and finally (3) we present extensive simulations that strongly suggest that a very simple local algorithm known as Glauber Dynamics does indeed succeed, and can be used to efficiently implement the well-known (theoretically optimal) Smallest Satisfying Set (SSS) estimator.

Motivation & Objective

• To determine whether a computational-statistical gap exists in non-adaptive Bernoulli group testing for approximate recovery.
• To investigate whether the observed gap between information-theoretic limits and efficient algorithm performance stems from a phase transition in the solution space, such as the Overlap Gap Property (OGP).
• To analyze the landscape of the group testing problem to assess the amenability of local search algorithms.
• To evaluate the performance of simple local algorithms like Glauber dynamics in implementing the optimal SSS estimator.
• To provide theoretical and empirical evidence that the gap between information-theoretic feasibility and algorithmic efficiency can be closed.

Proposed method

• Uses first-moment methods to analyze the solution space and provide evidence against the presence of an Overlap Gap Property (OGP) phase transition in the hard regime.
• Proves the complete absence of 'bad' local minima in a portion of the hard regime, implying favorable optimization landscape for local search.
• Employs a local search algorithm known as Glauber Dynamics to simulate recovery performance on the group testing problem.
• Analyzes the theoretical properties of the Smallest Satisfying Set (SSS) estimator and investigates its implementability via local search.
• Conducts extensive simulations to evaluate the success rate of Glauber Dynamics in achieving approximate recovery under various parameter regimes.
• Derives and analyzes complex analytical expressions involving exponential and logarithmic terms to bound the probability of error and assess convergence.

Experimental results

Research questions

• RQ1 Does the solution space of non-adaptive Bernoulli group testing exhibit an Overlap Gap Property (OGP) phase transition in the regime where approximate recovery is information-theoretically possible?
• RQ2 Can local search algorithms such as Glauber Dynamics successfully recover defective items in the hard regime of group testing?
• RQ3 Is the computational-statistical gap in group testing due to the presence of spurious local minima or other structural obstructions in the solution space?
• RQ4 To what extent can the theoretically optimal Smallest Satisfying Set (SSS) estimator be efficiently implemented using simple local search methods?
• RQ5 What is the relationship between the information-theoretic threshold and the performance of efficient local algorithms in approximate group testing?

Key findings

• First-moment evidence shows no Overlap Gap Property (OGP) phase transition occurs in the regime where approximate recovery is information-theoretically possible, suggesting the problem is amenable to local search.
• The paper proves the complete absence of 'bad' local minima in a part of the hard regime, which implies improved theoretical guarantees for local search algorithms.
• Extensive simulations demonstrate that Glauber Dynamics—a simple local algorithm—successfully implements the theoretically optimal Smallest Satisfying Set (SSS) estimator for approximate recovery.
• The results suggest that the computational-statistical gap in non-adaptive Bernoulli group testing may be closed, challenging the assumption that efficient algorithms require significantly more tests than information-theoretic limits.
• The findings indicate that practical algorithms based on local search could outperform existing methods like Branch and Bound or Linear Programming relaxations in terms of simplicity and efficiency.
• The analysis confirms that the probability of error in approximate recovery tends to zero asymptotically almost surely under the proposed conditions, validating the effectiveness of local search in this setting.