[Paper Review] Negative Examples for Sequential Importance Sampling of Binary Contingency Tables

Ivona Bezáková, Alistair Sinclair+2 more|arXiv (Cornell University)|Jun 26, 2006

Markov Chains and Monte Carlo Methods8 references4 citations

TL;DR

This paper demonstrates that sequential importance sampling (SIS) for binary contingency tables can systematically underestimate the true number of valid 0/1 matrices by an exponential factor, even after subexponential numbers of trials, regardless of row/column ordering. The authors present a theoretical family of counterexamples and experimental evidence showing SIS converges slowly and inaccurately on structured inputs, highlighting fundamental limitations of SIS despite its empirical popularity.

ABSTRACT

The sequential importance sampling (SIS) algorithm has gained considerable popularity for its empirical success. One of its noted applications is to the binary contingency tables problem, an important problem in statistics, where the goal is to estimate the number of 0/1 matrices with prescribed row and column sums. We give a family of examples in which the SIS procedure, if run for any subexponential number of trials, will underestimate the number of tables by an exponential factor. This result holds for any of the usual design choices in the SIS algorithm, namely the ordering of the columns and rows. These are apparently the first theoretical results on the efficiency of the SIS algorithm for binary contingency tables. Finally, we present experimental evidence that the SIS algorithm is efficient for row and column sums that are regular. Our work is a first step in determining the class of inputs for which SIS is effective.

Motivation & Objective

• To investigate the theoretical reliability of sequential importance sampling (SIS) for estimating the number of binary contingency tables with given row and column sums.
• To identify input structures where SIS fails to converge to accurate estimates, even with subexponential sampling.
• To challenge the common assumption that SIS is a robust and efficient method for this combinatorial estimation problem.
• To provide experimental and theoretical evidence that SIS can be exponentially inefficient for certain structured row and column sum configurations.

Proposed method

• The authors construct a family of binary contingency table instances with specific row and column sums that cause SIS to systematically underestimate the true count by an exponential factor.
• They analyze the SIS algorithm's behavior under any row and column ordering, proving that the estimator converges to a value exponentially smaller than the true cardinality.
• The method relies on analyzing the importance weights in SIS, showing that the probability distribution used in sampling leads to highly biased estimates for certain input types.
• The authors use a stopping heuristic to measure convergence time and compare it across different input types, including regular and irregular degree sequences.
• They extend the analysis to an enhanced SIS variant that alternates between updating rows and columns based on residual sums, testing its performance on new counterexamples.
• Experimental evaluation involves 30 million SIS trials on a specific hard instance, measuring variance and convergence of estimates across multiple runs.

Experimental results

Research questions

• RQ1 Can sequential importance sampling (SIS) for binary contingency tables be provably inaccurate even with subexponential numbers of trials?
• RQ2 Are there input configurations for which SIS consistently underestimates the true number of contingency tables by an exponential factor?
• RQ3 Does the performance of SIS depend on the ordering of rows and columns, or are there inherent structural limitations regardless of design choices?
• RQ4 Can an enhanced SIS strategy that alternates between rows and columns improve convergence on hard instances?
• RQ5 How do experimental results compare with theoretical predictions regarding convergence time and estimator bias?

Key findings

• The SIS algorithm can underestimate the true number of binary contingency tables by an exponential factor, even after a subexponential number of trials, for any row and column ordering.
• The authors construct a specific family of inputs where the SIS estimator converges to a value exponentially smaller than the true cardinality, proving that SIS is not universally reliable.
• For regular row and column sums (e.g., 5, 10, or n/2-regular), SIS converges in subexponential time and produces accurate estimates, indicating its effectiveness on structured inputs.
• On a more complex input with alternating low and high sums (e.g., 1s and floor(m/2)s), the enhanced SIS variant still fails to converge after 30 million trials, with estimates varying by a factor of 10^3.
• The experimental results suggest that the enhanced SIS algorithm may also require exponential time on certain structured inputs, though a formal proof remains elusive.
• The paper demonstrates that SIS, despite being an unbiased estimator in theory, can converge to a wrong value due to poor importance sampling weights, challenging its practical reliability.