[Paper Review] VC bounds on the cardinality of nearly orthogonal function classes

Lee-Ad Gottlieb, Leonid+2 more|arXiv (Cornell University)|Jul 28, 2010

Machine Learning and Algorithms22 references4 citations

TL;DR

This paper presents improved VC-dimension-based bounds on the maximum number of nearly orthogonal vectors in \{−1,1\}^n, using a novel projection argument and entropy-based analysis of binomial coefficients. The key contribution is a tighter bound on packing numbers for function classes with near-orthogonality (γ ≈ 0), significantly improving Haussler's bound and extending to k-ary alphabets.

ABSTRACT

We bound the number of nearly orthogonal vectors with fixed VC-dimension over $\setpm^n$. Our bounds are of interest in machine learning and empirical process theory and improve previous bounds by Haussler. The bounds are based on a simple projection argument and the generalize to other product spaces. Along the way we derive tight bounds on the sum of binomial coefficients in terms of the entropy function.

Motivation & Objective

• To improve existing bounds on the packing number M(ε,d) for function classes with VC-dimension d and nearly orthogonal vectors (ε ≈ 1/2).
• To resolve the open problem of tightening the constant gap in Haussler's bound by providing a simpler construction that closes the gap to [1, 2e].
• To extend the analysis beyond binary alphabets to k-ary function classes using generalized VC-dimension and entropy-based summation bounds.
• To show that M(ε,d) cannot be bounded by a polynomial in n when γ = 1−2ε = O(1/poly(n)), resolving a negative result for poly(n) bounds in this regime.

Proposed method

• Uses a projection argument to bound the size of ε-separated subsets in function classes with bounded VC-dimension.
• Applies entropy-based bounds on the sum of binomial coefficients ∑_{i=0}^d \binom{n}{i}, deriving tight upper bounds via Taylor expansion and concavity of the binary entropy function.
• Introduces a generalized entropy-based bound for k-ary alphabets: ∑_{i=0}^d \binom{n}{i}k^i < 0.94 · 2^{nH(d/n) + d log k}.
• Establishes a new upper bound M((1−γ)/2,d) ≤ 100 · 2^{dβ(γ)} where β(γ) is a function derived from entropy and vector orthogonality.
• Leverages the relationship between normalized Hamming distance ρ and inner product ⟨x,y⟩ via 2ρ(x,y) + ⟨x,y⟩ = 1 to reframe orthogonality in terms of γ = 1−2ε.
• Uses concentration and tail bounds on binomial coefficients to derive the entropy-based upper bounds, with explicit constants via Taylor expansion.

Experimental results

Research questions

• RQ1 Can the packing number M(ε,d) for nearly orthogonal function classes (ε ≈ 1/2) be bounded more tightly than Haussler’s bound?
• RQ2 What is the optimal asymptotic growth rate of log M(ε,d) as a function of d and ε near 1/2?
• RQ3 Can Haussler’s bound be improved by a constant factor, particularly closing the gap between 1/2e and 2e in the key constant?
• RQ4 Does a polynomial-in-n bound on M(ε,d) hold when γ = 1−2ε = O(1/poly(n))?
• RQ5 Can the binary VC-dimension bounds be generalized to k-ary alphabets using a generalized VC-dimension framework?

Key findings

• The paper establishes a new upper bound M((1−γ)/2,d) ≤ 100 · 2^{dβ(γ)}, which asymptotically grows as (ln 2)β(γ) per d, significantly improving Haussler’s bound of ln(4e/(1−γ)) per d for small γ.
• For small γ, the new bound improves upon Haussler’s by a super-exponential factor in d, with the ratio of logarithmic growth rates approaching a constant factor improvement.
• The authors derive a tight bound ∑_{i=0}^d \binom{n}{i} < 0.98 · 2^{nH(d/n)} using entropy function analysis and Taylor expansion, improving upon previous estimates.
• For k-ary alphabets, the bound ∑_{i=0}^d \binom{n}{i}k^i < 0.94 · 2^{nH(d/n) + d log k} is established, providing the first such packing bound in terms of generalized VC-dimension.
• The paper proves that M(ε,d) cannot be bounded by a polynomial in n when γ = 1−2ε = O(1/poly(n)), resolving a negative result for such bounds.
• A simpler construction is provided that achieves the lower bound M(ε,d) ≥ (1/(2e(ε + d/n)))^d, closing the constant gap in Haussler’s bound to [1, 2e], matching recent results but with a more direct proof.