[Paper Review] Nearly optimal classification for semimetrics

Lee-Ad Gottlieb, Aryeh Kontorovich|arXiv (Cornell University)|Feb 22, 2015

Machine Learning and Algorithms28 references3 citations

TL;DR

This paper introduces the density dimension as a key measure for classification in semimetric spaces—non-metric spaces lacking the triangle inequality—showing it governs sample complexity and algorithmic efficiency. It presents nearly optimal sample compression schemes and generalization bounds with fast rates, demonstrating that learning in semimetrics requires fundamentally different analysis than in metric spaces, where doubling dimension dominates.

ABSTRACT

We initiate the rigorous study of classification in semimetric spaces, which are point sets with a distance function that is non-negative and symmetric, but need not satisfy the triangle inequality. For metric spaces, the doubling dimension essentially characterizes both the runtime and sample complexity of classification algorithms --- yet we show that this is not the case for semimetrics. Instead, we define the {\em density dimension} and discover that it plays a central role in the statistical and algorithmic feasibility of learning in semimetric spaces. We present nearly optimal sample compression algorithms and use these to obtain generalization guarantees, including fast rates. The latter hold for general sample compression schemes and may be of independent interest.

Motivation & Objective

• To establish a rigorous statistical and algorithmic framework for classification in semimetric spaces, which lack the triangle inequality.
• To identify the density dimension as the central parameter governing learning feasibility in semimetric spaces, replacing the role of doubling dimension in metric spaces.
• To develop nearly optimal sample compression schemes that yield fast generalization rates for semimetric classification.
• To prove that sample complexity in semimetrics is inherently exponential in the density dimension, even under margin assumptions.

Proposed method

• Introduces the density dimension μ(S) as a measure of intrinsic geometric complexity in semimetric spaces, defined via the maximum number of nearly equidistant points.
• Proposes a sample compression scheme based on γ-nets of labeled subsets, where compression size depends on μ(S) and the logarithm of the ratio between radius and margin.
• Derives generalization bounds using a tractable variant of the optimal compression objective, leveraging a modified loss function to ensure computational feasibility.
• Employs a margin-based analysis to show that (k,γ)-separable samples admit compression of size μ(S)^{log₂(2rad(S)/γ)} with error at most k/|S|.
• Uses a VC-style lower bound argument with a distribution over k nearly equidistant points to show that sample complexity is exponential in the density dimension.
• Applies Bernstein-type concentration inequalities to derive high-probability generalization bounds under margin assumptions.

Experimental results

Research questions

• RQ1 How does the absence of the triangle inequality in semimetric spaces affect the sample complexity and algorithmic efficiency of classification?
• RQ2 What geometric parameter in semimetric spaces plays a role analogous to doubling dimension in metric spaces?
• RQ3 Can sample compression schemes in semimetric spaces achieve nearly optimal generalization bounds with fast rates?
• RQ4 Is the density dimension a tight characterization of sample complexity in semimetric classification?

Key findings

• The density dimension μ(S) is shown to be the central parameter governing both sample complexity and algorithmic efficiency in semimetric classification, replacing the role of doubling dimension.
• A sample compression scheme is constructed whose size is bounded by μ(S)^{log₂(2rad(S)/γ)}, achieving a generalization error of at most k/|S| for (k,γ)-separable samples.
• The paper establishes a generalization bound R(k,γ) = Q(d,k/n) with d = μ(S)^{log₂(2rad(S)/γ)}, which is efficiently optimizable and yields fast rates.
• A sample complexity lower bound is proven: for some distributions, any learning algorithm requires Ω(√(μ(𝒳)^{log₂(2rad(S)/marg(S))}) / n) error, showing exponential dependence on the density dimension.
• The optimal compression objective Q* is shown to be NP-hard to compute, but a tractable relaxation enables efficient optimization and generalization guarantees.
• The results demonstrate that semimetric spaces require fundamentally different analysis than metric spaces, as the packing-covering relationship breaks without the triangle inequality.