[Paper Review] Extrapolating Expected Accuracies for Large Multi-Class Problems

Charles Zheng, Rakesh Achanta+1 more|arXiv (Cornell University)|Dec 27, 2017

Domain Adaptation and Few-Shot Learning16 references3 citations

TL;DR

This paper proposes a method to extrapolate expected classification accuracy for large multi-class problems by modeling the expected accuracy as the (k−1)th moment of a discriminability distribution, estimated from small-k training data. The approach enables unbiased prediction of performance on larger label sets under i.i.d. class sampling and classifier independence assumptions, validated on facial recognition and OCR tasks with state-of-the-art accuracy extrapolation.

ABSTRACT

The difficulty of multi-class classification generally increases with the number of classes. Using data from a subset of the classes, can we predict how well a classifier will scale with an increased number of classes? Under the assumptions that the classes are sampled identically and independently from a population, and that the classifier is based on independently learned scoring functions, we show that the expected accuracy when the classifier is trained on k classes is the (k-1)st moment of a certain distribution that can be estimated from data. We present an unbiased estimation method based on the theory, and demonstrate its application on a facial recognition example.

Motivation & Objective

• To predict the expected generalization accuracy of a classifier on a larger, unobserved label set based on performance observed on a smaller, sampled subset.
• To formalize the relationship between classifier accuracy and label set size under i.i.d. class sampling and independent scoring functions.
• To develop an unbiased estimation procedure for the expected accuracy at larger k using only data from a smaller k, enabling performance extrapolation without retraining on full sets.
• To validate the method on real-world multi-class problems such as facial recognition and Telugu OCR, demonstrating robustness across diverse classifiers.

Proposed method

• Model the expected accuracy for k classes as the (k−1)th moment of a discriminability distribution D(u), derived from the classifier's scoring function and data distribution.
• Assume classes are i.i.d. samples from a population distribution π, and classifiers are trained independently per class, enabling conditional analysis on individual class performance.
• Estimate the discriminability function D(u) non-parametrically using spline-based smoothing on empirical data from the small-k training set.
• Use the estimated D(u) to compute higher-order moments, which correspond to expected accuracies for larger k, enabling extrapolation to unobserved k2 > k1.
• Implement an unbiased estimation algorithm based on the theoretical moment formula, minimizing bias in performance prediction across varying k.
• Validate the method using resampling and cross-validation across multiple classifiers (SVM, logistic regression, Deep CNN) on both facial recognition and Telugu OCR datasets.

Experimental results

Research questions

• RQ1 Can expected classification accuracy on a large label set be predicted from performance on a smaller, sampled subset under i.i.d. class assumptions?
• RQ2 How does the expected accuracy scale with increasing label set size k, and can this scaling be characterized mathematically?
• RQ3 What is the theoretical basis for estimating the (k−1)th moment of the discriminability distribution from limited training data?
• RQ4 How does the proposed method compare to existing extrapolation techniques (e.g., KDE-BCV, KDE-UCV) in terms of accuracy and bias?
• RQ5 To what extent does the method generalize across different classifiers and real-world multi-class tasks like facial recognition and OCR?

Key findings

• The expected accuracy for k classes is mathematically equivalent to the (k−1)th moment of a discriminability distribution D(u), which can be estimated from small-k training data.
• The proposed ClassExReg method achieves the closest match to ground truth accuracy in 20-to-100 and 100-to-400 extrapolations for Deep CNN and SVM, and performs comparably or better than KDE-BCV for logistic regression.
• In the 20-to-400 extrapolation (ratio 20), ClassExReg outperforms or matches other methods across all classifiers, though performance variability is high due to small training sets.
• For the Telugu OCR dataset, ClassExReg predicts 0.9837 accuracy when extrapolating from 100 to 400 classes using a Deep CNN, close to the true value of 0.9860.
• The method is unbiased under the theoretical assumptions, and the estimation procedure efficiently computes higher-order moments without requiring retraining on larger k.
• The approach is robust across diverse classifiers and datasets, with code publicly available at https://github.com/snarles/ClassEx.