[Paper Review] Candidates vs. Noises Estimation for Large Multi-Class Classification Problem

Lei Han, Yiheng Huang+1 more|arXiv (Cornell University)|Nov 2, 2017

Natural Language Processing Techniques4 citations

TL;DR

This paper proposes Candidates vs. Noises Estimation (CANE), a method for large multi-class classification that selects a small set of candidate classes and samples the rest as noise, achieving consistent estimation with low statistical variance and O(1) computational complexity per example. CANE outperforms NCE, its variants, and state-of-the-art tree classifiers in both accuracy and speed on image classification and neural language modeling tasks.

ABSTRACT

This paper proposes a method for multi-class classification problems, where the number of classes K is large. The method, referred to as Candidates vs. Noises Estimation (CANE), selects a small subset of candidate classes and samples the remaining classes. We show that CANE is always consistent and computationally efficient. Moreover, the resulting estimator has low statistical variance approaching that of the maximum likelihood estimator, when the observed label belongs to the selected candidates with high probability. In practice, we use a tree structure with leaves as classes to promote fast beam search for candidate selection. We further apply the CANE method to estimate word probabilities in learning large neural language models. Extensive experimental results show that CANE achieves better prediction accuracy over the Noise-Contrastive Estimation (NCE), its variants and a number of the state-of-the-art tree classifiers, while it gains significant speedup compared to standard O(K) methods.

Motivation & Objective

• Address the computational infeasibility of standard softmax logistic regression in large multi-class problems with K classes (e.g., tens to hundreds of thousands).
• Improve statistical efficiency over existing sampling-based methods like Noise-Contrastive Estimation (NCE), which treat all non-target classes uniformly as noise.
• Develop a consistent, low-variance estimator that maintains high accuracy when the true class is likely to be in the selected candidate set.
• Enable fast training and prediction by decoupling computation from the full class size K through candidate selection and noise sampling.
• Integrate CANE into neural language modeling to estimate word probabilities efficiently while preserving model performance.

Proposed method

• Propose CANE as a general framework that selects a small, adaptive subset of candidate classes for each input and treats the remaining classes as noise.
• Use a tree-structured representation with classes as leaves to enable fast beam search for candidate selection, reducing search cost.
• Formulate the estimation objective as a contrastive learning problem between candidate classes and sampled noise classes, avoiding full normalization over K classes.
• Ensure consistency and low statistical variance by leveraging high probability coverage of the true class within the candidate set, approaching MLE performance.
• Implement a stochastic gradient descent (SGD) procedure that scales independently of K, enabling efficient training.
• Apply CANE to neural language modeling using an LSTM with learned embeddings, where candidate words are selected via tree-based beam search and noise words are sampled from a power-raised unigram distribution.

Experimental results

Research questions

• RQ1 Can a candidate-based sampling strategy achieve consistent estimation and low statistical variance in large multi-class classification?
• RQ2 How does the performance of CANE compare to NCE and its variants in terms of prediction accuracy and training speed?
• RQ3 To what extent does candidate set coverage of the true class affect estimator variance and convergence?
• RQ4 Can CANE be effectively combined with tree-structured models to accelerate candidate selection without sacrificing accuracy?
• RQ5 Does CANE outperform state-of-the-art tree classifiers and sampling methods in neural language modeling tasks?

Key findings

• CANE achieves faster convergence and lower test perplexity than NCE and BlackOut on both Penn TreeBank and Gutenberg datasets, with lower perplexity approaching that of full softmax.
• On the Penn TreeBank dataset, CANE with 80 candidates achieved a test perplexity of 102.5, outperforming NCE (105.8) and BlackOut (106.1) under the same settings.
• On the Gutenberg dataset, CANE reduced training time to ~5 hours (vs. 6–8 hours for NCE and 35 hours for full softmax), while maintaining competitive perplexity.
• The probability that the true label is included in the candidate set exceeds 97% when selecting 9 candidates on ALOI and ImgNet-10K, indicating high coverage.
• CANE achieved 21.9% top-1 accuracy on ImageNet-10K, surpassing NCE and BlackOut, and outperforming some O(K) methods, though underperforming the state-of-the-art 28.4% result.
• Prediction time for CANE is significantly faster than NCE and BlackOut due to reduced candidate set size, despite slightly slower per-example processing from beam search.