[Paper Review] Fast Single-Class Classification and the Principle of Logit Separation

Gil Keren, Sivan Sabato+1 more|arXiv (Cornell University)|May 29, 2017

Domain Adaptation and Few-Shot Learning30 references3 citations

TL;DR

This paper introduces the Single Logit Classification (SLC) task, where a neural network is trained to classify examples using only the logit of a single target class, enabling fast inference. It proposes the Principle of Logit Separation (PoLS) as a design guideline for loss functions, showing that losses aligned with PoLS improve SLC accuracy by at least 20% relative to cross-entropy, while achieving up to 10x speedup with 400,000 classes.

ABSTRACT

We consider neural network training, in applications in which there are many possible classes, but at test-time, the task is a binary classification task of determining whether the given example belongs to a specific class, where the class of interest can be different each time the classifier is applied. For instance, this is the case for real-time image search. We define the Single Logit Classification (SLC) task: training the network so that at test-time, it would be possible to accurately identify whether the example belongs to a given class in a computationally efficient manner, based only on the output logit for this class. We propose a natural principle, the Principle of Logit Separation, as a guideline for choosing and designing losses suitable for the SLC. We show that the cross-entropy loss function is not aligned with the Principle of Logit Separation. In contrast, there are known loss functions, as well as novel batch loss functions that we propose, which are aligned with this principle. In total, we study seven loss functions. Our experiments show that indeed in almost all cases, losses that are aligned with the Principle of Logit Separation obtain at least 20% relative accuracy improvement in the SLC task compared to losses that are not aligned with it, and sometimes considerably more. Furthermore, we show that fast SLC does not cause any drop in binary classification accuracy, compared to standard classification in which all logits are computed, and yields a speedup which grows with the number of classes. For instance, we demonstrate a 10x speedup when the number of classes is 400,000. Tensorflow code for optimizing the new batch losses is publicly available at https://github.com/cruvadom/Logit Separation.

Motivation & Objective

• To address the high computational cost of inference in neural networks with very large numbers of classes, especially when only a single class is queried at test time.
• To enable fast, single-logit inference by training models such that the logit of the target class alone determines classification accuracy.
• To identify a fundamental principle—Logit Separation—that ensures the logit of the correct class is meaningfully separated from those of incorrect classes.
• To design and evaluate loss functions aligned with the Principle of Logit Separation, improving SLC performance over standard cross-entropy.

Proposed method

• Proposes the Single Logit Classification (SLC) task, where only the logit of the queried class is computed at test time.
• Introduces the Principle of Logit Separation (PoLS), requiring that correct-class logits be significantly higher than incorrect-class logits in the logit space.
• Analyzes seven loss functions, including standard cross-entropy and novel batch losses, for alignment with PoLS.
• Defines batch losses (e.g., batch cross-entropy, batch max-margin) that enforce PoLS by ensuring correct logits dominate incorrect ones within each mini-batch.
• Uses theoretical analysis to prove that PoLS-aligned losses ensure correct-class logits are separated from false-class logits in the limit of low loss.
• Employs a theoretical framework based on KL divergence and margin constraints to formalize the separation condition.

Experimental results

Research questions

• RQ1 Can a neural network be trained such that the logit of a single class alone determines whether the example belongs to that class?
• RQ2 Why does standard cross-entropy loss fail in the SLC setting despite being widely used?
• RQ3 What principle can guide the design of loss functions that make single-class logits meaningful for classification?
• RQ4 Can batch-based loss functions be constructed to enforce logit separation across all classes?
• RQ5 What is the empirical impact of PoLS-aligned losses on SLC accuracy and inference speed?

Key findings

• Losses aligned with the Principle of Logit Separation (PoLS) achieve at least a 20% relative improvement in SLC accuracy compared to cross-entropy.
• The cross-entropy loss is not aligned with PoLS, as it fails to ensure sufficient separation between correct and incorrect class logits.
• Novel batch losses—batch cross-entropy and batch max-margin—were proven to be PoLS-aligned and outperform standard cross-entropy in SLC.
• With 400,000 classes, the proposed SLC method achieves a 10x speedup in inference time compared to full softmax inference.
• The SLC approach maintains binary classification accuracy comparable to standard full-classification training, with no drop in performance.
• Theoretical analysis confirms that PoLS-aligned losses enforce a strict separation between correct-class and incorrect-class logits in the logit space.