[Paper Review] Efficient Loss-Based Decoding on Graphs For Extreme Classification

Itay Evron, Edward Moroshko+1 more|arXiv (Cornell University)|Mar 8, 2018

Machine Learning and Algorithms31 references4 citations

TL;DR

This paper proposes Wide-LTLS, a novel graph-based decoding framework for extreme multiclass classification that achieves logarithmic time and space complexity while enabling loss-based decoding for improved accuracy. By generalizing the Log-Time Log-Space (LTLS) framework using trellis graphs, it offers a tunable tradeoff between accuracy, model size, and inference speed, outperforming prior methods in both efficiency and predictive performance on large-scale label sets.

ABSTRACT

In extreme classification problems, learning algorithms are required to map instances to labels from an extremely large label set. We build on a recent extreme classification framework with logarithmic time and space, and on a general approach for error correcting output coding (ECOC) with loss-based decoding, and introduce a flexible and efficient approach accompanied by theoretical bounds. Our framework employs output codes induced by graphs, for which we show how to perform efficient loss-based decoding to potentially improve accuracy. In addition, our framework offers a tradeoff between accuracy, model size and prediction time. We show how to find the sweet spot of this tradeoff using only the training data. Our experimental study demonstrates the validity of our assumptions and claims, and shows that our method is competitive with state-of-the-art algorithms.

Motivation & Objective

• To address the scalability limitations of traditional multiclass classification methods when dealing with extremely large label sets (K >> 10^4).
• To overcome the accuracy limitations of the existing Log-Time Log-Space (LTLS) framework by incorporating loss-based decoding via error-correcting output codes (ECOC).
• To enable a flexible tradeoff between prediction accuracy, model size, and inference time in extreme classification settings.
• To develop a method that maintains logarithmic time and space complexity while supporting efficient, accurate decoding using general graph structures.

Proposed method

• Represents classes as paths in a trellis graph structure, generalizing the binary tree used in LTLS to allow for more expressive and flexible code representations.
• Employs error-correcting output codes (ECOC) with loss-based decoding, leveraging theoretical bounds from Allwein et al. (2001) to improve robustness to binary classifier errors.
• Uses a loss-based decoding strategy that minimizes a general margin-based loss function over the predictions of binary classifiers, rather than relying on Hamming distance or hard decoding.
• Constructs coding matrices from trellis graphs to induce non-uniform, potentially imbalanced subproblems, which improves learnability and enables sparsity through post-pruning.
• Designs a decoding algorithm that operates in O(log K) time and space by exploiting the hierarchical structure of the graph, avoiding explicit storage of the full coding matrix during inference.
• Introduces a method to tune the tradeoff between accuracy, model size, and inference time using only training data, by selecting optimal graph topologies and pruning strategies.

Experimental results

Research questions

• RQ1 Can loss-based decoding be efficiently integrated into a log-time, log-space framework for extreme classification without sacrificing computational efficiency?
• RQ2 How does using trellis graphs instead of binary trees improve the accuracy and flexibility of ECOC-based extreme classification?
• RQ3 To what extent can the tradeoff between model size, inference time, and prediction accuracy be controlled and optimized using graph-based coding structures?
• RQ4 Does the proposed method maintain logarithmic time and space complexity while achieving better accuracy than existing log-time methods like LTLS?

Key findings

• The proposed Wide-LTLS framework achieves O(log K) time and space complexity for both training and inference, matching the efficiency of LTLS while enabling loss-based decoding.
• By using trellis graphs, the method improves accuracy over standard LTLS by allowing more expressive code representations and better handling of classifier errors through loss-based decoding.
• The framework supports a tunable tradeoff between accuracy, model size, and inference time, with optimal configurations found using only training data.
• Empirical results show that Wide-LTLS is competitive with state-of-the-art extreme classification algorithms in terms of accuracy, while maintaining logarithmic complexity.
• The method enables sparsity through post-pruning of learned weights, further reducing model size and inference cost without significant accuracy loss.
• Theoretical analysis confirms that the method generalizes LTLS as a special case and provides a broader framework for graph-based ECOC with improved error-correcting capabilities.