[Paper Review] Online Learning of Combinatorial Objects via Extended Formulation

Holakou Rahmanian, David P. Helmbold+1 more|arXiv (Cornell University)|Sep 17, 2016

Machine Learning and Algorithms24 references3 citations

TL;DR

This paper introduces XF-Hedge, a novel online learning algorithm that leverages extended formulations to efficiently learn combinatorial objects like permutations and Huffman trees. By encoding complex polytopes in higher-dimensional spaces with polynomially many facets, XF-Hedge enables efficient Bregman projections and achieves regret bounds within a factor of $\mathcal{O}(\sqrt{\log n})$ of state-of-the-art algorithms for permutations and matches or improves upon existing bounds for Huffman trees across different loss regimes.

ABSTRACT

The standard techniques for online learning of combinatorial objects perform multiplicative updates followed by projections into the convex hull of all the objects. However, this methodology can be expensive if the convex hull contains many facets. For example, the convex hull of $n$-symbol Huffman trees is known to have exponentially many facets (Maurras et al., 2010). We get around this difficulty by exploiting extended formulations (Kaibel, 2011), which encode the polytope of combinatorial objects in a higher dimensional "extended" space with only polynomially many facets. We develop a general framework for converting extended formulations into efficient online algorithms with good relative loss bounds. We present applications of our framework to online learning of Huffman trees and permutations. The regret bounds of the resulting algorithms are within a factor of $O(\sqrt{\log(n)})$ of the state-of-the-art specialized algorithms for permutations, and depending on the loss regimes, improve on or match the state-of-the-art for Huffman trees. Our method is general and can be applied to other combinatorial objects.

Motivation & Objective

• To address the computational inefficiency of standard online learning methods that project onto high-dimensional convex hulls of combinatorial objects, especially when these hulls have exponentially many facets.
• To develop a general framework that transforms extended formulations of combinatorial polytopes into efficient online learning algorithms with provable regret guarantees.
• To enable the use of Component Hedge-style algorithms on complex combinatorial classes by exploiting redundant, higher-dimensional representations with polynomially many facets.
• To achieve regret bounds competitive with specialized algorithms for permutations and Huffman trees, while maintaining computational efficiency.
• To extend the applicability of online learning to combinatorial objects whose convex hulls are intractable to project onto directly.

Proposed method

• The method uses extended formulations to represent the polytope of combinatorial objects in a higher-dimensional space where the number of facets is polynomial in the problem size.
• It applies iterative Bregman projections in the extended space to maintain a distribution over combinatorial objects, avoiding explicit enumeration of all objects.
• The algorithm performs multiplicative updates on the parameters of the implicit representation, followed by projections via Bregman divergences to ensure feasibility in the extended formulation.
• The framework supports loss functions defined on the original combinatorial objects, even when the extended formulation introduces auxiliary variables and slack variables.
• The regret analysis accounts for approximation errors introduced by early termination of the projection procedure, bounding the additional loss incurred.
• The approach generalizes Component Hedge and inherits tools from expert learning, such as adaptability to bandit feedback and shifting comparator bounds.

Experimental results

Research questions

• RQ1 Can extended formulations be used to design efficient online learning algorithms for combinatorial objects whose convex hulls have exponentially many facets?
• RQ2 How can the regret of online learning be bounded when projections are performed in an extended, higher-dimensional space instead of the original combinatorial space?
• RQ3 What is the performance gap between XF-Hedge and state-of-the-art specialized algorithms for permutations and Huffman trees under different loss regimes?
• RQ4 In what ways can the redundant variables in extended formulations be leveraged to express more complex or structured loss functions?
• RQ5 Which classes of combinatorial objects admit both natural online losses and compact extended formulations suitable for online learning?

Key findings

• For permutations, XF-Hedge achieves a regret bound within a factor of $\mathcal{O}(\sqrt{\log n})$ of the state-of-the-art OnlineRank algorithm.
• For Huffman trees under unit cube loss vectors, XF-Hedge matches or improves upon the regret bounds of FPL and the generic Hedge algorithm, with a bound of $\mathcal{O}(n(\log n)^{1/2}\sqrt{L^*} + n^2\log n)$.
• In the unit simplex loss regime for Huffman trees, where per-trial losses are $\mathcal{O}(n)$, XF-Hedge's regret matches that of FPL and the generic Hedge algorithm, all achieving $\mathcal{O}(n(\log n)^{1/2}\sqrt{L^*} + n^2\log n)$.
• The method reduces the regret loss range factor present in generic Hedge algorithms by avoiding explicit weight tracking over exponentially many objects.
• The analysis shows that iterative Bregman projections in the extended space incur only a bounded additional loss when terminated early, enabling efficient implementation.
• The framework is general and can be applied to other combinatorial objects with suitable extended formulations, such as paths, $k$-sets, and submodular structures.