[Paper Review] Online Learning with Gated Linear Networks

Joel Veness, Tor Lattimore+4 more|arXiv (Cornell University)|Dec 5, 2017

Machine Learning and Algorithms29 references17 citations

TL;DR

This paper introduces Gated Linear Networks (GLNs), a family of probabilistic architectures for online density estimation under logarithmic loss, using data-conditioned gating instead of non-linear activation functions. The method achieves state-of-the-art performance—79.0 nats per image on MNIST—matching deep learning models in a single online pass, with theoretical guarantees of universal approximation under no-regret learning.

ABSTRACT

This paper describes a family of probabilistic architectures designed for online learning under the logarithmic loss. Rather than relying on non-linear transfer functions, our method gains representational power by the use of data conditioning. We state under general conditions a learnable capacity theorem that shows this approach can in principle learn any bounded Borel-measurable function on a compact subset of euclidean space; the result is stronger than many universality results for connectionist architectures because we provide both the model and the learning procedure for which convergence is guaranteed.

Motivation & Objective

• To develop a theoretically grounded, online learning framework for high-dimensional density modeling that avoids the limitations of offline deep learning methods.
• To explain the empirical success of the PAQ family of data compression algorithms through a unified architectural and learning-theoretic framework.
• To demonstrate that online learning with data-conditioned gating can achieve performance comparable to state-of-the-art batch-trained deep generative models.
• To provide a learnable capacity theorem proving that GLNs can approximate any bounded Borel-measurable function on compact subsets of Euclidean space.
• To enable efficient, scalable implementation on modern hardware by reformulating context-mixing architectures as differentiable GLNs.

Proposed method

• Uses a gating mechanism where each neuron's activation is conditioned on input data via context functions (e.g., skip-grams, max-pool, distance functions), replacing traditional non-linearities.
• Employs a local, no-regret online learning rule for weight updates, ensuring convergence to optimal predictions under logarithmic loss.
• Constructs a 4-layer GLN with 35-60-35-70 architecture, using 200 randomly distributed context functions per layer to enable expressive, data-adaptive modeling.
• Applies a learning rate schedule of $\min\{25/t, 0.005\}$ to balance early learning and convergence stability during online training.
• Uses the Zero-Redundancy estimator (Willems et al., 1997) for online estimation of per-context skip-gram probabilities in image modeling.
• Introduces image-specific context functions such as max-pool and distance-based contexts to improve modeling of local image structure.

Experimental results

Research questions

• RQ1 Can a neural network architecture based on data-conditioned gating achieve universal approximation for bounded Borel-measurable functions under online learning?
• RQ2 Why do context-mixing algorithms like PAQ and cmix achieve such strong compression and density estimation performance in practice?
• RQ3 Can online learning with no-regret optimization match the performance of batch-trained deep generative models in high-dimensional density estimation?
• RQ4 How do specialized context functions (e.g., max-pool, distance) improve modeling efficiency and accuracy in image density estimation?
• RQ5 To what extent can the GLN framework unify and generalize existing compression and online learning techniques with theoretical guarantees?

Key findings

• The GLN framework achieves an average loss of 79.0 nats per image on the MNIST dataset using a single online pass through the data, matching the state of the art among exact-probability batch models.
• The method demonstrates that data-conditioned gating enables universal approximation of bounded Borel-measurable functions on compact subsets of Euclidean space, with convergence guaranteed under no-regret learning.
• The PAQ family of compressors, particularly cmix, is shown to be a special case of GLNs, explaining their empirical success through a unified theoretical lens.
• Image-specific context functions—max-pool and distance-based contexts—significantly improve performance over standard skip-gram models alone.
• Theoretical analysis confirms that the local, no-regret learning rule used in GLNs is justified under piecewise-stationary sources, with convex instantaneous loss ensuring convergence.
• The GLN framework enables efficient GPU-accelerated training via vectorized matrix operations, suggesting scalability with hardware advances.