[Paper Review] ReduNet: A White-box Deep Network from the Principle of Maximizing Rate Reduction

Kwan Ho Ryan Chan, Yaodong Yu+4 more|arXiv (Cornell University)|May 21, 2021

Domain Adaptation and Few-Shot Learning146 references54 citations

TL;DR

ReduNet derives a white-box deep network by maximizing the coding rate reduction objective, yielding a layer-by-layer constructed network with interpretable operators and a convolutional form under shift/invariance, demonstrated through preliminary experiments.

ABSTRACT

This work attempts to provide a plausible theoretical framework that aims to interpret modern deep (convolutional) networks from the principles of data compression and discriminative representation. We argue that for high-dimensional multi-class data, the optimal linear discriminative representation maximizes the coding rate difference between the whole dataset and the average of all the subsets. We show that the basic iterative gradient ascent scheme for optimizing the rate reduction objective naturally leads to a multi-layer deep network, named ReduNet, which shares common characteristics of modern deep networks. The deep layered architectures, linear and nonlinear operators, and even parameters of the network are all explicitly constructed layer-by-layer via forward propagation, although they are amenable to fine-tuning via back propagation. All components of so-obtained "white-box" network have precise optimization, statistical, and geometric interpretation. Moreover, all linear operators of the so-derived network naturally become multi-channel convolutions when we enforce classification to be rigorously shift-invariant. The derivation in the invariant setting suggests a trade-off between sparsity and invariance, and also indicates that such a deep convolution network is significantly more efficient to construct and learn in the spectral domain. Our preliminary simulations and experiments clearly verify the effectiveness of both the rate reduction objective and the associated ReduNet. All code and data are available at \url{https://github.com/Ma-Lab-Berkeley}.

Motivation & Objective

• Motivate a principled objective for discriminative representation learning based on data compression and rate reduction.
• Show that the rate reduction objective leads to a constructive, layerwise network design (ReduNet) with explicit optimization and geometric interpretation.
• Demonstrate that network components become multi-channel convolutions under shift-invariance and analyze sparsity-invariance trade-offs.
• Provide a forward-constructed, white-box alternative to end-to-end backpropagation that is amenable to analysis and verification.

Proposed method

• Define the maximal coding rate reduction (MCR2) objective as the difference between the coding rate of the whole dataset and the sum of coding rates of class-specific subsets.
• Derive gradient ascent dynamics that naturally unfold into a deep network architecture (ReduNet) with layer-by-layer construction.
• Show that linear operators in ReduNet become multi-channel convolutions when enforcing shift/invariance.
• Establish a trade-off between sparsity and invariance in the invariant rate reduction setting and discuss spectral-domain computation.
• Present forward-propagation construction of network layers with explicit optimization interpretation, re-usable with backpropagation for fine-tuning.

Experimental results

Research questions

• RQ1 What principled objective best captures the structure of discriminative representations for high-dimensional multi-class data?
• RQ2 Can a deep network architecture be derived constructively from an optimization objective rather than designed heuristically?
• RQ3 How does rate reduction guide the design of linear and nonlinear operators, and what is the role of multi-channel convolutions?
• RQ4 What is the trade-off between invariance and sparsity in rate-reduced representations, and how can this be efficiently computed?
• RQ5 Do the resulting ReduNet components provide interpretable and verifiable optimization guarantees?

Key findings

• The rate reduction objective promotes within-class compactness and between-class discrimination by maximizing the coding rate difference between the whole dataset and class subsets.
• The gradient ascent scheme for rate reduction yields a multi-layer network (ReduNet) whose components have precise optimization, statistical, and geometric interpretations.
• Under shift/invariance constraints, ReduNet’s linear operators become multi-channel convolutions, enabling efficient spectral-domain computation.
• There is a fundamental sparsity-invariance trade-off in the invariant rate reduction setting, which informs network design for translation/rotation invariance.
• Preliminary experiments verify the effectiveness of the rate reduction objective and the ReduNet architecture, with code and data publicly available.