[Paper Review] Distributed optimization of deeply nested systems

Miguel Á. Carreira-Perpiñán, Weiran Wang|arXiv (Cornell University)|Dec 24, 2012

Sparse and Compressive Sensing TechniquesEngineering28 references104 citations

TL;DR

This paper introduces the Method of Auxiliary Coordinates (MAC), a novel optimization framework for deeply nested systems such as deep neural networks. By reformulating the nonconvex, nested optimization problem as a constrained problem in an augmented space using auxiliary variables, MAC enables provably convergent, massively parallelizable optimization that avoids vanishing gradients and reuses existing single-layer algorithms, achieving competitive performance with state-of-the-art methods even in serial settings.

ABSTRACT

In science and engineering, intelligent processing of complex signals such as images, sound or language is often performed by a parameterized hierarchy of nonlinear processing layers, sometimes biologically inspired. Hierarchical systems (or, more generally, nested systems) offer a way to generate complex mappings using simple stages. Each layer performs a different operation and achieves an ever more sophisticated representation of the input, as, for example, in an deep artificial neural network, an object recognition cascade in computer vision or a speech front-end processing. Joint estimation of the parameters of all the layers and selection of an optimal architecture is widely considered to be a difficult numerical nonconvex optimization problem, difficult to parallelize for execution in a distributed computation environment, and requiring significant human expert effort, which leads to suboptimal systems in practice. We describe a general mathematical strategy to learn the parameters and, to some extent, the architecture of nested systems, called the method of auxiliary coordinates (MAC). This replaces the original problem involving a deeply nested function with a constrained problem involving a different function in an augmented space without nesting. The constrained problem may be solved with penalty-based methods using alternating optimization over the parameters and the auxiliary coordinates. MAC has provable convergence, is easy to implement reusing existing algorithms for single layers, can be parallelized trivially and massively, applies even when parameter derivatives are not available or not desirable, and is competitive with state-of-the-art nonlinear optimizers even in the serial computation setting, often providing reasonable models within a few iterations.

Motivation & Objective

• To address the challenge of joint optimization in deeply nested, nonconvex systems such as deep neural networks.
• To overcome limitations of backpropagation, including vanishing gradients, poor parallelization, and reliance on differentiable parameters.
• To develop a general-purpose optimization strategy that reuses existing single-layer algorithms and supports distributed computation.
• To enable automatic architecture selection and parameter learning in hierarchical systems with provable convergence.

Proposed method

• Introduces auxiliary coordinates (Z) to represent hidden unit activations, transforming the deeply nested function into a constrained optimization problem in an augmented space.
• Replaces the original nested objective E1(W) with a constrained problem E(W, Z) subject to equality constraints Znkh = fkh(zn,k−1; Wk) for each data point and layer.
• Solves the constrained problem using a quadratic penalty method, minimizing the augmented Lagrangian EQ(W, Z; µ) = E(W, Z) + (µ/2)∑‖Z − F(W, X)‖².
• Alternates between optimizing over parameters W and auxiliary coordinates Z, enabling trivial and massive parallelization across data points and layers.
• Proves convergence to KKT points of the constrained problem, which correspond to stationary points of the original nested problem under mild regularity conditions.
• Allows use of non-gradient-based optimizers and handles non-differentiable mappings by relying on the auxiliary variable reformulation.

Experimental results

Research questions

• RQ1 Can a general-purpose optimization method be developed for deeply nested systems that avoids the vanishing gradient problem inherent in backpropagation?
• RQ2 How can joint parameter and architecture learning be achieved in hierarchical systems with provable convergence and scalability?
• RQ3 Can existing single-layer optimization algorithms be reused in a distributed, massively parallel setting for end-to-end training of nested systems?
• RQ4 What conditions ensure that the stationary points of the auxiliary coordinate reformulation correspond to meaningful solutions of the original nested problem?

Key findings

• MAC achieves provable convergence to KKT points of the constrained problem, which correspond to stationary points of the original nested optimization problem.
• The method is trivially parallelizable across data points and layers, enabling efficient distributed computation on cloud architectures.
• MAC converges rapidly, often yielding reasonable models within a few iterations, even when using simple local optimizers.
• The approach is robust to non-differentiable mappings and can be used with non-gradient-based optimization algorithms.
• Theoretical analysis confirms that minimizers, maximizers, and saddle points of the original nested problem are in one-to-one correspondence with those of the MAC-constrained problem.
• Empirical results demonstrate competitive performance against state-of-the-art nonlinear optimizers, even in serial computation settings.