[Paper Review] Surfing: Iterative optimization over incrementally trained deep networks

Ganlin Song, Zhou Fan+1 more|arXiv (Cornell University)|Jul 19, 2019

Sparse and Compressive Sensing Techniques19 references4 citations

TL;DR

This paper proposes 'surfing,' an iterative optimization method that leverages incrementally trained deep networks to improve input recovery for generative models. By sequentially optimizing the input $x$ using progressively trained network parameters $\theta_t$, the method navigates the evolving loss surface from a smooth initial state to a complex final form, achieving superior convergence over direct gradient descent on the final model.

ABSTRACT

We investigate a sequential optimization procedure to minimize the empirical risk functional $f_{\hatθ}(x) = \frac{1}{2}\|G_{\hatθ}(x) - y\|^2$ for certain families of deep networks $G_θ(x)$. The approach is to optimize a sequence of objective functions that use network parameters obtained during different stages of the training process. When initialized with random parameters $θ_0$, we show that the objective $f_{θ_0}(x)$ is "nice'' and easy to optimize with gradient descent. As learning is carried out, we obtain a sequence of generative networks $x \mapsto G_{θ_t}(x)$ and associated risk functions $f_{θ_t}(x)$, where $t$ indicates a stage of stochastic gradient descent during training. Since the parameters of the network do not change by very much in each step, the surface evolves slowly and can be incrementally optimized. The algorithm is formalized and analyzed for a family of expansive networks. We call the procedure {\it surfing} since it rides along the peak of the evolving (negative) empirical risk function, starting from a smooth surface at the beginning of learning and ending with a wavy nonconvex surface after learning is complete. Experiments show how surfing can be used to find the global optimum and for compressed sensing even when direct gradient descent on the final learned network fails.

Motivation & Objective

• To address the challenge of optimizing inputs to deep generative models when direct gradient descent on the final trained network fails due to nonconvex loss landscapes.
• To exploit the gradual evolution of network parameters during stochastic training to enable stable, incremental optimization of the input $x$.
• To formalize and analyze a novel algorithm—surfing—that rides the evolving empirical risk surface from random initialization to full training.
• To demonstrate that surfing outperforms direct optimization on the final model, especially in compressed sensing and global optimum recovery.

Proposed method

• The method initializes optimization at the input $x^*_{t-1}$ that minimized the loss for the previous network $G_{\theta_{t-1}}(x)$, then performs gradient descent on $f_{\theta_t}(x) = \frac{1}{2}\|G_{\theta_t}(x) - y\|^2$ at each training stage $t$.
• It leverages the fact that parameter updates during stochastic gradient descent are small, ensuring the loss surface evolves slowly and continuously.
• Theoretical analysis shows that for random ReLU networks with sufficient width, the initial loss surface $f_{\theta_0}(x)$ is well-behaved, with descent directions outside small neighborhoods of $x_0$ and $-x_0$.
• Projected gradient descent is used in the theoretical analysis to handle piecewise quadratic surfaces arising from ReLU activations, though simple gradient descent works well in practice.
• The algorithm is formalized for expansive ReLU networks and analyzed under the assumption that the global minimizer evolves continuously with $t$.
• Experiments validate the method on VAE- and GAN-trained networks, showing improved performance in compressed sensing and global optimum recovery.

Experimental results

Research questions

• RQ1 Can iterative optimization over incrementally trained networks improve input recovery compared to direct optimization on the final model?
• RQ2 How does the geometry of the empirical risk function evolve from random initialization to full training in deep generative models?
• RQ3 Under what conditions is the global minimizer of the loss function stable and continuous during training, enabling incremental optimization?
• RQ4 Can the surfing method reliably find the global optimum even when direct gradient descent fails on the final model?
• RQ5 What theoretical guarantees can be provided for the initial loss surface $f_{\theta_0}(x)$ when the network is randomly initialized?

Key findings

• The initial loss surface $f_{\theta_0}(x)$ for a randomly initialized ReLU network is well-behaved: it has a descent direction at every point outside a small neighborhood of $x_0$ and $-x_0$, enabling reliable optimization.
• Surfing significantly outperforms direct gradient descent on the final model in recovering inputs for compressed sensing and finding global optima, especially in nonconvex landscapes.
• Theoretical analysis confirms that for expansive ReLU networks, the loss surface evolves smoothly, supporting the incremental optimization strategy.
• Projected gradient descent is effective in theory for navigating the piecewise quadratic surfaces of ReLU networks, though simple gradient descent suffices in practice.
• Empirical results show that surfing maintains convergence even when direct optimization fails, demonstrating robustness across VAE and GAN architectures.
• The method assumes continuous evolution of the global minimizer with training; discontinuities in the minimizer would break the algorithm's effectiveness.