Chulhee Yun

Despite the widespread adoption of Transformer models for NLP tasks, the\nexpressive power of these models is not well-understood. In this paper, we\nestablish that Transformer models are universal approximators of continuous\npermutation equivariant sequence-to-sequence functions with compact support,\nwhich is quite surprising given the amount of shared parameters in these\nmodels. Furthermore, using positional encodings, we circumvent the restriction\nof permutation equivariance, and show tha

We study the error landscape of deep linear and nonlinear neural networks with the squared error loss. Minimizing the loss of a deep linear neural network is a nonconvex problem, and despite recent progress, our understanding of this loss surface is still incomplete. For deep linear networks, we present necessary and sufficient conditions for a critical point of the risk function to be a global minimum. Surprisingly, our conditions provide an efficiently checkable test for global optimality, whi

© 2019 Neural information processing systems foundation. All rights reserved. We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require N hidden nodes to memorize/interpolate arbitrary N data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with ?(vN) hidden nodes can perfectly memorize most datasets with N points. We also prove that width T(vN) is necessary and sufficient for memorizing N data points, proving tight bounds

The universal approximation property of width-bounded networks has been studied as a dual of classical universal approximation results on depth-bounded networks. However, the critical width enabling the universal approximation has not been exactly characterized in terms of the input dimension dx and the output dimension dy. In this work, we provide the first definitive result in this direction for networks using the ReLU activation functions: The minimum width required for the universal approxim

We investigate the loss surface of neural networks. We prove that even for\none-hidden-layer networks with "slightest" nonlinearity, the empirical risks\nhave spurious local minima in most cases. Our results thus indicate that in\ngeneral "no spurious local minima" is a property limited to deep linear\nnetworks, and insights obtained from linear networks may not be robust.\nSpecifically, for ReLU(-like) networks we constructively prove that for almost\nall practical datasets there exist infinite

Recently, Transformer networks have redefined the state of the art in many\nNLP tasks. However, these models suffer from quadratic computational cost in\nthe input sequence length $n$ to compute pairwise attention in each layer. This\nhas prompted recent research into sparse Transformers that sparsify the\nconnections in the attention layers. While empirically promising for long\nsequences, fundamental questions remain unanswered: Can sparse Transformers\napproximate any arbitrary sequence-to-se

We investigate the loss surface of deep linear and nonlinear neural networks. We show that for deep linear networks with differentiable losses, critical points after the multilinear parameterization inherit the structure of critical points of the underlying loss with linear parameterization. As corollaries we obtain minima are results that subsume most previous results, while showing how to distinguish global minima from saddle points. For nonlinear neural networks, we prove two theorems showing

We study the stochastic batched convex optimization problem, in which we use many \emph{parallel} observations to optimize a convex function given limited rounds of interaction. In each of \(M\) rounds, an algorithm may query for information at n points, and after issuing all \(n\) queries, it receives unbiased noisy function and/or (sub)gradient evaluations at the \(n\) points. After \(M\) such rounds, the algorithm must output an estimator. We provide lower and upper bounds on the performance

We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can identify the convergence direction of the network parameters as singular vectors of a tensor defined by the network

We study finite sample expressivity, i.e., memorization power of ReLU networks. Recent results require $N$ hidden nodes to memorize/interpolate arbitrary $N$ data points. In contrast, by exploiting depth, we show that 3-layer ReLU networks with $Ω(\sqrt{N})$ hidden nodes can perfectly memorize most datasets with $N$ points. We also prove that width $Θ(\sqrt{N})$ is necessary and sufficient for memorizing $N$ data points, proving tight bounds on memorization capacity. The sufficiency result can b

Attention based Transformer architecture has enabled significant advances in the field of natural language processing. In addition to new pre-training techniques, recent improvements crucially rely on working with a relatively larger embedding dimension for tokens. Unfortunately, this leads to models that are prohibitively large to be employed in the downstream tasks. In this paper we identify one of the important factors contributing to the large embedding size requirement. In particular, our a

We study the implicit bias of gradient flow (i.e., gradient descent with infinitesimal step size) on linear neural network training. We propose a tensor formulation of neural networks that includes fully-connected, diagonal, and convolutional networks as special cases, and investigate the linear version of the formulation called linear tensor networks. With this formulation, we can characterize the convergence direction of the network parameters as singular vectors of a tensor defined by the net

We study without-replacement SGD for solving finite-sum optimization problems. Specifically, depending on how the indices of the finite-sum are shuffled, we consider the RandomShuffle (shuffle at the beginning of each epoch) and SingleShuffle (shuffle only once) algorithms. First, we establish minimax optimal convergence rates of these algorithms up to poly-log factors. Notably, our analysis is general enough to cover gradient dominated nonconvex costs, and does not rely on the convexity of indi

Recent results in the literature indicate that a residual network (ResNet) composed of a single residual block outperforms linear predictors, in the sense that all local minima in its optimization landscape are at least as good as the best linear predictor. However, these results are limited to a single residual block (i.e., shallow ResNets), instead of the deep ResNets composed of multiple residual blocks. We take a step towards extending this result to deep ResNets. We start by two motivating