[Paper Review] One ticket to win them all: generalizing lottery ticket initializations across datasets and optimizers

Ari S. Morcos, Haonan Yu+2 more|arXiv (Cornell University)|Jun 6, 2019

Generative Adversarial Networks and Image SynthesisComputer Science33 references107 citations

TL;DR

The paper shows that winning ticket initializations found on one dataset or optimizer often transfer to other datasets and optimizers in natural image tasks, especially when sourced from larger datasets.

ABSTRACT

The success of lottery ticket initializations (Frankle and Carbin, 2019) suggests that small, sparsified networks can be trained so long as the network is initialized appropriately. Unfortunately, finding these "winning ticket" initializations is computationally expensive. One potential solution is to reuse the same winning tickets across a variety of datasets and optimizers. However, the generality of winning ticket initializations remains unclear. Here, we attempt to answer this question by generating winning tickets for one training configuration (optimizer and dataset) and evaluating their performance on another configuration. Perhaps surprisingly, we found that, within the natural images domain, winning ticket initializations generalized across a variety of datasets, including Fashion MNIST, SVHN, CIFAR-10/100, ImageNet, and Places365, often achieving performance close to that of winning tickets generated on the same dataset. Moreover, winning tickets generated using larger datasets consistently transferred better than those generated using smaller datasets. We also found that winning ticket initializations generalize across optimizers with high performance. These results suggest that winning ticket initializations generated by sufficiently large datasets contain inductive biases generic to neural networks more broadly which improve training across many settings and provide hope for the development of better initialization methods.

Motivation & Objective

• Investigate whether winning ticket initializations generalize across datasets within the natural images domain.
• Assess cross-optimizer transferability of winning tickets.
• examine how dataset size and class count affect generalization of winning tickets across tasks.

Proposed method

• Use iterative magnitude pruning with 20% pruning rate per iteration and late resetting to initial values.
• Compare global pruning vs. local pruning and find global pruning superior.
• Transfer winning tickets generated on a source dataset/optimizer to target datasets/optimizers and evaluate performance.
• Exclude the final classification layer when transferring due to differing output classes, reinitializing it randomly.

Experimental results

Research questions

• RQ1 Do winning tickets transfer across datasets within natural image classification tasks?
• RQ2 Do winning tickets transfer across optimizers (SGD with momentum vs. Adam)?
• RQ3 Does dataset size and class count of the source influence transfer effectiveness?

Key findings

• Winning tickets transferred across datasets often perform close to dataset-specific winning tickets across multiple target datasets.
• Winning tickets generated by larger datasets generalize better than those from smaller datasets.
• Transferred tickets generalize across optimizers, indicating optimizer-independent inductive biases.
• Global magnitude pruning outperforms layerwise pruning and tends to prune deeper layers more aggressively while preserving early layers.
• Transfered tickets can mitigate overfitting in overparameterized networks, especially on very small datasets.