[Paper Review] A Survey of Lottery Ticket Hypothesis

Bohan Liu, Zijie Zhang+7 more|arXiv (Cornell University)|Mar 7, 2024

Gambling Behavior and Treatments5 citations

TL;DR

This paper surveys the Lottery Ticket Hypothesis (LTH), summarizing theory, special models, algorithms, experiments, efficiency, relations to other topics, and applications, and discusses open issues and future directions.

ABSTRACT

The Lottery Ticket Hypothesis (LTH) states that a dense neural network model contains a highly sparse subnetwork (i.e., winning tickets) that can achieve even better performance than the original model when trained in isolation. While LTH has been proved both empirically and theoretically in many works, there still are some open issues, such as efficiency and scalability, to be addressed. Also, the lack of open-source frameworks and consensual experimental setting poses a challenge to future research on LTH. We, for the first time, examine previous research and studies on LTH from different perspectives. We also discuss issues in existing works and list potential directions for further exploration. This survey aims to provide an in-depth look at the state of LTH and develop a duly maintained platform to conduct experiments and compare with the most updated baselines.

Motivation & Objective

• Explain the Lottery Ticket Hypothesis and its theoretical foundations.
• Categorize and synthesize LTH research across diverse model types and tasks.
• Identify practical challenges, efficiency concerns, and scalability issues in LTH.
• Review algorithms and initialization/pruning strategies for discovering winning tickets.
• Propose directions and benchmarks to advance reproducibility and comparison in LTH research.

Proposed method

• Provide a formal notation framework for LTH including pretraining, pruning, and reinitialization.
• Summarize and compare iterative magnitude-based pruning and one-shot pruning approaches.
• Survey theoretical results proving existence of winning tickets and strong lottery tickets.
• Catalog LTH variants across special models such as GNNs, Transformers, and generative models.
• Review experimental insights on pruning extent, layerwise vs global pruning, and key factors like zeros, signs, and supermasks.
• Present a taxonomy of LTH literature and summarize public codebases and benchmarks.

Experimental results

Research questions

• RQ1 What is the current evidence for the existence of winning tickets across architectures and tasks?
• RQ2 How do initialization, pruning strategy, and retraining influence the success of lottery tickets in practice?
• RQ3 How can LTH be extended to special models (GNNs, Transformers, GANs/VAEs) and new domains?
• RQ4 What are the main efficiency and scalability bottlenecks in discovering and using lottery tickets, and how can they be mitigated?

Key findings

• Winning tickets can exist in dense networks and can match or outperform the original model when retrained from initialization.
• Strong lottery tickets may exist where subnetworks perform well without training, under certain formulations.
• Special models (GNNs, Transformers, Generative Models) require tailored LTH formulations and show varying transferability and efficiency gains.
• Various initialization and pruning strategies, including weight rewinding and learning rate rewinding, significantly affect ticket quality.
• There is a recognized need for standardized experimental settings, reproducibility benchmarks, and open-source platforms for fair comparisons.
• Pruning efficiency and hardware-aware structured pruning remain active areas to bridge theory with real-world deployment.