[Paper Review] Diet deep generative audio models with structured lottery

Philippe Esling, Ninon Devis+4 more|arXiv (Cornell University)|Jul 31, 2020

Music and Audio Processing22 references4 citations

TL;DR

This paper proposes structured lottery pruning to create ultra-lightweight deep generative audio models by identifying and training sparse, structured subnetworks that maintain high accuracy. By using a global mutual information criterion for structured pruning, the method reduces model size by up to 95% with minimal accuracy loss, enabling real-time inference on CPU and embeddable deployment on platforms like Raspberry Pi and Arduino.

ABSTRACT

Deep learning models have provided extremely successful solutions in most audio application fields. However, the high accuracy of these models comes at the expense of a tremendous computation cost. This aspect is almost always overlooked in evaluating the quality of proposed models. However, models should not be evaluated without taking into account their complexity. This aspect is especially critical in audio applications, which heavily relies on specialized embedded hardware with real-time constraints. In this paper, we build on recent observations that deep models are highly overparameterized, by studying the lottery ticket hypothesis on deep generative audio models. This hypothesis states that extremely efficient small sub-networks exist in deep models and would provide higher accuracy than larger models if trained in isolation. However, lottery tickets are found by relying on unstructured masking, which means that resulting models do not provide any gain in either disk size or inference time. Instead, we develop here a method aimed at performing structured trimming. We show that this requires to rely on global selection and introduce a specific criterion based on mutual information. First, we confirm the surprising result that smaller models provide higher accuracy than their large counterparts. We further show that we can remove up to 95% of the model weights without significant degradation in accuracy. Hence, we can obtain very light models for generative audio across popular methods such as Wavenet, SING or DDSP, that are up to 100 times smaller with commensurate accuracy. We study the theoretical bounds for embedding these models on Raspberry Pi and Arduino, and show that we can obtain generative models on CPU with equivalent quality as large GPU models. Finally, we discuss the possibility of implementing deep generative audio models on embedded platforms.

Motivation & Objective

• Address the high computational and memory cost of deep generative audio models, which limits deployment on real-time and embedded systems.
• Investigate whether the lottery ticket hypothesis—initializing sparse subnetworks for high performance—applies to generative audio models.
• Overcome the inefficiency of unstructured pruning by developing structured pruning that removes entire computation units (e.g., convolutional channels), reducing model size and inference time.
• Enable deployment of high-quality generative audio models on low-resource platforms such as Arduino and Raspberry Pi by optimizing for FLOPs, disk size, and memory usage.

Proposed method

• Apply the lottery ticket hypothesis to deep generative audio models (e.g., WaveNet, SING, DDSP), identifying sparse subnetworks at initialization that achieve high accuracy when trained in isolation.
• Introduce structured pruning by removing entire convolutional channels instead of individual weights, enabling true reductions in model size and inference cost.
• Develop a global selection criterion based on mutual information between computational units and the target output to guide structured pruning and preserve performance.
• Use iterative pruning and retraining with weight rewinding to identify and train the most effective subnetworks, ensuring robustness and accuracy.
• Evaluate model efficiency using three metrics: FLOPs (inference speed), disk size (storage), and read-write memory (RAM access), to assess real-time and embeddable feasibility.
• Benchmark pruned models on embedded platforms (Arduino, Raspberry Pi) using inferred FLOPS and hardware constraints to determine embeddability thresholds.

Experimental results

Research questions

• RQ1 Can the lottery ticket hypothesis be successfully applied to deep generative audio models, and does it yield smaller, more accurate models?
• RQ2 Does structured pruning—removing entire channels instead of individual weights—enable meaningful reductions in model size and inference time compared to unstructured pruning?
• RQ3 Can a global, information-theoretic criterion (based on mutual information) effectively guide structured pruning to maintain high performance during aggressive compression?
• RQ4 To what extent can pruned generative audio models be embedded on low-power platforms like Arduino and Raspberry Pi, and what trade-offs exist between model size, accuracy, and hardware constraints?

Key findings

• Smaller models, achieved through structured lottery pruning, achieve higher accuracy than their larger counterparts, confirming the surprising benefit of model compression.
• Up to 95% of model weights can be removed without significant accuracy degradation, demonstrating extreme sparsity is viable in generative audio models.
• The proposed mutual information-based criterion enables effective global structured pruning, outperforming local pruning in maintaining accuracy during aggressive compression.
• Lightweight models are up to 100 times smaller than original models while maintaining GPU-quality audio synthesis, enabling real-time CPU inference.
• On Raspberry Pi 1B, model size and memory usage are within constraints, though FLOPs remain borderline; more aggressive pruning can achieve embeddability with only marginal accuracy loss.
• On Arduino platforms, even the smallest models exceed FLOPS and memory limits, with embeddable models requiring error rates 2.5× higher than original, indicating current hardware constraints remain challenging.