[论文解读] Diet deep generative audio models with structured lottery

Philippe Esling, Ninon Devis+4 more|arXiv (Cornell University)|Jul 31, 2020

Music and Audio Processing参考文献 22被引用 4

一句话总结

本文提出结构化彩票剪枝方法，通过识别并训练稀疏、结构化的子网络，在保持高准确率的同时，创建超轻量级深度生成音频模型。通过使用全局互信息准则进行结构化剪枝，该方法将模型大小减少高达95%，且准确率损失极小，从而实现在CPU上的实时推理，并可部署于树莓派和Arduino等平台。

ABSTRACT

Deep learning models have provided extremely successful solutions in most audio application fields. However, the high accuracy of these models comes at the expense of a tremendous computation cost. This aspect is almost always overlooked in evaluating the quality of proposed models. However, models should not be evaluated without taking into account their complexity. This aspect is especially critical in audio applications, which heavily relies on specialized embedded hardware with real-time constraints. In this paper, we build on recent observations that deep models are highly overparameterized, by studying the lottery ticket hypothesis on deep generative audio models. This hypothesis states that extremely efficient small sub-networks exist in deep models and would provide higher accuracy than larger models if trained in isolation. However, lottery tickets are found by relying on unstructured masking, which means that resulting models do not provide any gain in either disk size or inference time. Instead, we develop here a method aimed at performing structured trimming. We show that this requires to rely on global selection and introduce a specific criterion based on mutual information. First, we confirm the surprising result that smaller models provide higher accuracy than their large counterparts. We further show that we can remove up to 95% of the model weights without significant degradation in accuracy. Hence, we can obtain very light models for generative audio across popular methods such as Wavenet, SING or DDSP, that are up to 100 times smaller with commensurate accuracy. We study the theoretical bounds for embedding these models on Raspberry Pi and Arduino, and show that we can obtain generative models on CPU with equivalent quality as large GPU models. Finally, we discuss the possibility of implementing deep generative audio models on embedded platforms.

研究动机与目标

• 解决深度生成音频模型带来的高计算与内存开销问题，从而限制其在实时与嵌入式系统中的部署。
• 探究彩票券假设（即通过稀疏子网络初始化实现高性能）是否适用于生成音频模型。
• 通过开发结构化剪枝方法，克服非结构化剪枝的低效性，实现对整个计算单元（如卷积通道）的移除，从而减少模型大小与推理时间。
• 通过优化FLOPs、磁盘大小与内存使用，实现在低资源平台（如Arduino与树莓派）上部署高质量生成音频模型。

提出的方法

• 将彩票券假设应用于深度生成音频模型（如WaveNet、SING、DDSP），在初始化阶段识别出稀疏子网络，这些子网络在独立训练时可实现高准确率。
• 通过移除整个卷积通道而非单个权重，引入结构化剪枝，实现模型大小与推理成本的真实降低。
• 提出基于计算单元与目标输出之间互信息的全局选择准则，以指导结构化剪枝并保持性能。
• 采用迭代剪枝与权重重置的再训练策略，识别并训练最有效的子网络，确保模型鲁棒性与准确性。
• 使用三项指标评估模型效率：FLOPs（推理速度）、磁盘大小（存储空间）与读写内存（RAM访问），以评估实时性与可嵌入性。
• 在嵌入式平台（Arduino、树莓派）上对剪枝后的模型进行基准测试，结合推断FLOPS与硬件约束，确定可嵌入性的阈值。

实验结果

研究问题

• RQ1 彩票券假设能否成功应用于深度生成音频模型？是否能产生更小、更准确的模型？
• RQ2 与非结构化剪枝相比，通过移除整个通道而非单个权重的结构化剪枝，是否能实现模型大小与推理时间的显著减少？
• RQ3 基于互信息的全局信息论准则能否有效指导结构化剪枝，在激进压缩下仍保持高性能？
• RQ4 剪枝后的生成音频模型在低功耗平台（如Arduino与树莓派）上的可嵌入程度如何？模型大小、准确率与硬件约束之间存在何种权衡？

主要发现

• 通过结构化彩票剪枝获得的更小模型，其准确率反而高于原始大模型，证实了模型压缩带来的意外优势。
• 高达95%的模型权重可被移除而不会造成显著准确率下降，表明在生成音频模型中极端稀疏性是可行的。
• 所提出的基于互信息的准则能够有效实现全局结构化剪枝，在激进压缩下表现优于局部剪枝，更有效地保持准确率。
• 轻量化模型相比原始模型缩小了100倍以上，同时保持GPU级音频合成质量，实现CPU上的实时推理。
• 在树莓派1B上，模型大小与内存使用在约束范围内，但FLOPs仍处于临界水平；进一步激进剪枝可实现可嵌入性，仅造成微小准确率损失。
• 在Arduino平台上，即使最小的模型也超出FLOPS与内存限制，可嵌入模型的误差率需比原始模型高出2.5倍，表明当前硬件约束仍具挑战性。