[论文解读] Sparks of Large Audio Models: A Survey and Outlook

Siddique Latif, Moazzam Shoukat+9 more|arXiv (Cornell University)|Aug 24, 2023

Music and Audio Processing被引用 11

一句话总结

本论文综述大型音频模型的崛起，分析架构、任务、数据集与挑战，并概述未来研究方向。

ABSTRACT

This survey paper provides a comprehensive overview of the recent advancements and challenges in applying large language models to the field of audio signal processing. Audio processing, with its diverse signal representations and a wide range of sources--from human voices to musical instruments and environmental sounds--poses challenges distinct from those found in traditional Natural Language Processing scenarios. Nevertheless, extit{Large Audio Models}, epitomized by transformer-based architectures, have shown marked efficacy in this sphere. By leveraging massive amount of data, these models have demonstrated prowess in a variety of audio tasks, spanning from Automatic Speech Recognition and Text-To-Speech to Music Generation, among others. Notably, recently these Foundational Audio Models, like SeamlessM4T, have started showing abilities to act as universal translators, supporting multiple speech tasks for up to 100 languages without any reliance on separate task-specific systems. This paper presents an in-depth analysis of state-of-the-art methodologies regarding extit{Foundational Large Audio Models}, their performance benchmarks, and their applicability to real-world scenarios. We also highlight current limitations and provide insights into potential future research directions in the realm of extit{Large Audio Models} with the intent to spark further discussion, thereby fostering innovation in the next generation of audio-processing systems. Furthermore, to cope with the rapid development in this area, we will consistently update the relevant repository with relevant recent articles and their open-source implementations at https://github.com/EmulationAI/awesome-large-audio-models.

研究动机与目标

• 调查大型人工智能模型在音频信号处理中的应用，涵盖语音与音乐。
• 分析基础大型音频模型及其跨模态能力。
• 识别该领域当前的局限性、挑战，以及有前景的研究方向。

提出的方法

• 评审并综合近期关于大型音频模型与基础音频模型的文献。
• 总结在基于变换器的音频模型中使用的架构与数据表示。
• 讨论多模态和跨任务能力，包括跨语言和翻译方面。
• 突出推动当前进展的关键数据集和训练策略。

实验结果

研究问题

• RQ1 当前最先进的大型音频模型及其在语音与音乐任务中的核心能力是什么？
• RQ2 基础音频模型如何处理音频处理中的跨模态与多语言任务？
• RQ3 阻碍大型音频模型在实际应用部署的主要局限性与未解决的挑战有哪些？
• RQ4 推动大型音频建模前进的未来方向与研究机会中，哪些最具潜力？

主要发现

• 本文首次对应用于音频信号处理的大型人工智能模型进行全面综述。
• 基础音频模型使语音任务具备跨任务和多语言能力。
• 对一系列最先进的模型（如 SpeechGPT、AudioPaLM、AudioLM、MusicGen、SeamlessM4T）在架构、数据与任务方面进行分析。
• 该综述讨论了局限性并勾勒了大型音频建模的潜在未来研究方向。
• 作者维护一个包含开源实现的公共仓库，以支持持续的工作。
• 该综述强调了基础音频模型在100种语言中的通用翻译能力（如文中所述）。