[Paper Review] LEAF: A Learnable Frontend for Audio Classification

Neil Zeghidour, Olivier Teboul+2 more|arXiv (Cornell University)|Jan 21, 2021

Music and Audio Processing50 references30 citations

TL;DR

LEAF introduces a fully learnable audio frontend that replaces mel-filterbanks and consistently outperforms them across diverse tasks, with far fewer parameters.

ABSTRACT

Mel-filterbanks are fixed, engineered audio features which emulate human perception and have been used through the history of audio understanding up to today. However, their undeniable qualities are counterbalanced by the fundamental limitations of handmade representations. In this work we show that we can train a single learnable frontend that outperforms mel-filterbanks on a wide range of audio signals, including speech, music, audio events and animal sounds, providing a general-purpose learned frontend for audio classification. To do so, we introduce a new principled, lightweight, fully learnable architecture that can be used as a drop-in replacement of mel-filterbanks. Our system learns all operations of audio features extraction, from filtering to pooling, compression and normalization, and can be integrated into any neural network at a negligible parameter cost. We perform multi-task training on eight diverse audio classification tasks, and show consistent improvements of our model over mel-filterbanks and previous learnable alternatives. Moreover, our system outperforms the current state-of-the-art learnable frontend on Audioset, with orders of magnitude fewer parameters.

Motivation & Objective

• Motivate replacing handcrafted mel-filterbanks with a learnable frontend for general-purpose audio classification.
• Propose a lightweight, end-to-end trainable frontend that learns filtering, pooling, and compression.
• Show that a single LEAF frontend can outperform mel-filterbanks and prior learnable frontends across multiple tasks.
• Demonstrate scalability to large-scale Audioset and multi-task settings.

Proposed method

• Decompose the frontend into three learnable components: filtering, pooling, and compression/normalization.
• Use a complex-valued 1-D filterbank represented via real-imaginary pairs or Gabor parameterization for learnable filtering.
• Apply per-channel Gaussian lowpass pooling implemented as depthwise convolution.
• Adopt a learnable Per-Channel Energy Normalization variant (sPCEN) for per-channel compression and normalization.
• Train end-to-end with a shared backbone encoder on eight datasets and evaluate on single-task, multi-task, and Audioset setups.

Experimental results

Research questions

• RQ1 Can a single, fully learnable frontend outperform fixed mel-filterbanks across a broad set of audio classification tasks?
• RQ2 Does jointly learning filtering, pooling, and compression improve performance in multi-task and large-scale settings?
• RQ3 How does LEAF compare to prior learnable frontends (e.g., Time-Domain filterbanks, SincNet) on diverse datasets and metrics?
• RQ4 What is the impact of per-channel compression (sPCEN) on performance and robustness?

Key findings

• LEAF outperforms mel-filterbanks and prior learnable frontends across eight single-task datasets on average.
• In multi-task settings, LEAF delivers the best overall performance across tasks while sharing parameters, matching or beating baselines on all tasks.
• On Audioset, LEAF with EfficientNetB0 achieves higher d-prime than mel-filterbanks and comparable or better than Wavegram and SincNet variants with far fewer parameters (e.g., LEAF 448 params vs hundreds of thousands).
• PCEN-based compression (especially sPCEN) significantly improves performance over log-compression for both mel- and LEAF-based frontends.
• LEAF demonstrates robustness to noise, outperforming mel-filterbanks under degraded SNR conditions.