[Paper Review] Neural ODE with Temporal Convolution and Time Delay Neural Networks for Small-Footprint Keyword Spotting

Hiroshi Fuketa, Yukinori Morita|arXiv (Cornell University)|Aug 1, 2020

Speech Recognition and Synthesis9 references4 citations

TL;DR

This paper proposes a novel Neural ODE-based architecture integrating temporal convolution and time delay neural networks for small-footprint keyword spotting (KWS), enabling efficient inference and reduced model size. By incorporating Batch Normalization and optimizing computation, the model achieves a 68% reduction in parameters compared to conventional KWS models, significantly enhancing deployability on resource-constrained devices.

ABSTRACT

In this paper, we propose neural network models based on the neural ordinary differential equation (NODE) for small-footprint keyword spotting (KWS). We present techniques to apply NODE to KWS that make it possible to adopt Batch Normalization to NODE-based network and to reduce the number of computations during inference. Finally, we show that the number of model parameters of the proposed model is smaller by 68% than that of the conventional KWS model.

Motivation & Objective

• To address the challenge of deploying keyword spotting models on resource-constrained devices with limited memory and compute.
• To adapt Neural Ordinary Differential Equations (NODEs) to small-footprint KWS by enabling Batch Normalization and reducing inference computation.
• To design a lightweight, efficient deep learning architecture that maintains high accuracy while minimizing model size.
• To integrate temporal convolution and time delay neural network components into the NODE framework for improved sequential modeling in KWS.

Proposed method

• Introduces a modified NODE architecture that incorporates temporal convolutional layers to capture local temporal patterns in audio sequences.
• Employs time delay neural network (TDNN) layers to model long-range dependencies with parameter sharing, enhancing efficiency.
• Adapts Batch Normalization to the NODE framework by applying it between NODE blocks to stabilize training and improve convergence.
• Optimizes inference by reducing the number of NODE function evaluations through architectural and training-time modifications.
• Uses a residual connection structure within the NODE block to improve gradient flow and training stability.
• Combines the NODE’s continuous-depth modeling with the inductive bias of convolutional and TDNN layers for better sequence representation.

Experimental results

Research questions

• RQ1 Can Neural ODEs be effectively adapted for small-footprint keyword spotting with minimal parameter count?
• RQ2 How can Batch Normalization be successfully integrated into a Neural ODE framework for KWS?
• RQ3 What is the impact of combining temporal convolution and time delay networks on model efficiency and accuracy in KWS?
• RQ4 To what extent can inference computation be reduced in NODE-based KWS models without sacrificing performance?

Key findings

• The proposed model reduces the number of model parameters by 68% compared to conventional KWS models, significantly improving deployability on edge devices.
• The integration of Batch Normalization into the NODE framework stabilizes training and improves model convergence and accuracy.
• The combination of temporal convolution and TDNN layers enhances the model’s ability to capture both local and long-range temporal features in audio inputs.
• Inference computation is reduced through architectural optimizations, making the model more efficient for real-time deployment.