[Paper Review] Convolutional RNN: an Enhanced Model for Extracting Features from Sequential Data

Gil Keren, Björn W. Schuller|arXiv (Cornell University)|Feb 18, 2016

Music and Audio Processing27 references18 citations

TL;DR

This paper proposes Convolutional Recurrent Neural Networks (CRNNs), which enhance feature extraction from sequential data by processing local windows frame-by-frame through recurrent layers (LSTM/GRU), leveraging temporal structure within each window. The method outperforms standard convolutional layers in audio classification tasks, especially with low-complexity features like log mel filter-banks, demonstrating improved classification accuracy by exploiting richer, multi-step nonlinear transformations over data patches.

ABSTRACT

Traditional convolutional layers extract features from patches of data by applying a non-linearity on an affine function of the input. We propose a model that enhances this feature extraction process for the case of sequential data, by feeding patches of the data into a recurrent neural network and using the outputs or hidden states of the recurrent units to compute the extracted features. By doing so, we exploit the fact that a window containing a few frames of the sequential data is a sequence itself and this additional structure might encapsulate valuable information. In addition, we allow for more steps of computation in the feature extraction process, which is potentially beneficial as an affine function followed by a non-linearity can result in too simple features. Using our convolutional recurrent layers we obtain an improvement in performance in two audio classification tasks, compared to traditional convolutional layers. Tensorflow code for the convolutional recurrent layers is publicly available in https://github.com/cruvadom/Convolutional-RNN.

Motivation & Objective

• To address the limitation of traditional convolutional layers, which use simple affine transformations followed by non-linearities, potentially yielding suboptimal features for sequential data.
• To exploit the inherent sequential structure within local data windows (e.g., frames in audio) by applying recurrent networks to extract more complex, context-aware features.
• To improve classification performance in audio tasks by replacing standard convolutional layers with recurrent-based feature extraction mechanisms.
• To evaluate whether recurrent processing of data patches leads to better representation learning than standard convolutional operations, especially when input features are low-complexity.

Proposed method

• The CRNN model processes each local window of sequential data frame-by-frame through a recurrent layer (LSTM or GRU), capturing temporal dependencies within the window.
• Hidden states, outputs, or cell states from the recurrent layer at each time step are used to compute the final feature representation for the window.
• Three variants are proposed: CLSTM (unidirectional), Extended CLSTM (with additional dense layers), and CBLSTM (bidirectional), where outputs from forward and backward LSTMs are combined via learnable weights.
• Feature extraction is performed by applying a pooling operation (e.g., max-pooling) over the sequence of recurrent outputs for each window.
• The model is integrated into a larger deep learning architecture for end-to-end training on audio classification tasks.
• TensorFlow code for the CRNN layers is publicly available on GitHub, enabling reproducibility and integration into other models.

Experimental results

Research questions

• RQ1 Can recurrent processing of local data windows improve feature representation compared to standard convolutional layers in sequential data?
• RQ2 Does leveraging the internal temporal structure of data patches (e.g., audio frames) lead to better classification performance in audio tasks?
• RQ3 How do different recurrent architectures (unidirectional vs. bidirectional) affect feature extraction and downstream classification accuracy?
• RQ4 To what extent can CRNNs improve performance when input features are low-complexity (e.g., log mel filter-banks) or higher-level hand-crafted features (e.g., eGeMAPS)?
• RQ5 Can the CRNN model generalize beyond audio and be applied effectively to other sequential data tasks such as speech recognition or video classification?

Key findings

• The CRNN model achieved improved classification accuracy on audio tasks compared to standard convolutional layers, even when using low-complexity log mel filter-bank features.
• The CBLSTM variant, which uses bidirectional LSTMs and combines forward and backward outputs, outperformed the unidirectional CLSTM variant, suggesting that future context enhances feature quality.
• The use of cell states in the CLSTM model led to conflicting roles—serving both as memory for the next time step and as feature input—potentially limiting performance, which the CBLSTM design mitigates by decoupling these roles.
• Even with higher-level eGeMAPS features, the CRNN model still yielded measurable improvements over standard convolutional layers, indicating its ability to extract additional useful information.
• The CRNN model scales efficiently with window size, as the number of parameters does not grow with window length (except in the extended CLSTM variant), making it suitable for long sequences.