[Paper Review] Unified recurrent neural network for many feature types

Alexander Stec, Diego Klabjan+1 more|arXiv (Cornell University)|Sep 24, 2018

Time Series Analysis and Forecasting12 references3 citations

TL;DR

This paper proposes a unified recurrent neural network (STLSTM) that natively handles five distinct feature types—dense, sparse, time-aware, static dense, and static delta features—by treating features with varying update frequencies differently. The model improves performance on asynchronous time series by separately updating hidden states for sparse and dense features and incorporating time intervals flexibly, achieving faster convergence and better accuracy than standard RNNs.

ABSTRACT

There are time series that are amenable to recurrent neural network (RNN) solutions when treated as sequences, but some series, e.g. asynchronous time series, provide a richer variation of feature types than current RNN cells take into account. In order to address such situations, we introduce a unified RNN that handles five different feature types, each in a different manner. Our RNN framework separates sequential features into two groups dependent on their frequency, which we call sparse and dense features, and which affect cell updates differently. Further, we also incorporate time features at the sequential level that relate to the time between specified events in the sequence and are used to modify the cell's memory state. We also include two types of static (whole sequence level) features, one related to time and one not, which are combined with the encoder output. The experiments show that the modeling framework proposed does increase performance compared to standard cells.

Motivation & Objective

• To address the limitations of standard RNNs in modeling asynchronous time series with heterogeneous feature update frequencies.
• To develop a unified framework that natively supports five distinct feature types: dense, sparse, time-related, and static features.
• To improve modeling accuracy and training efficiency by treating features with different sparsity patterns differently in the RNN cell update mechanism.
• To extend time-aware RNNs to handle arbitrary time features beyond just inter-event intervals.

Proposed method

• The model splits the hidden state into dense and sparse components, updating each based on the frequency of feature presence.
• Sparse features maintain individual hidden and memory states, updated only when the feature is observed.
• Time features are modeled via a flexible decay mechanism that supports multiple delta features, not just time since last step.
• Static features are processed separately: static delta features are embedded and used within the cell, while static dense features are embedded and concatenated with the final sequence output.
• The architecture integrates all five feature types into a single, unified RNN cell, enabling end-to-end training.
• The model uses stacked RNN layers with STLSTM at the base and standard LSTMs in higher layers for hierarchical representation learning.

Experimental results

Research questions

• RQ1 How does separating dense and sparse feature processing in an RNN cell affect performance on asynchronous time series?
• RQ2 Can a unified RNN framework effectively model multiple feature types, including time-aware and static features, within a single architecture?
• RQ3 Does the proposed STLSTM cell achieve faster convergence and better accuracy compared to standard RNNs and TLSTM on sparse and irregularly sampled data?
• RQ4 How do different sparsity levels of features influence the performance gain from using the STLSTM architecture?

Key findings

• The STLSTM model reaches convergence in 40–50 epochs, compared to 70–80 epochs for standard TLSTM, despite higher per-epoch training time due to increased parameters.
• On synthetic data, the model achieves a 2.5% relative improvement in performance over standard RNNs when using sparse features, with the largest gains seen in the most sparse feature subsets.
• On real-world churn prediction data, the model shows the largest performance improvement when using customer-specific and sequence-specific sparse features (sparsity ratio 0.03–0.3), indicating responsiveness to mixed sparsity.
• Raw count features (e.g., total number of people per account) yield a larger performance gain than relative change features, which may already be semantically encoded.
• The inclusion of static delta features provides a stronger performance boost as sparsity increases, highlighting their utility in low-frequency feature contexts.