[Paper Review] Multi-Modal Prototype Learning for Interpretable Multivariable Time Series Classification

Gaurav R. Ghosal, Reza Abbasi-Asl|arXiv (Cornell University)|Jun 17, 2021

Time Series Analysis and Forecasting31 references4 citations

TL;DR

This paper proposes a modular, multi-level prototype learning framework for interpretable multivariable time series classification that first learns single-variable prototypes via contrastive learning, then combines them into multivariable prototypes to capture cross-variable patterns. The method achieves state-of-the-art classification performance while providing human-interpretable insights into both individual variable patterns and their class-discriminative relationships, validated on synthetic and real human activity recognition data.

ABSTRACT

Multivariable time series classification problems are increasing in prevalence and complexity in a variety of domains, such as biology and finance. While deep learning methods are an effective tool for these problems, they often lack interpretability. In this work, we propose a novel modular prototype learning framework for multivariable time series classification. In the first stage of our framework, encoders extract features from each variable independently. Prototype layers identify single-variable prototypes in the resulting feature spaces. The next stage of our framework represents the multivariable time series sample points in terms of their similarity to these single-variable prototypes. This results in an inherently interpretable representation of multivariable patterns, on which prototype learning is applied to extract representative examples i.e. multivariable prototypes. Our framework is thus able to explicitly identify both informative patterns in the individual variables, as well as the relationships between the variables. We validate our framework on a simulated dataset with embedded patterns, as well as a real human activity recognition problem. Our framework attains comparable or superior classification performance to existing time series classification methods on these tasks. On the simulated dataset, we find that our model returns interpretations consistent with the embedded patterns. Moreover, the interpretations learned on the activity recognition dataset align with domain knowledge.

Motivation & Objective

• To address the challenge of interpretability in multivariable time series classification, where deep learning models often act as black boxes.
• To explicitly model both single-variable patterns and cross-variable relationships in a way that is transparent and interpretable.
• To develop a modular training procedure that enables effective learning of prototypes across heterogeneous variables with differing noise levels, time scales, and domains.
• To validate the interpretability of the learned prototypes using a synthetic dataset with known embedded patterns and a real-world human activity recognition dataset.
• To demonstrate that the framework can reveal meaningful, domain-aligned interpretations, such as distinguishing movement types by signal frequency and amplitude.

Proposed method

• The framework uses separate encoders for each variable, trained via contrastive learning to extract meaningful features from individual time series.
• Single-variable prototype layers represent each sample as a similarity score to a set of learned prototypes in the feature space of each variable.
• Similarity scores across all variables are concatenated to form an inherently interpretable multivariable representation.
• A second prototype layer learns multivariable prototypes by clustering the concatenated similarity representations, identifying class-discriminative patterns across variables.
• The method uses a two-stage training process: first pre-training encoders and single-variable prototypes, then jointly training the multivariable prototype layer.
• The framework allows for model-based interpretability by explicitly identifying which variable patterns and their combinations are most relevant for classification.

Experimental results

Research questions

• RQ1 Can a prototype-based framework effectively learn and represent both single-variable and cross-variable patterns in multivariable time series?
• RQ2 Does the framework produce interpretations that align with known patterns in synthetic data and domain knowledge in real-world data?
• RQ3 How does the model’s interpretability compare to existing deep learning methods in terms of clarity and scientific insight?
• RQ4 To what extent can the model distinguish between classes based on distinct signal characteristics such as frequency and amplitude?
• RQ5 Can the framework reveal overlapping patterns across classes, such as similar signals in one variable for different activities?

Key findings

• On the simulated dataset, the model successfully recovered the embedded patterns, correctly identifying that sawing had high-frequency, low-amplitude signals, walking had low-frequency, low-amplitude signals, and running had high-frequency, variable-amplitude signals.
• In the human activity recognition task, the multivariable prototypes revealed that the sawing and epilepsy classes shared similar patterns in the first variable, indicating potential overlap in signal characteristics.
• The model identified one-hot encoded multivariable prototypes for walking, running, and sawing, indicating that each class was associated with a single dominant pattern per variable, making them easily distinguishable.
• For the epilepsy class, two non-zero entries in the first variable block indicated the presence of two distinct signal patterns, which was confirmed by UMAP visualizations and revealed by the multivariable prototype structure.
• The framework demonstrated comparable or superior classification performance to existing methods while providing detailed, human-readable interpretations grounded in domain knowledge.
• The method enabled fine-grained feature importance analysis, showing that certain variables contributed more to class discrimination than others, and revealed which specific patterns were most relevant for classification.