[Paper Review] Deep Distributional Sequence Embeddings Based on a Wasserstein Loss

Ahmed Abdel‐Wahab, Niels Landwehr|arXiv (Cornell University)|Dec 4, 2019

Generative Adversarial Networks and Image Synthesis4 citations

TL;DR

This paper proposes deep distributional sequence embeddings using quantile layers to capture the distribution of deep features across sequences, with a novel Wasserstein-based loss for end-to-end metric learning. The method outperforms standard vector embeddings and other distance metrics in biometric recognition tasks on eye movement, gait, and EEG data.

ABSTRACT

Deep metric learning employs deep neural networks to embed instances into a metric space such that distances between instances of the same class are small and distances between instances from different classes are large. In most existing deep metric learning techniques, the embedding of an instance is given by a feature vector produced by a deep neural network and Euclidean distance or cosine similarity defines distances between these vectors. In this paper, we study deep distributional embeddings of sequences, where the embedding of a sequence is given by the distribution of learned deep features across the sequence. This has the advantage of capturing statistical information about the distribution of patterns within the sequence in the embedding. When embeddings are distributions rather than vectors, measuring distances between embeddings involves comparing their respective distributions. We propose a distance metric based on Wasserstein distances between the distributions and a corresponding loss function for metric learning, which leads to a novel end-to-end trainable embedding model. We empirically observe that distributional embeddings outperform standard vector embeddings and that training with the proposed Wasserstein metric outperforms training with other distance functions.

Motivation & Objective

• To develop a deep metric learning framework for variable-length sequences that captures distributional patterns of learned features.
• To address the limitation of standard vector embeddings by representing sequences as distributions of deep features across time.
• To enable end-to-end training of distributional embeddings using a differentiable Wasserstein distance-based loss function.
• To improve performance in biometric identification and verification tasks, especially under open-set and few-shot conditions.
• To demonstrate the superiority of distributional embeddings over standard vector embeddings and alternative distance metrics in real-world biometric data.

Proposed method

• Uses quantile layers to approximate the empirical quantile function of deep feature activations across a sequence, producing a distributional embedding per sequence.
• Represents each sequence as a set of quantile-estimated distributions over learned filter activations, enabling statistical characterization of temporal patterns.
• Employs a piecewise linear approximation of the quantile function to ensure differentiability and gradient flow through the embedding layer.
• Defines a Wasserstein distance between distributional embeddings as the distance metric in the embedding space, which respects the underlying metric of the activation space.
• Derives a closed-form solution for the Wasserstein distance between two piecewise linear quantile functions to enable efficient backpropagation.
• Trains the entire model end-to-end using a contrastive loss based on the Wasserstein distance, optimizing for intra-class compactness and inter-class separation.

Experimental results

Research questions

• RQ1 Can distributional embeddings based on quantile-estimated feature distributions improve sequence representation in deep metric learning?
• RQ2 Does using the Wasserstein distance as a metric between distributional embeddings lead to better generalization than traditional distance functions like cosine or Euclidean?
• RQ3 How does the proposed method perform in comparison to standard vector-based embeddings in biometric recognition tasks with variable-length sequences?
• RQ4 To what extent does the model generalize to unseen classes and handle imposters in open-set recognition scenarios?
• RQ5 Does the use of quantile layers with Wasserstein loss provide consistent performance gains across diverse sequence data types (e.g., eye movement, gait, EEG)?

Key findings

• Distributional embeddings based on quantile layers significantly outperform standard vector embeddings in all three data domains: eye movement, gait, and EEG.
• The proposed Wasserstein-based loss (QP-WL) reduces 1-AUC by up to 80% compared to the next-best baseline (QP-NPL) in the EEG domain when n=5 test sequences.
• In the gait domain, QP-WL achieves higher identification accuracy and robustness to multiclass imposters than all baselines, especially as the number of enrolled subjects increases.
• The model maintains strong performance even when data is simplified (e.g., reduced temporal resolution), outperforming a specialized microsaccade detection model (Jager et al., 2019) on the same simplified data.
• The use of quantile layers instead of max-pooling improves performance, and metric learning with the Wasserstein loss further enhances results over classification-based training.
• The model demonstrates strong generalization to unseen classes and robustness in open-set scenarios, with clear advantages in imposter detection across all domains.