[Paper Review] Soft-DTW: a Differentiable Loss Function for Time-Series

Marco Cuturi, Mathieu Blondel|arXiv (Cornell University)|Mar 5, 2017

Time Series Analysis and Forecasting24 references140 citations

TL;DR

The paper introduces soft-DTW, a differentiable, smoothed version of DTW, enabling gradient-based learning for time-series tasks such as averaging, clustering, and multistep prediction under the DTW geometry, with quadratic time and linear space complexity.

ABSTRACT

We propose in this paper a differentiable learning loss between time series, building upon the celebrated dynamic time warping (DTW) discrepancy. Unlike the Euclidean distance, DTW can compare time series of variable size and is robust to shifts or dilatations across the time dimension. To compute DTW, one typically solves a minimal-cost alignment problem between two time series using dynamic programming. Our work takes advantage of a smoothed formulation of DTW, called soft-DTW, that computes the soft-minimum of all alignment costs. We show in this paper that soft-DTW is a differentiable loss function, and that both its value and gradient can be computed with quadratic time/space complexity (DTW has quadratic time but linear space complexity). We show that this regularization is particularly well suited to average and cluster time series under the DTW geometry, a task for which our proposal significantly outperforms existing baselines. Next, we propose to tune the parameters of a machine that outputs time series by minimizing its fit with ground-truth labels in a soft-DTW sense.

Motivation & Objective

• Motivate learning with outputs that are time series, handling varying length, shifts, and dilations along the time axis.
• Introduce soft-DTW as a differentiable loss that generalizes DTW.
• Show that gradients of soft-DTW w.r.t. inputs can be computed efficiently.
• Demonstrate applications to averaging, clustering, and predicting time-series segments under DTW geometry.

Proposed method

• Define soft-DTW via a differentiable min^gamma operator that smooths the DTW alignment costs.
• Compute the forward Bellman-like recursion with min^gamma to obtain dtw_gamma(x,y) in O(nm) time and space.
• Derive gradient formulas: for gamma>0, grad_x dtw_gamma(x,y) = (∂Δ/∂x)^T E_gamma[A], where E_gamma[A] is the Gibbs-averaged alignment matrix.
• Provide a backward pass (Algorithm 2) that backpropagates through the DP to obtain gradients in O(nm) time and space.
• Show how to use soft-DTW as a fitting loss for time-series averaging (Fréchet means), clustering (k-means with soft-DTW), and multistep-ahead prediction via neural models.

Experimental results

Research questions

• RQ1 Can soft-DTW serve as a differentiable alternative to DTW for end-to-end learning with time-series outputs?
• RQ2 How can one compute gradients of soft-DTW efficiently to enable gradient-based optimization?
• RQ3 What gains does soft-DTW offer for averaging, clustering, and predicting time-series under the DTW geometry?
• RQ4 How does smoothing (gamma) affect optimization landscape and predictive performance compared to classical DTW/DBA baselines.

Key findings

• Soft-DTW is differentiable with gradients that can be computed alongside the loss in quadratic time/space.
• The backward pass reuses log-sum-exp computations for numerical stability and efficiency.
• Smoothing the DTW (choosing gamma>0) improves optimization and helps avoid poor local minima in time-series averaging (barycenters) and clustering compared to DTW/DBA baselines.
• Soft-DTW yields smoother barycenters and often lower fitting loss than DBA and subgradient approaches, especially as gamma decreases.
• When used in a learning setup (e.g., multistep-ahead prediction), soft-DTW can produce predictions that capture sharp changes with appropriate time shifts.