[Paper Review] Soft edit distance for differentiable comparison of symbolic sequences

Evgenii Ofitserov, В. Я. Цветков+1 more|arXiv (Cornell University)|Apr 29, 2019

Algorithms and Data Compression10 references4 citations

TL;DR

This paper introduces Soft Edit Distance (SED), a differentiable, smooth approximation of Levenshtein edit distance that enables gradient-based optimization for symbolic sequences. By representing sequences as continuous encoding matrices and applying a softmin operation with temperature parameter τ, SED supports efficient backpropagation and polynomial-time computation via recurrence relations, enabling effective sequence clustering and consensus search in bioinformatics and NLP.

ABSTRACT

Edit distance, also known as Levenshtein distance, is an essential way to compare two strings that proved to be particularly useful in the analysis of genetic sequences and natural language processing. However, edit distance is a discrete function that is known to be hard to optimize. This fact hampers the use of this metric in Machine Learning. Even as simple algorithm as K-means fails to cluster a set of sequences using edit distance if they are of variable length and abundance. In this paper we propose a novel metric - soft edit distance (SED), which is a smooth approximation of edit distance. It is differentiable and therefore it is possible to optimize it with gradient methods. Similar to original edit distance, SED as well as its derivatives can be calculated with recurrent formulas at polynomial time. We prove usefulness of the proposed metric on synthetic datasets and clustering of biological sequences.

Motivation & Objective

• Address the challenge of optimizing discrete edit distance in machine learning due to its non-differentiability.
• Enable gradient-based optimization for sequence clustering and centroid computation in variable-length symbolic sequences.
• Develop a continuous, differentiable metric that approximates edit distance while preserving its interpretability and computational efficiency.
• Facilitate application of standard ML algorithms like K-means to sequence data by replacing discrete edit distance with a differentiable alternative.
• Provide a scalable solution for large-scale sequence analysis in bioinformatics and NLP, overcoming limitations of traditional dynamic programming on big data.

Proposed method

• Represent symbolic sequences as continuous one-hot encoded matrices of shape L×|G|, where each row is a softmax-probability vector over the alphabet G.
• Define Soft Edit Distance (SED) using a softmin operation over all possible edit paths, weighted by exponential of negative edit cost scaled by temperature τ.
• Formulate SED and its gradients using recurrence relations over dynamic programming states, analogous to classic edit distance but differentiable.
• Introduce auxiliary variables αi,j and βi,j to recursively compute SED and its derivatives, enabling backpropagation through the sequence comparison process.
• Use temperature τ < 0 to control the softness of the minimum operation, with τ → −∞ recovering the original edit distance.
• Derive closed-form recurrence relations for SED and its gradients using weighted sums over edit operations (match, insert, delete, substitute), enabling polynomial-time computation.

Experimental results

Research questions

• RQ1 Can a differentiable approximation of edit distance be constructed that retains the interpretability and structure of the original metric?
• RQ2 Can such a differentiable metric enable effective gradient-based optimization for sequence clustering and consensus sequence detection?
• RQ3 Does the proposed soft edit distance maintain computational efficiency comparable to classic edit distance while being amenable to backpropagation?
• RQ4 How well does the soft edit distance perform in comparison to standard edit distance on synthetic and real biological sequence clustering tasks?
• RQ5 Can the soft edit distance be used to efficiently compute centroids in sequence space using gradient-based optimization?

Key findings

• The proposed Soft Edit Distance (SED) is a smooth, differentiable approximation of Levenshtein edit distance that allows gradient-based optimization in sequence space.
• SED and its gradients can be computed in polynomial time using recurrence relations over dynamic programming states, similar to classic edit distance.
• The method enables effective clustering of variable-length symbolic sequences using gradient-based algorithms such as K-means, which fail with discrete edit distance.
• The softmin operation with temperature τ provides a continuous relaxation of the discrete minimum, allowing backpropagation through the edit path selection process.
• Empirical evaluation on synthetic and biological sequences confirms that SED supports accurate consensus sequence detection and clustering, outperforming non-differentiable alternatives in optimization stability.
• The recurrence relations for αi,j and βi,j enable efficient computation of SED and its derivatives, making the method scalable for large-scale sequence analysis in bioinformatics and NLP.