[Paper Review] StackSeq2Seq: Dual Encoder Seq2Seq Recurrent Networks

Alessandro Bay, Biswa Sengupta|arXiv (Cornell University)|Oct 11, 2017

Machine Learning in Bioinformatics11 references4 citations

TL;DR

This paper proposes StackSeq2Seq, a dual encoder Seq2Seq recurrent network that improves shortest path prediction on graphs by combining context vectors from separate LSTM and GRU encoders. The method achieves 59.6% accuracy on shortest paths and 78.3% on successful paths—10% improvement over single-encoder models—when combined with homotopy continuation-based loss smoothing.

ABSTRACT

A widely studied non-deterministic polynomial time (NP) hard problem lies in finding a route between the two nodes of a graph. Often meta-heuristics algorithms such as $A^{*}$ are employed on graphs with a large number of nodes. Here, we propose a deep recurrent neural network architecture based on the Sequence-2-Sequence (Seq2Seq) model, widely used, for instance in text translation. Particularly, we illustrate that utilising a context vector that has been learned from two different recurrent networks enables increased accuracies in learning the shortest route of a graph. Additionally, we show that one can boost the performance of the Seq2Seq network by smoothing the loss function using a homotopy continuation of the decoder's loss function.

Motivation & Objective

• To improve the ability of recurrent neural networks to learn and generalize shortest paths in graphs with variable sequence lengths.
• To investigate whether combining two distinct recurrent encoders (LSTM and GRU) enhances sequence modeling fidelity compared to single-encoder architectures.
• To evaluate the impact of homotopy continuation on loss function smoothing for improving training stability and path prediction accuracy.
• To assess whether increased latent dimensionality or improved encoder dynamics drives performance gains in sequence-to-sequence learning for graph pathfinding.
• To provide a scalable, differentiable alternative to traditional meta-heuristics like A* for pathfinding, without replacing them.

Proposed method

• The model employs a dual encoder architecture: one LSTM and one GRU, each encoding the source and destination nodes of a graph path into separate hidden states.
• The context vectors from both encoders are concatenated to form a joint 512-dimensional latent representation, which serves as the initial hidden state for the decoder.
• The decoder generates the predicted path token-by-token using embedded node representations, with attention mechanisms to attend to relevant encoder states.
• A homotopy continuation-based loss smoothing technique is applied by diffusing the decoder’s loss function using a Gaussian kernel with standard deviation σ ∈ {30, 5, 1, 0.0001} to improve optimization stability.
• The model is trained using Adam optimization with β₁ = 0.9, β₂ = 0.999, and an initial learning rate of 10⁻³ over 400 epochs.
• Training uses a 67-33% split of paths generated via A* on the Minnesota road network graph (376 nodes, 455 edges), with path lengths averaging 19 hops.

Experimental results

Research questions

• RQ1 Can combining two distinct recurrent encoders (LSTM and GRU) improve the sequence modeling accuracy of a Seq2Seq network for shortest path prediction on graphs?
• RQ2 Does the use of a smoothed loss function via homotopy continuation enhance the generalization and convergence of the Seq2Seq model in pathfinding tasks?
• RQ3 To what extent does increasing the latent dimension of the context vector (from 256 to 512) improve path prediction performance compared to architectural diversity?
• RQ4 Does the dual-encoder architecture outperform single-encoder models in terms of shortest path recovery and successful path generation?
• RQ5 How does the model’s performance scale with increasing sequence length and graph complexity in a controlled pathfinding setting?

Key findings

• The dual encoder model (LSTM + GRU) achieved 57.7% accuracy on shortest paths and 77.1% on successful paths, representing a 10% improvement over single-encoder models.
• When trained with homotopy continuation-based loss smoothing, the dual encoder model reached 59.6% accuracy on shortest paths and 78.3% on successful paths, demonstrating a 2% relative gain over the non-smoothed version.
• Doubling the hidden state dimension from 256 to 512 yielded only marginal improvements—1% increase in shortest path accuracy and 0.2–1.6% in successful paths—indicating that encoder dynamics matter more than capacity.
• The performance gain from dual encoding is attributed to the complementary dynamics of LSTM and GRU, not increased latent dimensionality, as shown by consistent results across different hidden unit counts.
• The model successfully generalizes to longer sequences, with average path lengths of 19 hops, and maintains fidelity in sequence reconstruction despite the non-convex optimization landscape.
• The use of A* as a ground-truth path generator ensures that all training and test paths are optimal, allowing for a fair evaluation of the model’s ability to approximate NP-hard shortest path solutions.