[Paper Review] Attention-based Information Fusion using Multi-Encoder-Decoder Recurrent Neural Networks

Stephan Baier, Sigurd Spieckermann+1 more|arXiv (Cornell University)|Nov 13, 2017

Target Tracking and Data Fusion in Sensor Networks4 references3 citations

TL;DR

This paper proposes a multi-encoder-decoder RNN architecture with a spatial attention mechanism to fuse contextual information from multiple distributed sensor stations for improved sequence-to-sequence prediction. By dynamically weighting latent representations based on context-specific relevance, the model outperforms linear models and standard RNNs across three real-world sensor network datasets, demonstrating superior performance in modeling cross-correlations in multivariate time series.

ABSTRACT

With the rising number of interconnected devices and sensors, modeling distributed sensor networks is of increasing interest. Recurrent neural networks (RNN) are considered particularly well suited for modeling sensory and streaming data. When predicting future behavior, incorporating information from neighboring sensor stations is often beneficial. We propose a new RNN based architecture for context specific information fusion across multiple spatially distributed sensor stations. Hereby, latent representations of multiple local models, each modeling one sensor station, are jointed and weighted, according to their importance for the prediction. The particular importance is assessed depending on the current context using a separate attention function. We demonstrate the effectiveness of our model on three different real-world sensor network datasets.

Motivation & Objective

• To address the challenge of modeling complex, distributed sensor networks with interdependent multivariate time series data.
• To improve sequence-to-sequence prediction in sensor networks by fusing information from multiple spatially distributed stations.
• To enable context-aware information fusion by learning dynamic, attention-based weighting of latent representations from different sensor stations.
• To demonstrate the effectiveness of the proposed architecture on real-world climatological and smart grid datasets.
• To show that attention-based fusion outperforms both linear models and standard RNNs in capturing non-linear cross-correlations across sensor stations.

Proposed method

• The model uses separate RNN encoders for each sensor station to learn latent representations from multivariate input sequences.
• An interconnection layer with a spatial attention mechanism combines the latent representations of all encoders, assigning dynamic weights based on current context.
• The attention mechanism computes context-dependent importance scores for each encoder’s representation, enabling selective fusion.
• Each decoder is initialized with the contextually weighted fused representation and generates predictions for the target sequence.
• The entire architecture is end-to-end differentiable and trained using backpropagation through time.
• The model is applied to multivariate time series from sensor networks, with input and output sequences defined as three-way tensors across stations, time, and features.

Experimental results

Research questions

• RQ1 Can an attention-based fusion mechanism improve sequence-to-sequence prediction in distributed sensor networks compared to fixed or linear fusion?
• RQ2 How effectively can a multi-encoder-decoder RNN architecture exploit cross-correlations between spatially distributed sensor stations?
• RQ3 Does context-aware attention weighting lead to better performance than standard RNNs or linear models in multivariate time series forecasting?
• RQ4 To what extent do joint models of all stations outperform per-station models in sensor network prediction tasks?
• RQ5 Can the proposed architecture generalize to diverse sensor network applications such as climatological monitoring and smart grid load forecasting?

Key findings

• On the Quebec climatological dataset, the proposed attention-based multi-encoder-decoder RNN achieved a mean squared error (MSE) of 32.28%, outperforming the next best model (34.77%) using a joint RNN.
• On the Alberta climatological dataset, the model achieved an MSE of 32.89%, compared to 34.68% for the joint RNN baseline.
• On the smart grid dataset, the model achieved an MSE of 28.84%, outperforming the joint RNN baseline (29.56%) and the linear model with all stations (31.64%).
• The performance gain from the attention mechanism was most pronounced in datasets with strong cross-correlations, such as the climatological data, where joint models significantly outperformed per-station models.
• The model consistently outperformed both linear regression and standard RNNs in all three datasets, demonstrating the value of context-aware fusion.
• The use of GRUs or LSTMs did not significantly improve results, suggesting that the attention mechanism's contribution is distinct from recurrent cell design.