[Paper Review] Graph Sequential Network for Reasoning over Sequences

Ming Tu, Jing Huang+2 more|arXiv (Cornell University)|Apr 4, 2020

Topic Modeling30 references4 citations

TL;DR

This paper proposes Graph Sequential Network (GSN), a novel GNN variant that performs message passing directly over sequential node features using co-attention, avoiding information loss from early summarization. GSN enables sequential labeling and improves reasoning on sequence-based graphs, outperforming standard GNNs on HotpotQA and FEVER with gains of 1.29% in FEVER score and 1.19% in accuracy.

ABSTRACT

Recently Graph Neural Network (GNN) has been applied successfully to various NLP tasks that require reasoning, such as multi-hop machine reading comprehension. In this paper, we consider a novel case where reasoning is needed over graphs built from sequences, i.e. graph nodes with sequence data. Existing GNN models fulfill this goal by first summarizing the node sequences into fixed-dimensional vectors, then applying GNN on these vectors. To avoid information loss inherent in the early summarization and make sequential labeling tasks on GNN output feasible, we propose a new type of GNN called Graph Sequential Network (GSN), which features a new message passing algorithm based on co-attention between a node and each of its neighbors. We validate the proposed GSN on two NLP tasks: interpretable multi-hop reading comprehension on HotpotQA and graph based fact verification on FEVER. Both tasks require reasoning over multiple documents or sentences. Our experimental results show that the proposed GSN attains better performance than the standard GNN based methods.

Motivation & Objective

• To address information loss from early summarization in GNNs when processing sequential node features in NLP tasks.
• To enable sequential labeling tasks on GNN outputs by preserving sequence-level representations.
• To improve multi-hop reasoning over graphs built from multiple text sequences, such as in reading comprehension and fact verification.
• To develop a message passing mechanism that dynamically attends between sequences on connected nodes.
• To validate the model on reasoning-intensive NLP tasks requiring cross-document inference.

Proposed method

• GSN introduces a co-attention-based message passing mechanism between a node and each of its neighbors, replacing traditional vector-based aggregation.
• The co-attention mechanism computes attention weights between tokens of the current node and its neighbor’s sequence, enabling context-aware feature propagation.
• Each node’s representation is updated via a weighted aggregation of neighbor features using the attention matrix, preserving sequence-level structure.
• The model processes raw BERT-encoded sequences directly without pooling, maintaining full sequential context for reasoning.
• A two-step attentive pooling is applied after GSN to produce graph-level representations for classification tasks.
• The framework supports both node-level sequence labeling and graph-level classification, unlike standard GNNs that only output fixed-size vectors.

Experimental results

Research questions

• RQ1 Can a GNN variant preserve sequence-level information during message passing when nodes are represented by sequences?
• RQ2 Does co-attention-based message passing improve reasoning performance on multi-hop NLP tasks compared to pooling-based GNNs?
• RQ3 Can GSN support sequential labeling tasks, which are infeasible with standard GNNs due to vector-only outputs?
• RQ4 Does GSN achieve better performance on complex reasoning tasks such as multi-hop reading comprehension and fact verification?
• RQ5 How does the attention mechanism in GSN reveal meaningful cross-sentence attention patterns during reasoning?

Key findings

• GSN achieved a 74.89% ACC and 70.51% FEVER score on the FEVER development set, outperforming the baseline GNN system (73.72% ACC, 69.26% FEVER).
• On the FEVER test set, GSN achieved 72.00% ACC and 67.13% FEVER score, surpassing the baseline system by 1.20% in ACC and 1.25% in FEVER score.
• On HotpotQA, GSN demonstrated improved performance in interpretable multi-hop reading comprehension, with qualitative attention heatmaps showing alignment with answer spans and bridging entities.
• Visualization confirmed that GSN attends to key tokens such as 'YG Entertainment' and 'Y' in answer spans, indicating effective cross-sentence reasoning.
• The model’s ability to preserve sequence-level representations enabled feasible sequential labeling, a limitation of standard GNNs.
• The proposed co-attention mechanism effectively captures relevant information across sequences of varying lengths, even across documents.