[Paper Review] Hybrid Neural Models For Sequence Modelling: The Best Of Three Worlds

Marco Dinarelli, Loïc Grobol|arXiv (Cornell University)|Sep 16, 2019

Natural Language Processing Techniques39 references4 citations

TL;DR

This paper proposes a hybrid neural architecture integrating bidirectional GRUs, sequence-to-sequence modeling, and Transformer self-attention for sequence labeling tasks. The model achieves state-of-the-art or competitive results on three benchmark datasets—MEDIA (French SLU), WSJ (English POS tagging), and TIGER (German morpho-syntactic tagging)—demonstrating the effectiveness of combining these architectures for sequence modeling.

ABSTRACT

We propose a neural architecture with the main characteristics of the most successful neural models of the last years: bidirectional RNNs, encoder-decoder, and the Transformer model. Evaluation on three sequence labelling tasks yields results that are close to the state-of-the-art for all tasks and better than it for some of them, showing the pertinence of this hybrid architecture for this kind of tasks.

Motivation & Objective

• To design a unified neural architecture that integrates the strengths of bidirectional RNNs, sequence-to-sequence models, and the Transformer for sequence labeling.
• To evaluate the hybrid model on diverse sequence labeling tasks beyond just POS tagging, including SLU and morpho-syntactic tagging.
• To determine whether combining these architectures improves performance over existing models, especially in low-resource or less-explored settings.
• To assess the model's generalization potential by applying it to tasks like machine translation and syntactic parsing, despite not being evaluated on them here.
• To investigate whether the one-to-one input-output correspondence in sequence labeling is better preserved by hybrid models than by pure Transformers.

Proposed method

• The encoder uses a bidirectional GRU to process input tokens, combining word embeddings with character-level representations computed via a separate bidirectional GRU over characters.
• The character-level representation is obtained by summing the final hidden states of the character-level GRU and passing them through a feedforward network.
• The decoder employs a bidirectional context mechanism inspired by prior work, allowing predictions to depend on both past and future labels, enhancing context awareness.
• The model integrates self-attention mechanisms from the Transformer architecture into the decoder to capture long-range dependencies in the output sequence.
• The architecture is trained end-to-end using cross-entropy loss, with no pre-trained embeddings in some experiments, and compared against strong baselines including LSTM+CRF and Transformer models.
• The model is evaluated on three sequence labeling tasks: French SLU (MEDIA), English POS tagging (WSJ), and German morpho-syntactic tagging (TIGER), with ablation studies on decoder variants.

Experimental results

Research questions

• RQ1 Can a hybrid neural architecture combining bidirectional RNNs, sequence-to-sequence modeling, and Transformer attention achieve state-of-the-art performance on sequence labeling tasks?
• RQ2 Does the integration of self-attention in a sequence-to-sequence framework with bidirectional context improve performance on sequence labeling compared to standard RNN-based models?
• RQ3 How does the model perform on less commonly evaluated sequence labeling datasets like TIGER, especially when compared to non-neural or older neural models?
• RQ4 To what extent does the absence of pre-trained word embeddings affect the model’s performance, and can it still outperform models using such embeddings?
• RQ5 Why do pure Transformer models underperform on sequence labeling tasks, and can a hybrid architecture mitigate this issue by preserving input-output alignment?

Key findings

• The proposed hybrid model achieves a test F1 score of 91.86 on the TIGER morpho-syntactic tagging task, outperforming the previous state-of-the-art model (VO-CRF) which achieved 88.78.
• On the MEDIA corpus for French SLU, the model achieves an accuracy of 93.90 (F1: 98.30 on dev), demonstrating strong performance on spoken language understanding tasks.
• On the WSJ corpus for POS tagging, the model reaches 91.86 F1 on test, surpassing the LSTM+CRF baseline and approaching state-of-the-art results without using GloVe pre-trained embeddings.
• The model outperforms the Transformer and Star-Transformer models reported in Guo et al. (2019), even when the latter used pre-trained embeddings, indicating the advantage of architectural hybridization.
• The model shows better training and development loss curves than its predecessor (Seq2Biseq), suggesting improved optimization dynamics despite similar final performance.
• The model generalizes well to other tasks: it is adaptable to machine translation and syntactic parsing, indicating its broader applicability beyond sequence labeling.