[Paper Review] BERT-DRE: BERT with Deep Recursive Encoder for Natural Language Sentence Matching

Ehsan Tavan, Ali Rahmati+3 more|arXiv (Cornell University)|Nov 3, 2021

Topic Modeling4 citations

TL;DR

This paper proposes BERT-DRE, a deep recursive encoder architecture that enhances BERT by adding three bidirectional LSTM layers with residual connections and an attention mechanism to improve sentence matching in natural language processing. The model achieves state-of-the-art performance on multiple benchmarks, including a 90.29% accuracy on a novel Persian religious question dataset, outperforming standard BERT by 0.59%.

ABSTRACT

This paper presents a deep neural architecture, for Natural Language Sentence Matching (NLSM) by adding a deep recursive encoder to BERT so called BERT with Deep Recursive Encoder (BERT-DRE). Our analysis of model behavior shows that BERT still does not capture the full complexity of text, so a deep recursive encoder is applied on top of BERT. Three Bi-LSTM layers with residual connection are used to design a recursive encoder and an attention module is used on top of this encoder. To obtain the final vector, a pooling layer consisting of average and maximum pooling is used. We experiment our model on four benchmarks, SNLI, FarsTail, MultiNLI, SciTail, and a novel Persian religious questions dataset. This paper focuses on improving the BERT results in the NLSM task. In this regard, comparisons between BERT-DRE and BERT are conducted, and it is shown that in all cases, BERT-DRE outperforms BERT. The BERT algorithm on the religious dataset achieved an accuracy of 89.70%, and BERT-DRE architectures improved to 90.29% using the same dataset.

Motivation & Objective

• To improve the performance of BERT in Natural Language Sentence Matching (NLSM) tasks by addressing its limitations in capturing complex semantic relationships.
• To design a deep recursive encoder that enhances BERT’s contextual representations through hierarchical modeling of sentence structure.
• To evaluate the model on both multilingual benchmarks and a newly collected Persian religious question dataset for real-world applicability.
• To analyze the impact of architectural components such as residual connections, number of LSTM layers, and LSTM units via ablation studies.
• To demonstrate the generalization potential of the model across low-resource and high-resource NLP tasks, including question-answering and entailment.

Proposed method

• Integrates BERT as a contextual embedding layer to provide rich initial representations for input sentence pairs.
• Applies a deep recursive encoder composed of three stacked bidirectional LSTM layers with residual connections to model long-range dependencies and hierarchical semantics.
• Employs an attention mechanism on top of the LSTM outputs to dynamically weigh important words and improve semantic representation quality.
• Uses a pooling layer combining average and max pooling to produce a robust, invariant sentence-level representation.
• Fine-tunes the entire model end-to-end on downstream NLSM tasks using cross-entropy loss.
• Conducts ablation studies to evaluate the contribution of residual connections, number of LSTM layers, and number of LSTM units.

Experimental results

Research questions

• RQ1 Can a deep recursive encoder with residual connections improve BERT’s performance on sentence matching tasks?
• RQ2 How does the number of LSTM layers and the number of hidden units affect the model’s performance across different NLSM benchmarks?
• RQ3 What is the impact of residual connections on training stability and performance in the recursive encoder module?
• RQ4 Does BERT-DRE generalize effectively to low-resource languages, as demonstrated on a newly collected Persian religious question dataset?
• RQ5 How does BERT-DRE compare to other SOTA models like MT-DNN and BERT-wwm on standard NLSM benchmarks?

Key findings

• BERT-DRE achieves 90.29% accuracy on the novel Persian religious question dataset, outperforming standard BERT’s 89.70%.
• On the MultiNLI benchmark, BERT-DRE achieves 87.34% F1-score on the test set, surpassing BERT’s 85.9% and outperforming several SOTA models.
• The ablation study shows that three Bi-LSTM layers with 128 units per layer and residual connections yield optimal performance across all datasets.
• Removing residual connections reduces performance significantly, indicating their critical role in training deep recursive networks.
• The model shows consistent improvements over BERT across all four benchmark datasets: SNLI, FarsTail, MultiNLI, and SciTail.
• The attention mechanism and pooling layer contribute to more robust and informative sentence representations, especially in handling semantic similarity in paraphrase and entailment tasks.