[Paper Review] Syntax-BERT: Improving Pre-trained Transformers with Syntax Trees

Jiangang Bai, Yujing Wang+5 more|arXiv (Cornell University)|Mar 7, 2021

Topic Modeling27 references4 citations

TL;DR

Syntax-BERT proposes a plug-and-play framework that enhances pre-trained Transformers like BERT, RoBERTa, and T5 by integrating syntax trees through decomposed, relationship-specific self-attention sub-networks (ancestor, offspring, sibling) with shared parameters. It uses a topical attention layer to dynamically weight these sub-networks, achieving consistent performance gains across GLUE tasks—boosting T5-Large’s score from 86.3 to 86.8 with minimal parameter increase.

ABSTRACT

Pre-trained language models like BERT achieve superior performances in various NLP tasks without explicit consideration of syntactic information. Meanwhile, syntactic information has been proved to be crucial for the success of NLP applications. However, how to incorporate the syntax trees effectively and efficiently into pre-trained Transformers is still unsettled. In this paper, we address this problem by proposing a novel framework named Syntax-BERT. This framework works in a plug-and-play mode and is applicable to an arbitrary pre-trained checkpoint based on Transformer architecture. Experiments on various datasets of natural language understanding verify the effectiveness of syntax trees and achieve consistent improvement over multiple pre-trained models, including BERT, RoBERTa, and T5.

Motivation & Objective

• To investigate whether explicit syntax tree integration improves pre-trained Transformer models in the pre-training phase.
• To address the challenge of efficiently incorporating syntactic structures into existing pre-trained checkpoints without retraining from scratch.
• To develop a method that preserves pre-trained knowledge while selectively leveraging syntactic inductive bias for downstream tasks.
• To enable task-adaptive selection of syntactic relationships through a lightweight, learnable attention mechanism.
• To demonstrate generalizability across diverse pre-trained models, including BERT, RoBERTa, and T5.

Proposed method

• Decompose the standard self-attention mechanism into multiple sub-networks based on syntactic tree relationships: ancestor, offspring, and sibling with varying hop counts.
• Share all sub-network parameters with the pre-trained model to enable direct transfer from existing checkpoints.
• Introduce a topical attention layer that computes task-specific weights for each syntactic sub-network based on input-dependent relevance.
• Aggregate representations from all sub-networks via weighted summation using the topical attention scores.
• Use parent, sibling, and ancestor masks derived from dependency trees to define the attention masks for each sub-network.
• Apply the framework in a plug-and-play manner to any pre-trained Transformer checkpoint, avoiding retraining from scratch.

Experimental results

Research questions

• RQ1 Can explicit syntax tree integration improve the performance of pre-trained Transformers in the pre-training phase?
• RQ2 Is it possible to inject syntactic inductive bias into a pre-trained model efficiently without retraining from scratch?
• RQ3 How do different syntactic relationships (e.g., ancestor, sibling) contribute to downstream task performance?
• RQ4 Can the framework be generalized across diverse pre-trained models such as BERT, RoBERTa, and T5?
• RQ5 Does the topical attention mechanism effectively identify and prioritize the most relevant syntactic representations for a given task?

Key findings

• Syntax-BERT improves the overall GLUE benchmark score of T5-Large from 86.3 to 86.8, demonstrating consistent gains with minimal parameter overhead.
• On the structural probe, Syntax-BERT-Large achieves a UUAS of 83.4 and Spearman correlation of 0.90, outperforming BERT-Large (79.8 UUAS, 0.86 correlation).
• Syntax-RoBERTa-Large achieves a UUAS of 84.6 and Spearman correlation of 0.93, showing significant syntactic knowledge enhancement over RoBERTa-Large.
• The case study shows Syntax-BERT correctly identifies grammatical errors (e.g., "anyone" with period) by highlighting relevant syntactic relationships, while vanilla BERT fails.
• The framework maintains the same time complexity as BERT when the number of sub-networks is small, and sparse implementation further reduces complexity to O(MD_Q E).
• Ablation studies confirm that the decomposition and topical attention components are essential, as removing them leads to performance drops, validating the design’s effectiveness.