[Paper Review] Tree-Transformer: A Transformer-Based Method for Correction of Tree-Structured Data

Jacob Harer, Christopher Reale+1 more|arXiv (Cornell University)|Aug 1, 2019

Natural Language Processing TechniquesComputer Science42 references45 citations

TL;DR

Introduces Tree-Transformer, a Transformer variant that operates on tree-structured data for code and natural language correction, achieving strong gains over sequential models on code and competitive/leading results on GEC benchmarks.

ABSTRACT

Many common sequential data sources, such as source code and natural language, have a natural tree-structured representation. These trees can be generated by fitting a sequence to a grammar, yielding a hierarchical ordering of the tokens in the sequence. This structure encodes a high degree of syntactic information, making it ideal for problems such as grammar correction. However, little work has been done to develop neural networks that can operate on and exploit tree-structured data. In this paper we present the Tree-Transformer extemdash{} a novel neural network architecture designed to translate between arbitrary input and output trees. We applied this architecture to correction tasks in both the source code and natural language domains. On source code, our model achieved an improvement of $25\%$ $ ext{F}0.5$ over the best sequential method. On natural language, we achieved comparable results to the most complex state of the art systems, obtaining a $10\%$ improvement in recall on the CoNLL 2014 benchmark and the highest to date $ ext{F}0.5$ score on the AESW benchmark of $50.43$.

Motivation & Objective

• Motivate correction tasks on tree-structured data (code and natural language) where grammar provides rich syntactic information.
• Develop a Transformer-based architecture that directly processes trees via a Tree Convolution Block to capture parent and left-sibling context.
• Enable tree-to-tree correction by modeling generation of tree structures with depth-first ordering and no positional encoding.
• Demonstrate improvements over sequence-based models in code correction and competitive performance in grammatical error correction (GEC).
• Explore training strategies and regularization tailored to relatively small correction datasets.

Proposed method

• Replace Transformer feed-forward sublayers with a Tree Convolution Block (TCB) that combines node, parent, and left-sibling representations via a learned affine transform and non-linearity.
• Incorporate a top-down encoder/decoder with TCBs; enable generation of tree structure by treating siblings as a sequence with end-of-sequence tokens.
• Use depth-first ordering to ensure parent and left-sibling inputs are available during decoding and apply self-attention masked accordingly.
• Omit positional encoding because TCBs provide structural locality and to avoid overfitting on small datasets.
• Train with Adam optimizer, dropout, label smoothing, and beam search; leverage encoder-decoder attention to condition on input and a self-attention mechanism to condition on previously produced nodes.
• Extend training with pretraining on large monolingual data and denoising-autoencoder objectives for grammatical error correction (GEC).

Experimental results

Research questions

• RQ1 Can a Transformer-based architecture be effectively adapted to operate directly on tree-structured data for correction tasks?
• RQ2 Does incorporating a parent-sibling tree convolution block improve grammar-corrected outputs over sequence-based methods in code and language domains?
• RQ3 What training strategies (pretraining, denoising, beam search, regularization) best support tree-to-tree correction on small datasets?
• RQ4 How does the Tree-Transformer perform on code correction versus grammatical error correction benchmarks compared to state-of-the-art sequential models?
• RQ5 What role does depth-first node ordering and lack of positional encoding play in model performance and overfitting?

Key findings

• On code correction with SATE IV data, Tree-Transformer achieves 84.7 F0.5, outperforming 4-layer LSTM (51.7) and standard Transformer (63.5).
• In GEC (CoNLL 2014), Tree-Transformer shows higher recall than prior approaches but with lower precision, yielding 47.50 F0.5 without monolingual augmentation and improved variants with augmentations.
• On AESW, Tree-Transformer achieves the highest reported F0.5 score to date at 50.43, outperforming sequence-based Transformer (48.03).
• The model demonstrates favorable training efficiency due to batchable attention over tree structures, reducing training time from months to under a day on large datasets.
• The architecture avoids positional encoding, as TCBs and tree structure provide sufficient localization, reducing overfitting on small correction datasets.