[Paper Review] Table-To-Text generation and pre-training with TabT5

Ewa Andrejczuk, Julian Martin Eisenschlos+3 more|arXiv (Cornell University)|Oct 17, 2022

Statistics Education and Methodologies4 citations

TL;DR

TabT5 is a T5-based encoder-decoder model that enhances table-to-text generation by incorporating row and column positional embeddings and pre-training via a denoising autoencoding objective on web tables. It achieves state-of-the-art results across multiple tasks, including a 15% improvement in sequence accuracy for spreadsheet formula prediction and 2.5% gains in BLEU and sequence accuracy for question answering and data-to-text generation.

ABSTRACT

Encoder-only transformer models have been successfully applied to different table understanding tasks, as in TAPAS (Herzig et al., 2020). A major limitation of these architectures is that they are constrained to classification-like tasks such as cell selection or entailment detection. We present TABT5, an encoder-decoder model that generates natural language text based on tables and textual inputs. TABT5 overcomes the encoder-only limitation by incorporating a decoder component and leverages the input structure with table specific embeddings and pre-training. TABT5 achieves new state-of-the-art results on several domains, including spreadsheet formula prediction with a 15% increase in sequence accuracy, QA with a 2.5% increase in sequence accuracy and data-to-text generation with a 2.5% increase in BLEU.

Motivation & Objective

• To address the limitation of encoder-only models in generating full-text outputs from tables, particularly for complex tasks requiring natural language generation.
• To develop a unified framework that leverages both tabular structure and textual context for improved sequence-to-sequence generation.
• To pre-train the model on large-scale web tables using a denoising objective to enhance understanding of table structure and content.
• To evaluate the model across diverse tasks, including formula prediction, question answering, and data-to-text generation, under zero-shot and few-shot settings.
• To ablate key components such as row/column embeddings and denoising pretraining to validate their contribution to performance.

Proposed method

• TabT5 linearizes table inputs row by row, encoding each cell with its row and column position using dedicated positional embeddings.
• The model uses a sequence-to-sequence architecture based on T5, with a shared encoder-decoder transformer architecture for end-to-end learning.
• It employs a denoising pre-training objective where 15% of table cells are masked and the decoder is trained to reconstruct the original content.
• The input representation includes component types (e.g., question, table header, cell) to distinguish semantic roles in the input sequence.
• The model is fine-tuned on downstream tasks such as formula prediction, question answering, and data-to-text generation using standard sequence generation loss.
• Ablation studies are conducted by removing row/column embeddings or the denoising pretraining objective to assess their impact on performance.

Experimental results

Research questions

• RQ1 Can an encoder-decoder transformer architecture with structured positional embeddings outperform encoder-only models in table-to-text generation tasks?
• RQ2 To what extent does denoising pretraining on web tables improve generalization across diverse table-to-text tasks?
• RQ3 How do row and column positional embeddings contribute to the model’s ability to understand and generate text from tabular data?
• RQ4 Does the model generalize well to low-resource or out-of-domain table formats, especially when fine-tuned with limited supervision?
• RQ5 What types of errors are most common in generated outputs, and how do they relate to the pretraining objective and model architecture?

Key findings

• TabT5 achieves a 15% relative improvement in sequence accuracy on the Enron spreadsheet formula prediction benchmark compared to prior SOTA methods.
• On the ToTTo question-answering dataset, TabT5-base achieves 71.61% sequence accuracy, outperforming the T5-base baseline by 2.21 percentage points.
• For the FinQA reasoning benchmark, TabT5-large achieves 70.79% program accuracy and 68.00% execution accuracy, significantly outperforming prior approaches.
• Ablation studies show that removing either row/column embeddings or the denoising pretraining objective leads to a drop in performance, confirming their critical role.
• Error analysis reveals that 55% of TabT5’s outputs are paraphrases of T5’s outputs, and 72.5% are acceptable (correct content with minor omissions), indicating strong fluency and coherence.
• A significant portion of hallucinations (50%) are mild and semantically similar to the ground truth, suggesting the model is biased toward generating plausible but incorrect named entities due to pretraining.