[Paper Review] TaBERT: Pretraining for Joint Understanding of Textual and Tabular Data

Pengcheng Yin, Graham Neubig+2 more|arXiv (Cornell University)|May 17, 2020

Topic Modeling54 references36 citations

TL;DR

TaBert pretrains a Transformer to jointly encode natural language and (semi-)structured tables, enabling improved semantic parsing over both text-to-SQL and weakly supervised table QA benchmarks.

ABSTRACT

Recent years have witnessed the burgeoning of pretrained language models (LMs) for text-based natural language (NL) understanding tasks. Such models are typically trained on free-form NL text, hence may not be suitable for tasks like semantic parsing over structured data, which require reasoning over both free-form NL questions and structured tabular data (e.g., database tables). In this paper we present TaBERT, a pretrained LM that jointly learns representations for NL sentences and (semi-)structured tables. TaBERT is trained on a large corpus of 26 million tables and their English contexts. In experiments, neural semantic parsers using TaBERT as feature representation layers achieve new best results on the challenging weakly-supervised semantic parsing benchmark WikiTableQuestions, while performing competitively on the text-to-SQL dataset Spider. Implementation of the model will be available at http://fburl.com/TaBERT .

Motivation & Objective

• Motivate the need for representations that jointly capture free-form NL and table schemas.
• Propose TaBert, a BERT-based pretraining framework that encodes both utterances and table structures through content-aware snapshots and vertical attention.
• Show that TaBert can be plugged into semantic parsers to improve performance on SQL-based and weakly supervised table QA tasks.

Proposed method

• TaBert linearizes table content into per-row sequences and uses a Transformer encoder to produce row-wise representations of utterances and cells.
• Content snapshots sample the most relevant rows (K) to the input utterance to handle large tables efficiently.
• Vertical self-attention layers cross-attend across rows for the same column to produce column representations.
• Column representations include both column name/type and sampled cell values (through a Masked Column Prediction and Cell Value Recovery objectives).
• Pretraining data consists of 26.6 million parallel table–NL examples drawn from English Wikipedia and the WDC WebTable corpus, with MLM for NL context and MCP/CVR objectives for schema/content alignment.
• TaBert is evaluated as a drop-in encoder for semantic parsers on two regimes: supervised text-to-SQL (Spider) and weakly supervised WikiTableQuestions.

Experimental results

Research questions

• RQ1 Can joint NL and tabular representations learned by TaBert improve semantic parsing over tables?
• RQ2 Does pretraining on aligned NL and tabular data transfer across both structured (SQL-based) and semi-structured (table QA) tasks?
• RQ3 What is the impact of content snapshots and vertical attention on encoding quality for table-driven QA?
• RQ4 How do TaBert variants (base vs large; different snapshot sizes) compare to BERT baselines on these tasks?

Key findings

• TaBert with large models and content snapshots achieves state-of-the-art single-model performance on WikiTableQuestions.
• TaBert provides competitive or improved exact-match accuracy on Spider compared to strong baselines, closing the gap with top systems.
• Content snapshots and vertical attention substantially improve performance over non-snapshot or single-row encodings, especially for TaBert.
• Pretraining on large parallel NL–table data yields better joint representations than using Bert alone.
• TaBert's representations are effective across both structured and semi-structured table QA paradigms.