[论文解读] Table-GPT: Table-tuned GPT for Diverse Table Tasks

Peng Li, Yeye He+7 more|arXiv (Cornell University)|Oct 13, 2023

Topic Modeling被引用 8

一句话总结

Table-GPT 继续在多样化的合成表格任务上对 GPT-3.5/ChatGPT 进行训练，以提升表格理解能力并推广到未见表格任务。

ABSTRACT

Language models, such as GPT-3.5 and ChatGPT, demonstrate remarkable abilities to follow diverse human instructions and perform a wide range of tasks. However, when probing language models using a range of basic table-understanding tasks, we observe that today's language models are still sub-optimal in many table-related tasks, likely because they are pre-trained predominantly on \emph{one-dimensional} natural-language texts, whereas relational tables are \emph{two-dimensional} objects. In this work, we propose a new "\emph{table-tuning}" paradigm, where we continue to train/fine-tune language models like GPT-3.5 and ChatGPT, using diverse table-tasks synthesized from real tables as training data, with the goal of enhancing language models' ability to understand tables and perform table tasks. We show that our resulting Table-GPT models demonstrate (1) better \emph{table-understanding} capabilities, by consistently outperforming the vanilla GPT-3.5 and ChatGPT, on a wide-range of table tasks, including holdout unseen tasks, and (2) strong \emph{generalizability}, in its ability to respond to diverse human instructions to perform new table-tasks, in a manner similar to GPT-3.5 and ChatGPT.

研究动机与目标

• 在二维关系表与一维自然语言文本存在差异的前提下，激发对更好表格理解的需求。
• 提出一种表格微调范式，通过权重微调而非任务特定提示来提升语言模型在表格任务上的能力。
• 开发从真实表格中合成的多样化表格任务，以训练表格微调模型。
• 证明 Table-GPT 在已见与未见的表格任务上均优于原生 GPT-3.5/ChatGPT，并能对新指令进行泛化。

提出的方法

• 引入一种类指令微调的表格微调范式，在 (instruction, table, completion) 三元组上进行训练。
• 从真实表格中合成多样化的表格任务，创建丰富的训练语料库（合成-再增强）。
• 定义一个包含 18 个表格相关任务（T-1 到 T-18）的集合，覆盖表格理解、问答、匹配、清洗与变换等方面。
• 应用任务级、表格级、指令级和完成级的数据增强，以防止过拟合并提升泛化能力。
• 证明 Table-GPT 比原生 GPT 在下游微调和提示工程中具有更好的起点。

Figure 1 . Two simple tests to probe language-models’ basic ability to read and understand tables. (Left) T-1: Missing cells identification, which is to identify the column-header/row-id of a missing cell. (Right) T-2: Column-Finding, which is to identify the column-name of a given value. Even large

实验结果

研究问题

• RQ1 与自然语言文本相比，大型语言模型是否能够可靠地读取和推理二维表格？
• RQ2 在多样化表格任务上继续训练（表格微调）是否会提升表格理解和任务性能，相较于原生 GPT-3.5/ChatGPT？
• RQ3 Table-GPT 模型是否对未见的表格任务具有泛化能力，且能遵循新的表格相关人类指令？
• RQ4 合成-再增强数据管线如何影响泛化能力以及对表格变体的鲁棒性？

主要发现

• Table-GPT 在广泛的表格任务上持续优于原生 GPT-3.5 与 ChatGPT。
• Table-GPT 展现出强泛化能力，能够处理新颖且未见过的表格任务，并具有类似于 GPT-3.5/ChatGPT 的指令遵循行为。
• 表格微调方法与提示工程互为补充，惠及原生和表格微调模型。
• 研究强调当前语言模型在二维表格结构与纵向读取方面的挑战，推动将表格微调作为解决方案。

Figure 2 . Example table-tasks, where the ability of language models to “read” tables vertically is important. (Left) T-3: Table Question-Answering. (Right) T-8: Data Imputation. More tasks like these are shown in Table 2 .