[Paper Review] Towards Zero-Shot and Few-Shot Table Question Answering using GPT-3

Pragya Srivastava, Tanuja Ganu+1 more|arXiv (Cornell University)|Oct 31, 2022

Topic Modeling4 citations

TL;DR

This paper explores zero-shot and few-shot table question answering using GPT-3 without fine-tuning, demonstrating that GPT-3 can understand tabular data from serialized JSON representations and answer natural language questions. With prompt engineering using few-shot examples, accuracy improves significantly from 73.6% (zero-shot) to 86.8% on a novel dataset of newspaper infographic tables.

ABSTRACT

We present very early results on using GPT-3 to perform question answering on tabular data. We find that stock pre-trained GPT-3 is able to zero-shot learn the table structure from a serialized JSON array-of-arrays representation, and able to answer lookup queries and simple comparison questions in natural language without any fine-tuning. We further find that simple prompt engineering to include few-shot static Q&A examples significantly improves accuracy. Lastly, we find that intermixing passage text improves accuracy even further on heterogeneous data. We apply our approach on a novel dataset of simple tables in newspaper infographics with promising results. Overall, we find much cause for optimism in this basic approach.

Motivation & Objective

• To investigate whether GPT-3 can perform table question answering without fine-tuning by leveraging its in-context learning ability.
• To evaluate the effectiveness of prompt engineering with few-shot examples in improving GPT-3's performance on table QA tasks.
• To explore the impact of intermixing passage text on GPT-3's ability to answer questions involving heterogeneous data sources.
• To develop and release a novel dataset of simple tables from newspaper infographics for future research in table QA.
• To assess the potential of foundational models like GPT-3 for accessible data interpretation, especially for visually impaired users and K-6 education.

Proposed method

• Constructed prompts using serialized JSON representations of tables, unstructured passage text from OCR, and natural language questions.
• Applied unmodified pre-trained GPT-3 to perform zero-shot inference on table QA tasks without any fine-tuning.
• Enhanced prompts with few-shot synthetic Q&A examples from the same category (e.g., sports, weather) but unrelated to the target table.
• Generated synthetic Q&A pairs using a fine-tuned T5 model to include basic statistical queries (min, max, mean) over table columns.
• Used object detection (Faster R-CNN) and OCR (Azure Form Recognizer) to extract tables and passage text from newspaper infographic images.
• Classified tables into categories (weather, sports, financial, others) using fine-tuned RoBERTa for downstream prompt engineering consistency.

Experimental results

Research questions

• RQ1 Can GPT-3 perform zero-shot table question answering on tabular data without any fine-tuning?
• RQ2 To what extent does few-shot prompting improve GPT-3’s accuracy in answering table-based questions?
• RQ3 How does the inclusion of surrounding passage text affect GPT-3’s performance on heterogeneous table and text QA?
• RQ4 How does GPT-3’s zero-shot performance compare to fine-tuned models like TAPAS on a novel dataset of infographic tables?
• RQ5 Can synthetic few-shot examples effectively generalize to improve zero-shot performance on unseen table QA tasks?

Key findings

• GPT-3 achieved a zero-shot accuracy of 73.6% on a small test set of 12 infographic tables without any fine-tuning.
• Few-shot prompting with synthetic Q&A examples improved accuracy to 86.8%, significantly outperforming zero-shot and fine-tuned TAPAS (18.4%).
• GPT-3 demonstrated the ability to understand table structure from serialized JSON input and correctly answer lookup and comparison questions.
• The model successfully located relevant cells and passage text to answer questions, even when the table was embedded within unstructured text.
• Interleaving passage text with table data improved answer accuracy, suggesting contextual grounding enhances reasoning.
• Despite strong zero-shot performance, some failure cases—such as misidentifying the minimum value or incorrect player rankings—were resolved with few-shot prompting.