[Paper Review] Can ChatGPT Understand Too? A Comparative Study on ChatGPT and Fine-tuned BERT

Qihuang Zhong, Liang Ding+3 more|arXiv (Cornell University)|Feb 19, 2023

Topic ModelingComputer Science147 citations

TL;DR

The paper evaluates ChatGPT against fine-tuned BERT-style models on the GLUE benchmark, finding strong inference/reasoning performance but weaker paraphrase and similarity handling; advanced prompting can improve ChatGPT, sometimes surpassing RoBERTa-large on certain tasks.

ABSTRACT

Recently, ChatGPT has attracted great attention, as it can generate fluent and high-quality responses to human inquiries. Several prior studies have shown that ChatGPT attains remarkable generation ability compared with existing models. However, the quantitative analysis of ChatGPT's understanding ability has been given little attention. In this report, we explore the understanding ability of ChatGPT by evaluating it on the most popular GLUE benchmark, and comparing it with 4 representative fine-tuned BERT-style models. We find that: 1) ChatGPT falls short in handling paraphrase and similarity tasks; 2) ChatGPT outperforms all BERT models on inference tasks by a large margin; 3) ChatGPT achieves comparable performance compared with BERT on sentiment analysis and question-answering tasks. Additionally, by combining some advanced prompting strategies, we show that the understanding ability of ChatGPT can be further improved.

Motivation & Objective

• Assess ChatGPT's understanding capability using the GLUE benchmark across diverse NLU tasks.
• Compare ChatGPT with representative fine-tuned BERT-style models (BERT-base, BERT-large, RoBERTa-base, RoBERTa-large).
• Investigate the impact of prompting strategies on ChatGPT's performance (few-shot, zero-shot CoT, manual few-shot CoT).
• Identify task-specific strengths and weaknesses of ChatGPT in entailment, paraphrase, and similarity tasks.
• Explore potential limitations and failure cases of ChatGPT and propose avenues to improve understanding through prompting.

Proposed method

• Evaluate ChatGPT and four BERT-style baselines on GLUE tasks using task-specific prompts.
• Sample 25 dev-set instances per class (or 50 for STS-B) to create evaluation subsets.
• Use standard metrics per task (Accuracy, F1, Pearson/Spearman, MCC) as appropriate.
• Compare zero-shot ChatGPT with fine-tuned baselines to establish lower-bound understanding.
• Apply advanced prompting strategies (standard few-shot, zero-shot CoT, manual few-shot CoT) to assess gains.
• Analyze per-class results and provide case studies illustrating failures and strengths.

Experimental results

Research questions

• RQ1 Can ChatGPT achieve comparable understanding to base-sized BERT on GLUE tasks in a zero-shot setting?
• RQ2 How does ChatGPT perform on inference versus paraphrase/similarity tasks within GLUE?
• RQ3 Do advanced prompting strategies improve ChatGPT's understanding, and which strategy is most effective?
• RQ4 What are the notable failure modes of ChatGPT in NLU tasks, and how might prompting address them?
• RQ5 How close can prompting bring ChatGPT to or beyond RoBERTa-large on certain tasks?

Key findings

• ChatGPT shows strong performance on inference tasks but weaker results on paraphrase and similarity tasks, especially on negative samples.
• ChatGPT surpasses all BERT-style models on some inference tasks, illustrating notable reasoning ability.
• With advanced prompting, ChatGPT can significantly close the gap to RoBERTa-large and even beat it on certain tasks, though gaps remain on average performance.
• Zero-shot ChatGPT achieves comparable performance to base RoBERTa when enhanced with prompting strategies, but still lags behind the strongest models overall.
• Manual few-shot chain-of-thought prompting yields the largest performance gains among the tested prompting methods.
• Performance is sensitive to one-shot prompting exemplars, indicating the importance of exemplar relevance and similarity to test data.