[Paper Review] ChatGPT: Beginning of an End of Manual Linguistic Data Annotation? Use Case of Automatic Genre Identification

Taja Kuzman, Igor Mozetič+1 more|arXiv (Cornell University)|Mar 7, 2023

Text Readability and Simplification66 citations

TL;DR

The paper evaluates zero-shot ChatGPT for automatic genre identification and compares it to a fine-tuned X-GENRE model across English and Slovenian, showing ChatGPT can outperform fine-tuned models on unseen data and that prompt language affects performance.

ABSTRACT

ChatGPT has shown strong capabilities in natural language generation tasks, which naturally leads researchers to explore where its abilities end. In this paper, we examine whether ChatGPT can be used for zero-shot text classification, more specifically, automatic genre identification. We compare ChatGPT with a multilingual XLM-RoBERTa language model that was fine-tuned on datasets, manually annotated with genres. The models are compared on test sets in two languages: English and Slovenian. Results show that ChatGPT outperforms the fine-tuned model when applied to the dataset which was not seen before by either of the models. Even when applied on Slovenian language as an under-resourced language, ChatGPT's performance is no worse than when applied to English. However, if the model is fully prompted in Slovenian, the performance drops significantly, showing the current limitations of ChatGPT usage on smaller languages. The presented results lead us to questioning whether this is the beginning of an end of laborious manual annotation campaigns even for smaller languages, such as Slovenian.

Motivation & Objective

• Assess ChatGPT's zero-shot performance on automatic genre identification in English and Slovenian.
• Compare ChatGPT with a fine-tuned X-GENRE classifier trained on multilingual genre datasets.
• Investigate how prompt language influences ChatGPT's classification performance.
• Analyze model agreement and complementarity between ChatGPT and X-GENRE results.
• Discuss implications for manual annotation efforts in NLP research.

Proposed method

• Use EN-GINCO (English) and GINCO (Slovenian) datasets, each with 100 test instances, labeled under the X-GENRE schema.
• Fine-tune X-GENRE (XLM-RoBERTa) on ~1,700 manually annotated texts from CORE, FTD, and GINCO datasets and map to X-GENRE labels.
• Prompt ChatGPT in three scenarios: English prompt with English text, English prompt with Slovenian text, Slovenian prompt with Slovenian text.
• Extract ChatGPT predictions and explanations from outputs, evaluate against true labels under the X-GENRE schema.
• Compare micro F1, macro F1, and accuracy against X-GENRE; analyze disagreements to assess complementary strengths.

Figure 1: Comparison of differences in correct and incorrect predictions between ChatGPT and X-GENRE.

Experimental results

Research questions

• RQ1 Can ChatGPT perform zero-shot genre identification comparably to a fine-tuned multilingual classifier on unseen data?
• RQ2 How does ChatGPT's performance differ between English and Slovenian texts and when prompted in different languages?
• RQ3 What is the impact of prompting language on ChatGPT's genre classification accuracy?
• RQ4 Do ChatGPT and X-GENRE produce complementary predictions that could be merged for improved performance?

Key findings

• ChatGPT outperforms X-GENRE on the English EN-GINCO test set when the prompt is in English (micro F1 0.74, macro F1 0.66, accuracy 0.72 vs 0.67/0.61/0.67).
• On Slovenian data (GINCO), X-GENRE substantially outperforms ChatGPT when prompted in Slovenian with Slovenian text (micro F1 0.91, macro F1 0.91, accuracy 0.91 for X-GENRE; ChatGPT 0.68/0.56/0.68 for Slovenian prompt).
• ChatGPT's performance on Slovenian text is comparable to English when the prompt is in English, despite Slovenian being under-resourced; performance declines when prompted in Slovenian.
• ChatGPT's predictions are largely complementary to X-GENRE, with different majority errors, suggesting potential ensemble gains in specific use cases.