[Paper Review] Can ChatGPT Detect Intent? Evaluating Large Language Models for Spoken Language Understanding

Mutian He, Philip N. Garner|arXiv (Cornell University)|May 22, 2023

Topic Modeling4 citations

TL;DR

This paper evaluates large language models, particularly ChatGPT and GPT3.5, for spoken language understanding (SLU) using zero-shot and in-context prompting on intent classification and slot filling tasks. It finds that the largest models achieve near-supervised performance on intent classification with oracle transcripts, but struggle significantly with slot filling and are highly sensitive to ASR errors, revealing limitations for real-world SLU deployment.

ABSTRACT

Recently, large pretrained language models have demonstrated strong language understanding capabilities. This is particularly reflected in their zero-shot and in-context learning abilities on downstream tasks through prompting. To assess their impact on spoken language understanding (SLU), we evaluate several such models like ChatGPT and OPT of different sizes on multiple benchmarks. We verify the emergent ability unique to the largest models as they can reach intent classification accuracy close to that of supervised models with zero or few shots on various languages given oracle transcripts. By contrast, the results for smaller models fitting a single GPU fall far behind. We note that the error cases often arise from the annotation scheme of the dataset; responses from ChatGPT are still reasonable. We show, however, that the model is worse at slot filling, and its performance is sensitive to ASR errors, suggesting serious challenges for the application of those textual models on SLU.

Motivation & Objective

• To assess whether large language models like ChatGPT can perform spoken language understanding (SLU) tasks without fine-tuning.
• To investigate the effectiveness of zero-shot and in-context prompting on intent classification and slot filling across multiple languages.
• To evaluate the robustness of these models when faced with automatic speech recognition (ASR) errors.
• To identify failure modes and limitations of LLMs in SLU, particularly due to annotation scheme complexity and label ambiguity.
• To explore whether the models' actual understanding exceeds their performance on intermediate SLU tasks like intent and slot labeling.

Proposed method

• The study evaluates GPT3.5, ChatGPT, GPT2, and OPT models on SLURP and MINDS-14 benchmarks using zero-shot and few-shot prompting.
• Oracle transcripts are used to isolate language understanding from ASR errors, while Whisper ASR outputs are used to assess robustness to real-world transcription errors.
• Prompts are designed with task descriptions and in-context examples to guide intent classification and slot filling.
• Error analysis is conducted on 100 misclassified examples to assess whether misclassifications stem from model misunderstanding or dataset annotation issues.
• The models are evaluated under both fixed prompting and conversational setups to assess their ability to request clarification or adapt dynamically.
• The study compares performance across model sizes, highlighting emergent capabilities in the largest models.

Experimental results

Research questions

• RQ1 Can large language models like ChatGPT achieve high intent classification accuracy on SLU benchmarks using only prompting, without any fine-tuning?
• RQ2 How does model size affect zero-shot and in-context learning performance on intent classification and slot filling?
• RQ3 To what extent do ASR errors degrade the performance of LLMs in SLU tasks when using real ASR transcripts?
• RQ4 Why do error cases in slot filling often arise from annotation scheme complexity rather than model misunderstanding?
• RQ5 Can the model's actual understanding be better than its performance on intermediate SLU tasks like intent and slot labeling?

Key findings

• The largest models, including ChatGPT and GPT3.5, achieve intent classification accuracy close to that of supervised models when given oracle transcripts, demonstrating emergent zero-shot and in-context learning capabilities.
• Smaller models like GPT2 and OPT perform significantly worse, with near-random performance in zero-shot settings, indicating that emergent abilities are specific to large-scale models.
• For slot filling, performance is substantially worse than for intent classification, with high error rates due to overlapping and counter-intuitive label definitions in the dataset.
• Model performance drops significantly when using ASR transcripts instead of oracle transcripts, indicating high sensitivity to ASR errors and limited awareness of pronunciation or phonetic variations.
• Many 'errors' in intent classification are not true misunderstandings; 79% of 100 analyzed error cases are contextually reasonable, often due to sentence ambiguity or dataset annotation inconsistencies.
• The models show a tendency to overfit to patterns in few-shot examples, such as labeling 'Definition word' based on limited examples, even when inappropriate.