[Paper Review] A Persian Benchmark for Joint Intent Detection and Slot Filling

Masoud Akbari, Amir Hossein Karimi+6 more|arXiv (Cornell University)|Mar 1, 2023

Topic Modeling5 citations

TL;DR

This paper introduces the first publicly available Persian benchmark for joint intent detection and slot filling, constructed by semi-automatically translating the English ATIS dataset into Persian. It evaluates state-of-the-art models on the new benchmark, showing that JointBERT achieves the best performance on Persian, while Co-Interactive Transformer performs best on English, highlighting the need for improved multilingual representations for low-resource languages like Persian.

ABSTRACT

Natural Language Understanding (NLU) is important in today's technology as it enables machines to comprehend and process human language, leading to improved human-computer interactions and advancements in fields such as virtual assistants, chatbots, and language-based AI systems. This paper highlights the significance of advancing the field of NLU for low-resource languages. With intent detection and slot filling being crucial tasks in NLU, the widely used datasets ATIS and SNIPS have been utilized in the past. However, these datasets only cater to the English language and do not support other languages. In this work, we aim to address this gap by creating a Persian benchmark for joint intent detection and slot filling based on the ATIS dataset. To evaluate the effectiveness of our benchmark, we employ state-of-the-art methods for intent detection and slot filling.

Motivation & Objective

• To address the lack of publicly available Persian NLP datasets for intent detection and slot filling.
• To create a high-quality, publicly accessible Persian benchmark based on the ATIS dataset.
• To evaluate state-of-the-art models on the new Persian benchmark for joint intent detection and slot filling.
• To provide a standardized baseline for future research in Persian Natural Language Understanding.
• To support the development of multilingual NLU systems by extending the ATIS benchmark to Persian.

Proposed method

• A semi-automatic translation pipeline was used to convert the original English ATIS dataset into Persian, preserving intent and slot annotations.
• The dataset was split into training, validation, and test sets while maintaining the same distribution and correspondence between English and Persian samples.
• Pre-trained multilingual models such as mBERT were fine-tuned for both intent detection and slot filling tasks.
• Sequence classification and sequence labeling heads were applied on top of contextual embeddings to perform joint prediction of intent and slots.
• Models including Co-Interactive Transformer, JointBERT, and CRF-based RNNs were trained and evaluated using standard metrics.
• Evaluation focused on accuracy for intent detection and F1-score for slot filling, with class-wise analysis to assess performance on low-frequency categories.

Experimental results

Research questions

• RQ1 How well do state-of-the-art models perform on a newly created Persian benchmark for joint intent detection and slot filling?
• RQ2 What are the performance differences between models on Persian versus English ATIS data?
• RQ3 How does class imbalance in intent and slot categories affect model generalization on the Persian dataset?
• RQ4 To what extent does the quality of multilingual embeddings impact performance on low-resource languages like Persian?
• RQ5 Can existing English-based NLU models be effectively adapted to Persian with minimal fine-tuning?

Key findings

• JointBERT achieved the highest intent detection accuracy on the Persian dataset, outperforming other models including Co-Interactive Transformer.
• The Co-Interactive Transformer achieved the best performance on the English ATIS dataset, with only a negligible difference from JointBERT.
• Performance on both tasks was lower for Persian than for English, primarily due to lower-dimensional embeddings (50D for Persian GloVe vs. 300D for English).
• Models showed significantly lower performance on rare intent and slot categories, with accuracy decreasing as class frequency decreased.
• The slot filling performance on Persian showed a strong correlation between label frequency and model F1-score, similar to intent detection.
• CNN-LSTM-CRF achieved second-best performance on Persian slot filling in single learning mode, indicating strong sequence modeling capability despite lower overall accuracy.