[Paper Review] Multi-lingual Intent Detection and Slot Filling in a Joint BERT-based Model

Giuseppe Castellucci, Valentina Bellomaria+2 more|arXiv (Cornell University)|Jul 5, 2019

Topic Modeling28 references55 citations

TL;DR

The paper presents Bert-Joint, a multilingual joint model built on BERT for simultaneous intent detection and slot filling, achieving state-of-the-art results on ATIS and SNIPS and extending to Italian via a new SLU dataset.

ABSTRACT

Intent Detection and Slot Filling are two pillar tasks in Spoken Natural Language Understanding. Common approaches adopt joint Deep Learning architectures in attention-based recurrent frameworks. In this work, we aim at exploiting the success of "recurrence-less" models for these tasks. We introduce Bert-Joint, i.e., a multi-lingual joint text classification and sequence labeling framework. The experimental evaluation over two well-known English benchmarks demonstrates the strong performances that can be obtained with this model, even when few annotated data is available. Moreover, we annotated a new dataset for the Italian language, and we observed similar performances without the need for changing the model.

Motivation & Objective

• Motivate joint language understanding tasks: improve efficiency and accuracy for intent detection and slot filling in SLU.
• Propose a recurrence-free, BERT-based joint framework to model sentence-level intents and token-level slots.
• Show that joint training with a combined loss improves performance under limited data and across languages.

Proposed method

• Extend BERT with a sentence-level intent classifier and a token-level slot labeler on top of shared representations.
• Use final hidden state of [CLS] for intent and final hidden state of each token for slot predictions.
• Train jointly with a combined loss L = alpha * L_c + beta * L_o, updating all BERT parameters plus classifier parameters.
• Fine-tune with dropout, Adam optimizer, and a small learning-rate schedule over 5–50 epochs, using standard SLU datasets for evaluation.

Experimental results

Research questions

• RQ1 Can a single BERT-based model jointly optimize intent detection and slot filling with a unified loss function?
• RQ2 Does joint modeling surpass separate INTENT-only and SLOT-only BERT baselines on standard English SLU benchmarks?
• RQ3 Is the joint approach effective in low-data regimes and with multilingual data, including Italian?
• RQ4 How does a multilingual training setup affect performance on English and Italian SLU datasets?

Key findings

• Bert-Joint achieves state-of-the-art results on ATIS and SNIPS for sentence accuracy and F1 of slots, outperforming separate Bert-Intent and Bert-Slot baselines.
• Joint training provides notable gains in both ID and SF, with substantial improvements in sentence-level accuracy (intent+slots).
• In low-data regimes, Bert-Joint shows larger relative gains over non-joint baselines, especially for intent detection.
• An Italian SLU dataset (SLU-IT) shows Bert-Joint maintaining strong performance and outperforming non-joint baselines in Italian.
• Multi-lingual training (EN+IT) yields improvements on Italian without significant degradation on English, and increases sentence-level accuracy.