[Paper Review] Benchmarking Natural Language Understanding Services for building Conversational Agents

Xingkun Liu, Arash Eshghi+2 more|arXiv (Cornell University)|Mar 13, 2019

Topic Modeling2 references88 citations

TL;DR

The paper conducts a large-scale, cross-platform evaluation of four NLU services (Rasa, Dialogflow, LUIS, Watson) on a 25k-utterance, 21-domain dataset annotated with 64 intents and 54 entity types, comparing their intent and entity recognition performance. It finds Watson excels in intent but underperforms on entity recognition, while Dialogflow, LUIS, and Rasa show more balanced results.

ABSTRACT

We have recently seen the emergence of several publicly available Natural Language Understanding (NLU) toolkits, which map user utterances to structured, but more abstract, Dialogue Act (DA) or Intent specifications, while making this process accessible to the lay developer. In this paper, we present the first wide coverage evaluation and comparison of some of the most popular NLU services, on a large, multi-domain (21 domains) dataset of 25K user utterances that we have collected and annotated with Intent and Entity Type specifications and which will be released as part of this submission. The results show that on Intent classification Watson significantly outperforms the other platforms, namely, Dialogflow, LUIS and Rasa; though these also perform well. Interestingly, on Entity Type recognition, Watson performs significantly worse due to its low Precision. Again, Dialogflow, LUIS and Rasa perform well on this task.

Motivation & Objective

• Provide a wide-coverage, unbiased evaluation of popular NLU services for building conversational agents.
• Release a large annotated NLU dataset to enable benchmarking across multiple domains.
• Compare platform performance on intent classification and named entity recognition (NER).
• Identify strengths and weaknesses of each platform to guide developers in tool selection.

Proposed method

• Assemble a large, multi-domain dataset of 25,716 utterances annotated with 64 intents and 54 entity types across 21 domains.
• Crowdsource data via Amazon Mechanical Turk and manually correct annotation inconsistencies.
• Evaluate four NLU services (Rasa, Dialogflow, LUIS, Watson) using 10-fold cross-validation with platform-specific training configurations.
• Use micro-averaged precision, recall, and F1 for intents and entities, and perform pairwise t-tests to assess significance.

Experimental results

Research questions

• RQ1 How do the four NLU services compare in overall F1 when combining Intent and Entity Type tasks?
• RQ2 How do the services differ in precision, recall, and F1 for Intent classification versus Entity Type recognition?
• RQ3 What are the statistical differences in performance between Dialogflow, LUIS, Rasa, and Watson on intents and entities?
• RQ4 Does context or annotation nuance affect platform performance, particularly for entity recognition?
• RQ5 What dataset features (21 domains, 64 intents, 54 entities) reveal strengths/limitations of each service?

Key findings

• Watson achieves the highest Intent precision/recall/F1 among the four platforms (F1 = 0.882) but has significantly lower Entity F1 due to high false positives.
• Dialogflow, LUIS, and Rasa show similar overall performance with no significant difference in Intent F1 among them.
• Luis attains the top Entity F1 score (0.777) among the four services.
• Watson’s Entity F1 is significantly lower (0.657) with large effect sizes, driven by low precision.
• Overall, there is no clear winner when pooling intents and entities; Watson trades strong intent performance for weaker entity performance.
• The study notes that no evaluation of Watson’s Contextual Entity feature was performed at the time of this camera-ready version.
• The authors provide the released dataset and toolkit for public use to support ongoing benchmarking.