[Paper Review] Adobe-MIT submission to the DSTC 4 Spoken Language Understanding pilot task

Franck Dernoncourt, Ji Young Lee+2 more|arXiv (Cornell University)|May 7, 2016

Speech and dialogue systems17 references3 citations

TL;DR

This paper presents a joint system for spoken language understanding in task-oriented dialogues, using SVM and logistic regression for speech act recognition and CRFs for semantic tagging. The best system achieved 0.67 F1-score on speech act recognition for the guide and 0.52 F1-score for semantic tagging across both roles in the DSTC 4 pilot task.

ABSTRACT

The Dialog State Tracking Challenge 4 (DSTC 4) proposes several pilot tasks. In this paper, we focus on the spoken language understanding pilot task, which consists of tagging a given utterance with speech acts and semantic slots. We compare different classifiers: the best system obtains 0.52 and 0.67 F1-scores on the test set for speech act recognition for the tourist and the guide respectively, and 0.52 F1-score for semantic tagging for both the guide and the tourist.

Motivation & Objective

• To address the spoken language understanding (SLU) pilot task in the Dialog State Tracking Challenge 4 (DSTC 4), focusing on speech act recognition and semantic tagging.
• To develop and compare multiple classification models for accurately identifying speech acts and semantic slots in spoken dialogues.
• To evaluate the effectiveness of speaker-dependent vs. speaker-independent models and feature engineering strategies in SLU.
• To design a CRF-based system for semantic tagging that incorporates linguistic features and ontology constraints.
• To achieve high performance on both speech act recognition and semantic tagging using structured prediction and feature engineering.

Proposed method

• Used a support vector machine (SVM) with 5000 most frequent unigrams, bigrams, trigrams, and speaker change features, trained separately per speaker for speech act recognition.
• Implemented a logistic regression model with a single speaker-independent classifier, improving performance over speaker-dependent models on the development set.
• Designed a rule-based baseline system (System 1) with around 10 handcrafted rules to predict speech acts based on utterance context.
• Applied conditional random fields (CRFs) using features from 7 consecutive words, including case, character suffixes, capitalization, digit presence, and part-of-speech tags.
• Trained four independent CRFs for main category, subcategory, relation, and from-to modifiers, with results combined using ontology constraints.
• Combined predictions by ensuring subcategory, relation, and from-to tags were only included if valid within the main category’s ontology.

Experimental results

Research questions

• RQ1 How do speaker-dependent and speaker-independent models compare in performance for speech act recognition in task-oriented dialogues?
• RQ2 What is the impact of incorporating utterance history and speaker identity features on speech act classification accuracy?
• RQ3 How effective are CRFs with rich linguistic features in capturing semantic slot boundaries and attributes?
• RQ4 Can ontology-constrained CRF fusion improve the consistency and accuracy of semantic tagging?
• RQ5 What is the contribution of simple rule-based systems as baselines in SLU tasks with limited training data?

Key findings

• System 5, using logistic regression with a speaker-independent model, achieved the highest F1-score of 0.6117 on the test set for speech act recognition in the tourist role.
• System 3, based on SVM with speaker-dependent models, achieved an F1-score of 0.6740 for the guide role, outperforming other systems in that setting.
• The semantic tagging system achieved an F1-score of 0.5239 for the guide and 0.5207 for the tourist, demonstrating consistent performance across roles.
• The use of speaker identity and utterance history features improved performance over baseline systems, with System 5 showing a 5.5% improvement in F1 over System 1 for the tourist role.
• The CRF-based semantic tagging system with 12 feature types and ontology-based fusion achieved a 0.52 F1-score, indicating strong generalization on semantic slot detection.
• Rule-based System 1 served as a weak baseline, achieving only 0.3252 F1 for the tourist role, highlighting the value of learning-based models in this task.