[Paper Review] Zero-Shot Visual Slot Filling as Question Answering.

Larry Heck, Simon Heck|arXiv (Cornell University)|Nov 24, 2020

Multimodal Machine Learning Applications24 references4 citations

TL;DR

This paper proposes a zero-shot visual slot filling method by reformulating slot filling as question answering: it converts slot tags into natural language questions, uses a fine-tuned ALBERT model on SQuaD2 to extract answers from user utterances, and introduces a multi-task training approach for refinement. The method achieves F1 scores of 0.52–0.60 on zero-shot Visual Slot and ATIS datasets, demonstrating effective transfer learning without labeled training data.

ABSTRACT

This paper presents a new approach to visual zero-shot slot filling. The approach extends previous approaches by reformulating the slot filling task as Question Answering. Slot tags are converted to rich natural language questions that capture the semantics of visual information and lexical text on the GUI screen. These questions are paired with the user's utterance and slots are extracted from the utterance using a state-of-the-art ALBERT-based Question Answering system trained on the Stanford Question Answering dataset (SQuaD2). An approach to further refine the model with multi-task training is presented. The multi-task approach facilitates the incorporation of a large number of successive refinements and transfer learning across similar tasks. A new Visual Slot dataset and a visual extension of the popular ATIS dataset is introduced to support research and experimentation on visual slot filling. Results show F1 scores between 0.52 and 0.60 on the Visual Slot and ATIS datasets with no training data (zero-shot).

Motivation & Objective

• To address the challenge of zero-shot visual slot filling in GUI-based dialog systems where labeled training data is scarce.
• To improve generalization in visual slot filling by leveraging natural language question formulation to capture semantic and lexical cues from GUIs.
• To enable transfer learning across related tasks through a multi-task training framework that supports iterative refinement.
• To introduce a new Visual Slot dataset and a visual extension of the ATIS dataset to support research in visual slot filling.

Proposed method

• Reformulate each slot tag into a rich natural language question that captures visual semantics and GUI text.
• Pair each user utterance with these question templates and use a pre-trained ALBERT model fine-tuned on SQuaD2 for answer extraction.
• Apply multi-task training to jointly optimize for slot filling and auxiliary tasks, enabling transfer learning and iterative refinement.
• Introduce a new Visual Slot dataset and a visual extension of the ATIS dataset to evaluate zero-shot performance.
• Use the ALBERT-based QA system to predict span-based answers from utterances, treating slot filling as a reading comprehension task.
• Leverage the semantic richness of natural language questions to improve zero-shot generalization beyond traditional slot tagging.

Experimental results

Research questions

• RQ1 Can reformulating slot filling as a question-answering task improve zero-shot performance on visual GUIs?
• RQ2 How effective is multi-task training in refining visual slot filling models without labeled data?
• RQ3 To what extent can a pre-trained QA model generalize to visual slot filling when fine-tuned on a related QA dataset?
• RQ4 How does the use of natural language questions compare to direct slot tagging in zero-shot settings?
• RQ5 What is the impact of dataset design on zero-shot visual slot filling performance?

Key findings

• The proposed QA-based approach achieves F1 scores between 0.52 and 0.60 on the zero-shot Visual Slot and ATIS datasets, demonstrating strong generalization without any labeled training data.
• Multi-task training enables effective transfer learning and supports successive refinements, improving model robustness and performance.
• The use of natural language questions significantly enhances semantic alignment between GUI content and user utterances, improving slot detection.
• The introduced Visual Slot dataset and visual extension of ATIS provide valuable benchmarks for future zero-shot visual slot filling research.
• Fine-tuning the ALBERT model on SQuaD2 enables strong zero-shot transfer to visual slot filling, highlighting the effectiveness of pre-trained QA models in this context.
• The method outperforms traditional slot filling approaches in zero-shot scenarios by better capturing contextual and visual semantics.