[Paper Review] A Response Retrieval Approach for Dialogue Using a Multi-Attentive Transformer

Matteo Antonio Senese, Alberto Benincasa+2 more|arXiv (Cornell University)|Dec 15, 2020

Topic Modeling4 citations

TL;DR

This paper proposes a multi-attentive Transformer model for response retrieval in goal-oriented, multimodal dialogue systems, conditioning responses on user requests and referred products. The model achieves second-best performance on all retrieval metrics (MRR, R@1, R@5, R@10, Mean Rank) on the SIMMC Fashion dataset, outperforming baselines by up to 43.9% on R@10 and 27.7% on MRR with the semantic score integration.

ABSTRACT

This paper presents our work for the ninth edition of the Dialogue System Technology Challenge (DSTC9). Our solution addresses the track number four: Simulated Interactive MultiModal Conversations. The task consists in providing an algorithm able to simulate a shopping assistant that supports the user with his/her requests. We address the task of response retrieval, that is the task of retrieving the most appropriate agent response from a pool of response candidates. Our approach makes use of a neural architecture based on transformer with a multi-attentive structure that conditions the response of the agent on the request made by the user and on the product the user is referring to. Final experiments on the SIMMC Fashion Dataset show that our approach achieves the second best scores on all the retrieval metrics defined by the organizers. The source code is available at https://github.com/D2KLab/dstc9-SIMMC.

Motivation & Objective

• To develop a response retrieval system for situated, multimodal dialogues in a shopping assistant scenario.
• To address the challenge of grounding responses in both user requests and specific product attributes from a catalog.
• To improve retrieval performance by conditioning the model on multiple contextual inputs: user request, referred product, and response candidates.
• To evaluate the model on the SIMMC Fashion dataset, a benchmark for interactive, multimodal, goal-oriented dialogues.
• To demonstrate the effectiveness of multi-attentive mechanisms in enhancing response relevance and retrieval accuracy.

Proposed method

• The model uses a BERT-based encoder with frozen parameters to encode user requests and product attributes.
• A single-layer Transformer decoder with three parallel attention heads is employed: self-attention over generated tokens, cross-attention over the user request, and cross-attention over the referred product.
• The decoder attends to both the user request and the product embedding to condition the response generation on relevant context.
• A semantic score is computed by comparing the candidate response with the user request and product attributes, enhancing retrieval accuracy.
• The final response score is computed as a weighted sum of the decoder’s cross-attention scores and the semantic score.
• The model is fine-tuned end-to-end on the SIMMC Fashion dataset using a contrastive learning objective to maximize the rank of the correct response.

Experimental results

Research questions

• RQ1 Can a multi-attentive Transformer architecture effectively retrieve the most appropriate response in a multimodal, goal-oriented dialogue setting?
• RQ2 How does conditioning the response retrieval on both the user request and the referred product improve retrieval performance?
• RQ3 What is the impact of integrating a semantic score based on request and product embeddings on retrieval metrics?
• RQ4 How does the proposed model compare to strong baselines and state-of-the-art methods on the SIMMC Fashion benchmark?
• RQ5 To what extent does the multi-attention mechanism enhance contextual grounding and response relevance?

Key findings

• On the official test-std split, the model achieved the second-best Mean Reciprocal Rank (MRR) of 0.390, outperforming the baseline by 27.7%.
• The model achieved a recall at 1 (R@1) of 26.7%, representing a 22.2% improvement over the baseline on the test-std set.
• On the test-dev split, the model achieved an MRR of 0.419 with the semantic score, a 2.1% improvement over the version without it.
• The model achieved a recall at 5 (R@5) of 55.9% on test-dev and 52.1% on test-std, significantly outperforming the baseline.
• The Mean Rank metric improved from 27.4 (baseline) to 13.6 (ours) on test-dev, indicating a substantial reduction in average ranking of the correct response.
• The model achieved the second-best overall performance on all five evaluation metrics (MRR, R@1, R@5, R@10, Mean Rank) across both test splits.