[Paper Review] A Corpus of Controlled Opinionated and Knowledgeable Movie Discussions for Training Neural Conversation Models

Fabian Galetzka, Chukwuemeka U. Eneh+1 more|arXiv (Cornell University)|Mar 30, 2020

Topic Modeling4 citations

TL;DR

This paper introduces KOMODIS, a new dialogue dataset featuring movie discussions with explicitly controlled knowledge and opinion profiles for each participant. Trained on this data, a self-attention decoder model generates natural, consistent, and opinionated responses that reflect both factual knowledge and stable personal stances, outperforming random distractor baselines in knowledgeability and personality consistency.

ABSTRACT

Fully data driven Chatbots for non-goal oriented dialogues are known to suffer from inconsistent behaviour across their turns, stemming from a general difficulty in controlling parameters like their assumed background personality and knowledge of facts. One reason for this is the relative lack of labeled data from which personality consistency and fact usage could be learned together with dialogue behaviour. To address this, we introduce a new labeled dialogue dataset in the domain of movie discussions, where every dialogue is based on pre-specified facts and opinions. We thoroughly validate the collected dialogue for adherence of the participants to their given fact and opinion profile, and find that the general quality in this respect is high. This process also gives us an additional layer of annotation that is potentially useful for training models. We introduce as a baseline an end-to-end trained self-attention decoder model trained on this data and show that it is able to generate opinionated responses that are judged to be natural and knowledgeable and show attentiveness.

Motivation & Objective

• To address inconsistency in end-to-end neural chatbots, particularly in knowledge retention and opinion stability during non-goal-oriented dialogues.
• To create a large-scale, high-quality dialogue dataset where each participant’s knowledge and opinions are explicitly controlled and annotated.
• To enable training of neural models that maintain global consistency in both factual knowledge and personal opinions across dialogue turns.
• To validate that crowd workers adhere strictly to their assigned knowledge and opinion profiles during dialogue creation.
• To demonstrate that models trained on this data can generate responses that are judged as natural, knowledgeable, and opinion-consistent by human evaluators.

Proposed method

• The dataset was collected via Amazon Mechanical Turk, with each dialogue pair assigned unique, pre-defined knowledge facts and opinion profiles about movies and actors.
• Knowledge facts were extracted from IMDb and include trivia such as character relationships or director quotes.
• Opinion profiles were created to include stance toward entities (e.g., 'loves the director', 'dislikes the lead actor'), with multiple opinions per entity possible.
• Each dialogue was validated by human raters on adherence to knowledge and opinion profiles, ensuring high consistency.
• A baseline end-to-end self-attention decoder model was trained on the dataset using two types of negative sampling: random and rule-based distractors.
• The model was evaluated via human annotation (on naturalness, consistency, personality, and knowledgeability) and automated metrics (perplexity, hits@n).

Experimental results

Research questions

• RQ1 Can a neural dialogue model be trained to maintain consistent opinions over time when generating responses in non-goal-oriented conversations?
• RQ2 To what extent can a model learn to incorporate and retain factual knowledge from a controlled, profile-based dataset?
• RQ3 Does the inclusion of explicit opinion and knowledge annotations improve model consistency and perceived naturalness in dialogue generation?
• RQ4 How do rule-based vs. random negative sampling strategies affect model performance in knowledge and opinion consistency?
• RQ5 Can automated metrics like hits@n correlate with human judgments of model quality in this controlled setting?

Key findings

• Human evaluation showed high agreement (4.55 out of 5) on personality consistency, confirming that crowd workers adhered well to their assigned opinion profiles.
• The model trained with rule-based distractors achieved significantly better performance in personality and knowledgeability (hits@1: 79.33) compared to random distractors (hits@1: 74.22).
• Despite higher perplexity, the rule-based model outperformed the random one in human evaluation, indicating that automated metrics like perplexity do not always align with human perception.
• The hits@n metric showed strong correlation with human evaluation, suggesting it is a reliable proxy for model quality in this context.
• The model successfully generated natural, opinionated, and knowledge-aware responses, though it struggled with topic shifts outside the movie domain.
• The results confirm that explicit labeling of knowledge and opinions enables training of more consistent and engaging neural dialogue systems.