[Paper Review] Learning to mirror speaking styles incrementally

Siyi Liu, Ziang Leng+1 more|arXiv (Cornell University)|Mar 5, 2020

Natural Language Processing Techniques13 references4 citations

TL;DR

This paper proposes an incremental method for learning and mirroring individual speaking styles using frequent n-grams extracted via efficient itemset mining. It combines BERT-based pattern injection and seq2seq modeling to transform sentences into target styles, showing superior performance with limited data using explicit pattern injection, while seq2seq excels with larger datasets, enabling dynamic, human-like style mirroring in chatbots.

ABSTRACT

Mirroring is the behavior in which one person subconsciously imitates the gesture, speech pattern, or attitude of another. In conversations, mirroring often signals the speakers enjoyment and engagement in their communication. In chatbots, methods have been proposed to add personas to the chatbots and to train them to speak or to shift their dialogue style to that of the personas. However, they often require a large dataset consisting of dialogues of the target personalities to train. In this work, we explore a method that can learn to mirror the speaking styles of a person incrementally. Our method extracts ngrams that capture a persons speaking styles and uses the ngrams to create patterns for transforming sentences to the persons speaking styles. Our experiments show that our method is able to capture patterns of speaking style that can be used to transform regular sentences into sentences with the target style.

Motivation & Objective

• To develop a method that learns individual speaking styles incrementally from growing conversational data, unlike static models requiring large pre-labeled datasets.
• To address the limitation of existing persona and style transfer models that focus on domain-level or profile-based personas rather than individual linguistic patterns.
• To enable chatbots to dynamically adapt their style to mirror users’ speech patterns in real time, enhancing engagement and naturalness in human-computer interaction.
• To design a scalable, efficient approach that captures stylistic features like fillers, repetition, and intensifiers without relying on manual heuristics or large-scale pre-training.

Proposed method

• Extracts frequent n-grams from a speaker’s utterances using incremental frequent itemset mining algorithms, which efficiently identify recurring linguistic patterns.
• Uses BERT to encode sentence representations and computes cosine similarity between input and transformed sentences to preserve semantic context during style transfer.
• Applies explicit pattern injection by selecting the most contextually similar and stylistically aligned n-gram patterns from the mined set to insert into input sentences.
• Trains a seq2seq model with attention to jointly learn pattern selection and sentence transformation, improving fluency when sufficient training data is available.
• Combines pattern-based transformation with neural generation, allowing the system to adapt its strategy based on data availability—pattern injection for low-data, seq2seq for high-data settings.
• Employs a two-stage evaluation: perplexity for fluency and cosine similarity for context preservation, using weighted average word vectors from BERT.

Experimental results

Research questions

• RQ1 Can a model learn and reflect individual speaking styles incrementally as more conversational data becomes available?
• RQ2 How does explicit pattern injection using frequent n-grams compare to end-to-end seq2seq learning in style transfer performance with limited training data?
• RQ3 To what extent can BERT-based pattern injection maintain input sentence context while transforming it into a target speaker’s style?
• RQ4 Does the performance of the model improve with increasing amounts of conversational data, and does the optimal strategy shift from pattern injection to neural generation?
• RQ5 Can the method effectively capture and reproduce idiosyncratic stylistic features such as fillers, repetition, and intensifiers found in individual speakers like Donald Trump?

Key findings

• With only 5% of the data, the BERT-based explicit pattern injection method achieved a perplexity of 27.5 and a cosine similarity of 0.98, significantly outperforming the seq2seq model, which had a perplexity of 11.7 but a low similarity of 0.68.
• As data increased to 20%, the BERT+pattern method maintained high similarity (0.98) and moderate perplexity (22.5), while the seq2seq model’s similarity improved to 0.74 but its perplexity rose to 24.2.
• The seq2seq model produced incoherent, repetitive outputs with low data, evidenced by high fluency (low perplexity) but poor context preservation, indicating memorization of training examples.
• The BERT+explicit pattern method preserved input context more effectively across all data settings, as shown by consistently high cosine similarities (0.98 at 5%, 10%, and 20% data).
• At full data (100%), the seq2seq model’s perplexity increased to 36.7, indicating reduced fluency, while its similarity improved to 0.85, suggesting better context retention but worse overall performance due to overfitting or instability.
• The results confirm the hypothesis that explicit pattern injection is more effective than end-to-end learning when data is scarce, while seq2seq performs better with sufficient data, validating the hybrid strategy.