[Paper Review] Joint Effects of Context and User History for Predicting Online Conversation Re-entries

Xingshan Zeng, Jing Li+2 more|arXiv (Cornell University)|Jun 4, 2019

Complex Network Analysis Techniques25 references4 citations

TL;DR

This paper proposes a neural framework with bi-attention mechanisms to jointly model conversation context and user chatting history for predicting online re-entry behavior. It achieves an F1 score of 61.1 on Twitter, outperforming prior state-of-the-art methods by leveraging end-to-end learning of user interests and topic alignment in dialogue contexts.

ABSTRACT

As the online world continues its exponential growth, interpersonal communication has come to play an increasingly central role in opinion formation and change. In order to help users better engage with each other online, we study a challenging problem of re-entry prediction foreseeing whether a user will come back to a conversation they once participated in. We hypothesize that both the context of the ongoing conversations and the users' previous chatting history will affect their continued interests in future engagement. Specifically, we propose a neural framework with three main layers, each modeling context, user history, and interactions between them, to explore how the conversation context and user chatting history jointly result in their re-entry behavior. We experiment with two large-scale datasets collected from Twitter and Reddit. Results show that our proposed framework with bi-attention achieves an F1 score of 61.1 on Twitter conversations, outperforming the state-of-the-art methods from previous work.

Motivation & Objective

• To address the challenge of predicting whether users will re-engage in online conversations they previously joined.
• To investigate how conversation context and user chatting history jointly influence re-entry behavior.
• To develop an end-to-end neural framework that captures interactions between context and history without relying on handcrafted features.
• To evaluate the model’s performance across varying conversation lengths and compare it with human prediction ability.

Proposed method

• A three-layer neural framework is designed, with separate encoders for conversation context and user history, followed by a bi-attention mechanism to model cross-modality interactions.
• The bi-attention mechanism computes attention weights in both directions—context to history and history to context—enabling dynamic alignment of user interests with current discussion topics.
• The model integrates representations through concatenation, attention over dialogue turns, memory networks, and bi-attention, with end-to-end training using supervised learning.
• Four variants are evaluated: simple concatenation, BiLSTM with attention, BiLSTM with memory networks, and BiLSTM with bi-attention, with the latter showing best performance.
• The framework is trained and tested on two large-scale datasets from Twitter and Reddit, with hyperparameters tuned via cross-validation.
• A human evaluation study is conducted to compare model performance against two human annotators on the same re-entry prediction task.

Experimental results

Research questions

• RQ1 How do conversation context and user chatting history jointly influence a user’s likelihood of re-entering a discussion?
• RQ2 Can a neural model with end-to-end learning outperform traditional methods relying on handcrafted features for re-entry prediction?
• RQ3 Does the bi-attention mechanism effectively align user interests with evolving conversation topics across different contexts?
• RQ4 How does the model’s performance compare to human judgment in predicting first re-entry behavior?

Key findings

• The proposed model with bi-attention achieves an F1 score of 61.1 on the Twitter dataset, surpassing the previous state-of-the-art method (F1 = 57.0).
• The model consistently outperforms all baselines, including those using handcrafted features, across varying lengths of conversation context and user history.
• The bi-attention mechanism successfully learns meaningful representations by aligning user interests (from history) with relevant topics in current conversations, as shown by attention heatmaps.
• Human performance on the re-entry prediction task is marginally better than random guessing (25/50), while the model achieves 35 correct predictions out of 50 on Twitter, indicating superior learning from large-scale data.
• The model performs better on Reddit than on Twitter, likely due to more focused and formal discussion structures on Reddit, which aid in topic identification.