[Paper Review] Thread Reconstruction in Conversational Data using Neural Coherence Models

Dat Tien Nguyen, Shafiq Joty+2 more|arXiv (Cornell University)|Jul 24, 2017

Topic Modeling14 references3 citations

TL;DR

This paper proposes a neural coherence model using a convolutional neural network (CNN) with an extended entity grid representation to reconstruct thread structures in forum discussions. By scoring entire candidate tree structures globally rather than individual replies, the method achieves significant improvements—up to 57% relative accuracy gain—over strong baselines like All-previous and COS-sim on CNET forum data.

ABSTRACT

Discussion forums are an important source of information. They are often used to answer specific questions a user might have and to discover more about a topic of interest. Discussions in these forums may evolve in intricate ways, making it difficult for users to follow the flow of ideas. We propose a novel approach for automatically identifying the underlying thread structure of a forum discussion. Our approach is based on a neural model that computes coherence scores of possible reconstructions and then selects the highest scoring, i.e., the most coherent one. Preliminary experiments demonstrate promising results outperforming a number of strong baseline methods.

Motivation & Objective

• To address the challenge of reconstructing complex, interleaved conversation threads in discussion forums where traditional local edge classifiers fail to capture global coherence.
• To improve thread reconstruction by modeling the entire tree structure at once, rather than making independent predictions for each reply link.
• To adapt neural coherence models—originally designed for monologues—to the non-sequential, multi-branch nature of forum conversations.
• To enhance performance through a grid-based representation that encodes thread structure and enables end-to-end training with pairwise ranking.
• To demonstrate that global coherence modeling leads to more accurate and coherent thread structure predictions than local, edge-level methods.

Proposed method

• Extend the original grid representation from coherence modeling to encode thread structure by tracking entity mentions across posts and their hierarchical reply relationships.
• Represent each candidate thread tree as a grid where rows correspond to posts and columns to entities, with cell values indicating entity presence or salience.
• Train a convolutional neural network (CNN) on the grid representations using pairwise ranking loss to learn to score the coherence of entire candidate tree structures.
• Optimize the model to rank correct thread trees higher than incorrect ones by comparing positive and negative tree hypotheses during training.
• Use the highest-scoring tree from the model's output as the final predicted thread structure for a given discussion.
• Apply the model end-to-end to reconstruct full thread trees from raw, time-ordered forum posts without relying on explicit reply metadata.

Experimental results

Research questions

• RQ1 Can a neural coherence model trained on grid representations effectively reconstruct complex, multi-branch thread structures in forum discussions?
• RQ2 Does modeling the entire thread structure globally, rather than predicting reply links individually, lead to better performance in thread reconstruction?
• RQ3 How does the proposed method compare to strong local baselines such as All-previous, All-first, and cosine-similarity in terms of tree-level and edge-level accuracy?
• RQ4 To what extent do the model's predictions improve over trivial reply patterns (e.g., replying to the first or previous post) in real-world forum data?
• RQ5 What role does the extended entity grid representation play in capturing discourse coherence across interleaved sub-conversations in forums?

Key findings

• The Grid-CNN model achieved a tree-level accuracy of 26.40%, representing a 32% to 57% relative improvement over the strongest baselines, including All-previous and All-first.
• On the edge-level prediction task, Grid-CNN achieved an F1-score of 60.55 and accuracy of 66.12%, outperforming all baselines by 4% to 13% in F1 and 1% to 12% in accuracy.
• The model significantly reduced false positives on trivial reply patterns, suggesting it learns more nuanced coherence patterns beyond simple temporal proximity.
• Manual inspection revealed that most incorrect predictions were for trivial structures (e.g., All-previous or All-first), indicating the model may be biased by training data distribution.
• The performance gap between Grid-CNN and baselines like COS-sim (16.80% accuracy) highlights the importance of modeling global coherence over local similarity alone.
• The results demonstrate that global coherence modeling via neural networks on structured grid representations is a promising approach for thread reconstruction in conversational data.