[Paper Review] Interpreting the Syntactic and Social Elements of the Tweet Representations via Elementary Property Prediction Tasks

J Ganesh, Manish Gupta+1 more|arXiv (Cornell University)|Nov 15, 2016

Topic Modeling10 references3 citations

TL;DR

This paper introduces a set of elementary property prediction tasks to interpret tweet representations by probing syntactic (e.g., length, word order, slang, hashtags, named entities) and social (e.g., reply status, reply time) properties. Using 9 unsupervised and 4 supervised models, it finds that recurrent models like BLSTM and LSTM best capture both syntactic and social features, while BOW surprisingly excels at word order prediction despite losing sequential information.

ABSTRACT

Research in social media analysis is experiencing a recent surge with a large number of works applying representation learning models to solve high-level syntactico-semantic tasks such as sentiment analysis, semantic textual similarity computation, hashtag prediction and so on. Although the performance of the representation learning models are better than the traditional baselines for the tasks, little is known about the core properties of a tweet encoded within the representations. Understanding these core properties would empower us in making generalizable conclusions about the quality of representations. Our work presented here constitutes the first step in opening the black-box of vector embedding for social media posts, with emphasis on tweets in particular. In order to understand the core properties encoded in a tweet representation, we evaluate the representations to estimate the extent to which it can model each of those properties such as tweet length, presence of words, hashtags, mentions, capitalization, and so on. This is done with the help of multiple classifiers which take the representation as input. Essentially, each classifier evaluates one of the syntactic or social properties which are arguably salient for a tweet. This is also the first holistic study on extensively analysing the ability to encode these properties for a wide variety of tweet representation models including the traditional unsupervised methods (BOW, LDA), unsupervised representation learning methods (Siamese CBOW, Tweet2Vec) as well as supervised methods (CNN, BLSTM).

Motivation & Objective

• To understand what core syntactic and social properties are encoded in tweet representations, moving beyond high-level task performance.
• To address the opacity of neural tweet representation models by introducing fine-grained, application-agnostic probing tasks.
• To compare traditional, unsupervised, and supervised representation learning models across multiple intrinsic properties of tweets.
• To investigate how model architecture and training objective affect the encoding of structural and social features in short, informal text.

Proposed method

• Define eight elementary property prediction tasks: tweet length, content presence, word order, slang normalization, hashtag detection, named entity recognition, reply status, and reply time.
• Use a two-layer neural network classifier (representation + softmax) to predict each property from the tweet representation.
• For each task, apply negative sampling or binning strategies to construct balanced training data (e.g., binning tweet length into groups of 4 words).
• Concatenate embeddings when multiple inputs are involved (e.g., for word order, concatenate representations of two words).
• Evaluate 13 models: 9 unsupervised (BOW, LDA, BOM, DSSM, CDSSM, PV, STV, SCBOW, T2V) and 4 supervised (CNN, LSTM, BLSTM, FT).
• Use F1-score to compare performance across models and tasks, analyzing sensitivity to word order and tweet length.

Experimental results

Research questions

• RQ1 Which syntactic and social properties are effectively encoded in different tweet representation models?
• RQ2 How do supervised versus unsupervised models differ in their ability to preserve elementary tweet properties?
• RQ3 To what extent is word order preserved in different representation models, and how does this affect performance on order-sensitive tasks?
• RQ4 How does tweet length influence the prediction accuracy of elementary properties across different models?
• RQ5 Can performance on elementary property tasks predict the suitability of a model for downstream social media applications like user profiling or conversation modeling?

Key findings

• BLSTM achieves the highest F1-score (92.76%) on the 'Named Entity' task, outperforming even specialized models like T2V.
• LSTM and BLSTM achieve F1-scores above 99% on 'Length' and 'Is Reply' tasks, indicating strong encoding of tweet length and social structure.
• BOW performs surprisingly well on 'Word Order' (99.79% F1), suggesting strong correlation between word frequency patterns and sequential structure.
• LDA is invariant to word order, showing near-zero performance on 'Word Order' (F1 < 10), as expected from its bag-of-words assumption.
• CNN, LSTM, and BLSTM show high sensitivity to word order, with performance dropping significantly when word order is reversed.
• Models like STV and T2V perform poorly on most tasks, with T2V underperforming even baseline models, likely due to overfitting to hashtag prediction during pretraining.