[Paper Review] Characterizing English Variation across Social Media Communities with BERT

Li Lucy, David Bamman|arXiv (Cornell University)|Feb 12, 2021

Digital Communication and Language74 references4 citations

TL;DR

This paper proposes a BERT-based method to detect both lexical and semantic variation in English language use across 474 Reddit communities, showing that communities with highly distinctive language—especially in word senses—are medium-sized and feature loyal, highly engaged users in dense interaction networks. The approach identifies community-specific language by combining word type specificity with sense-level embeddings, validating findings using user-created glossaries and sociolinguistic theory.

ABSTRACT

Much previous work characterizing language variation across Internet social groups has focused on the types of words used by these groups. We extend this type of study by employing BERT to characterize variation in the senses of words as well, analyzing two months of English comments in 474 Reddit communities. The specificity of different sense clusters to a community, combined with the specificity of a community's unique word types, is used to identify cases where a social group's language deviates from the norm. We validate our metrics using user-created glossaries and draw on sociolinguistic theories to connect language variation with trends in community behavior. We find that communities with highly distinctive language are medium-sized, and their loyal and highly engaged users interact in dense networks.

Motivation & Objective

• To develop a computational method for measuring linguistic variation in online communities beyond just word frequency.
• To investigate how word senses—beyond just word types—vary across social media communities.
• To validate the proposed metrics using user-created glossaries and sociolinguistic theories.
• To explore the relationship between user behavior, network structure, and community-specific language use.
• To determine whether topic of discussion or user-level attributes better predict linguistic distinctiveness.

Proposed method

• The authors use BERT to extract contextualized word embeddings for words across 474 Reddit communities, capturing different word senses in context.
• They compute a community-specificity score (F) based on both word type frequency and sense-level clustering in BERT embedding space.
• Sense clusters are identified using t-SNE visualization and k-means clustering on BERT embeddings of target words like 'python'.
• The study compares BERT-based sense variation with traditional word type specificity metrics (e.g., PMI-based specificity from prior work).
• OLS regression models are used to test the impact of user-based attributes (e.g., activity, network density) and topic categories on linguistic distinctiveness (F score).
• User-created glossaries from 57 Reddit communities are used to validate that BERT-identified sense clusters align with community-defined jargon.

Experimental results

Research questions

• RQ1 How do word senses vary across different online communities, and can BERT effectively capture these variations?
• RQ2 To what extent do user behavior and network structure predict linguistic distinctiveness in online communities?
• RQ3 Does the topic of discussion (e.g., technology, sports) have a stronger effect on language variation than user-level attributes?
• RQ4 How well do BERT-identified sense clusters align with community-defined glossaries of jargon?
• RQ5 Are communities with highly distinctive language more likely to be medium-sized with dense, loyal user networks?

Key findings

• Communities with the highest linguistic distinctiveness (F scores) are medium-sized, not the largest or smallest.
• Highly engaged and loyal users are more prevalent in communities with distinctive language, and they form dense interaction networks.
• After controlling for user behavior, network density has a negative effect on linguistic variation, suggesting its earlier positive marginal effect was confounded by correlated features.
• Topic-related features (e.g., Technology, Video Games, Hobbies) are associated with higher linguistic distinctiveness, but user behavior remains a stronger predictor.
• The F score, combining word type and sense specificity, shows strong alignment with user-created glossaries, validating the method’s ability to detect community-specific jargon.
• BERT embeddings successfully isolate different meanings of polysemous words (e.g., 'python' as programming language vs. spacecraft), with distinct clusters in embedding space across communities.