[Paper Review] Social-LLM: Modeling User Behavior at Scale using Language Models and Social Network Data

Julie Jiang, Emilio Ferrara|arXiv (Cornell University)|Dec 31, 2023

Complex Network Analysis Techniques4 citations

TL;DR

Social-LLM proposes a scalable, inductive user representation model that combines large language model (LLM)-derived user content features with first-order social network interactions (e.g., retweets, mentions) to enable efficient user detection across diverse computational social science tasks. By leveraging social homophily and modeling only direct edges, it achieves state-of-the-art performance on 7 real-world datasets while supporting out-of-sample inference without retraining.

ABSTRACT

The proliferation of social network data has unlocked unprecedented opportunities for extensive, data-driven exploration of human behavior. The structural intricacies of social networks offer insights into various computational social science issues, particularly concerning social influence and information diffusion. However, modeling large-scale social network data comes with computational challenges. Though large language models make it easier than ever to model textual content, any advanced network representation methods struggle with scalability and efficient deployment to out-of-sample users. In response, we introduce a novel approach tailored for modeling social network data in user detection tasks. This innovative method integrates localized social network interactions with the capabilities of large language models. Operating under the premise of social network homophily, which posits that socially connected users share similarities, our approach is designed to address these challenges. We conduct a thorough evaluation of our method across seven real-world social network datasets, spanning a diverse range of topics and detection tasks, showcasing its applicability to advance research in computational social science.

Motivation & Objective

• To address the scalability limitations of graph representation learning in large-scale social network data.
• To enable inductive user detection—applicable to out-of-sample users—by integrating LLMs with social network structure.
• To evaluate the effectiveness of different social interaction types (e.g., retweets, mentions) and edge properties (direction, weight) in user representation.
• To demonstrate the utility of learned embeddings for visualization and interpretability in social network analysis.
• To establish a robust, general-purpose method for user detection tasks across diverse topics, including political polarization, hate speech, and account suspension.

Proposed method

• Social-LLM constructs user representations by combining LLM-embedded user profile descriptions with first-order social network edges (e.g., retweets, mentions).
• It models social network homophily by assuming that connected users share linguistic and behavioral similarities, enabling effective representation learning.
• The method uses a simple yet effective graph approximation—treating only direct edges as signals—avoiding complex graph neural network training and enabling inductive inference.
• Edge types are combined indiscriminately (e.g., retweets + mentions as one edge type) when performance is improved, and edge directionality is shown to be critical for performance.
• Edge weights are optionally added to reflect interaction strength, and user tweet embeddings are incorporated when available to enhance representation quality.
• The final user embeddings are used for downstream classification tasks and visualized via t-SNE to assess clusterability and interpretability.

Experimental results

Research questions

• RQ1 Can a simple, scalable method that leverages LLMs and first-order social network edges outperform state-of-the-art models in user detection tasks?
• RQ2 How do different types of social interactions (e.g., retweets vs. mentions) affect model performance when modeled collectively or separately?
• RQ3 To what extent does edge directionality and weighting improve user representation quality and detection accuracy?
• RQ4 Can Social-LLM embeddings effectively visualize meaningful social structures, such as political polarization or account suspension patterns?
• RQ5 How robust is the model across diverse datasets with varying topics and detection tasks?

Key findings

• Social-LLM achieves state-of-the-art performance across 7 real-world social media datasets, outperforming content-only and network-only baselines in 6 out of 7 datasets.
• The model demonstrates statistically significant improvements over baselines, with average Macro-F1 gains of up to 4.1% on the most favorable datasets.
• Combining retweets and mentions as a single edge type yields better performance than treating them separately, suggesting that distinctions between interaction types are less critical for detection tasks.
• Incorporating edge directionality consistently improves performance, highlighting the importance of information flow in social networks.
• Adding edge weights further enhances performance in some cases, indicating that interaction strength carries meaningful signal.
• User tweet embeddings improve performance by an average of 4% Macro-F1 when available, underscoring the value of incorporating full user content.