[Paper Review] Averaging Gone Wrong: Using Time-Aware Analyses to Better Understand Behavior

Samuel Barbosa, Dan Cosley+2 more|arXiv (Cornell University)|Mar 22, 2016

Social Media and Politics56 references20 citations

TL;DR

This paper argues that aggregate analyses of online community behavior are misleading due to unaccounted temporal dynamics. By analyzing Reddit users in yearly cohorts and using user-centric time references, it reveals that comment length and activity increase over user tenure—contradicting aggregate trends due to Simpson’s Paradox—highlighting the critical need for time-aware methodologies in social network research.

ABSTRACT

Online communities provide a fertile ground for analyzing people's behavior and improving our understanding of social processes. Because both people and communities change over time, we argue that analyses of these communities that take time into account will lead to deeper and more accurate results. Using Reddit as an example, we study the evolution of users based on comment and submission data from 2007 to 2014. Even using one of the simplest temporal differences between users---yearly cohorts---we find wide differences in people's behavior, including comment activity, effort, and survival. Further, not accounting for time can lead us to misinterpret important phenomena. For instance, we observe that average comment length decreases over any fixed period of time, but comment length in each cohort of users steadily increases during the same period after an abrupt initial drop, an example of Simpson's Paradox. Dividing cohorts into sub-cohorts based on the survival time in the community provides further insights; in particular, longer-lived users start at a higher activity level and make more and shorter comments than those who leave earlier. These findings both give more insight into user evolution in Reddit in particular, and raise a number of interesting questions around studying online behavior going forward.

Motivation & Objective

• To challenge the validity of aggregate, time-agnostic analyses in online community research.
• To investigate how user behavior evolves over time by accounting for when users join the community.
• To demonstrate that ignoring temporal dynamics leads to misleading conclusions about user activity and effort.
• To explore the impact of user survival duration on behavior patterns, such as commenting vs. submission habits.
• To highlight the importance of cohort-based and user-centric time references in understanding long-term user evolution.

Proposed method

• Cohort-based analysis: grouping Reddit users by their year of first visible activity (e.g., 2007, 2008, etc.).
• Comparing aggregate trends (over calendar time) with cohort-specific trends (over user tenure).
• Using user-referential time: measuring behavior relative to each user’s first activity date, not calendar time.
• Analyzing comment length, posting frequency, and comment-to-submission ratios across cohorts and over time.
• Applying statistical techniques to detect and explain paradoxical trends, such as Simpson’s Paradox.
• Focusing on surviving users (those active beyond a threshold) to isolate long-term behavioral evolution from attrition effects.

Experimental results

Research questions

• RQ1 How does user posting activity change over time when analyzed by cohort rather than aggregated across all users?
• RQ2 Does comment length increase or decrease over a user’s tenure in the community, and how does this compare to overall aggregate trends?
• RQ3 How does the ratio of comments to submissions evolve for users who survive longer in the community?
• RQ4 To what extent do cohort-level differences in behavior mask or distort overall trends in aggregate data?
• RQ5 What role does user survival duration play in shaping observed behavioral patterns in online communities?

Key findings

• While aggregate analysis shows a decreasing trend in average comment length over time, each cohort’s comment length increases steadily over user tenure, demonstrating Simpson’s Paradox.
• Users who survive longer in Reddit start with higher activity levels and make more, shorter comments than those who leave early.
• Older surviving users are significantly more active than newer survivors, and younger cohorts do not catch up in activity levels.
• The percentage of low-activity survivors is increasing in younger cohorts, indicating a shift in user engagement patterns over time.
• Longer-lived users substitute commenting for submissions over time, increasing their comment-to-submission ratio, even as overall posting activity remains stable.
• Aggregate views misrepresent user behavior: the rise in average activity for surviving users is driven not by increased effort, but by the departure of low-activity users early in their tenure.