[Paper Review] From Twitter to Traffic Predictor: Next-Day Morning Traffic Prediction Using Social Media Data

Weiran Yao, Sean Qian|arXiv (Cornell University)|Sep 29, 2020

Traffic Prediction and Management Techniques78 references67 citations

TL;DR

This paper proposes tweet2traffic, a social media-driven framework that predicts next-day morning traffic congestion using Twitter data collected by midnight. By analyzing tweet sentiment, timing, and geolocation patterns—especially nighttime and early morning activity—it extracts meaningful travel demand signals, significantly improving morning congestion forecasts, particularly upstream of bottlenecks with high day-to-day variation.

ABSTRACT

The effectiveness of traditional traffic prediction methods is often extremely limited when forecasting traffic dynamics in early morning. The reason is that traffic can break down drastically during the early morning commute, and the time and duration of this break-down vary substantially from day to day. Early morning traffic forecast is crucial to inform morning-commute traffic management, but they are generally challenging to predict in advance, particularly by midnight. In this paper, we propose to mine Twitter messages as a probing method to understand the impacts of people's work and rest patterns in the evening/midnight of the previous day to the next-day morning traffic. The model is tested on freeway networks in Pittsburgh as experiments. The resulting relationship is surprisingly simple and powerful. We find that, in general, the earlier people rest as indicated from Tweets, the more congested roads will be in the next morning. The occurrence of big events in the evening before, represented by higher or lower tweet sentiment than normal, often implies lower travel demand in the next morning than normal days. Besides, people's tweeting activities in the night before and early morning are statistically associated with congestion in morning peak hours. We make use of such relationships to build a predictive framework which forecasts morning commute congestion using people's tweeting profiles extracted by 5 am or as late as the midnight prior to the morning. The Pittsburgh study supports that our framework can precisely predict morning congestion, particularly for some road segments upstream of roadway bottlenecks with large day-to-day congestion variation. Our approach considerably outperforms those existing methods without Twitter message features, and it can learn meaningful representation of demand from tweeting profiles that offer managerial insights.

Motivation & Objective

• Address the critical challenge of next-day morning traffic prediction, which remains difficult for traditional models due to high day-to-day variation in congestion patterns.
• Overcome the limitation of conventional traffic prediction models that rely on real-time or historical traffic data, which are uninformative before 5am due to free-flow conditions.
• Explore whether social media data—specifically Twitter—can serve as a proxy for population-level travel demand and behavioral patterns influencing next-day morning traffic.
• Develop a data-driven framework that integrates social media signals with traffic incidents, weather, and temporal variables to improve early prediction of congestion onset and duration.
• Demonstrate that tweet-derived features can enhance prediction accuracy, especially for road segments with high sensitivity to demand fluctuations and upstream of bottlenecks.

Proposed method

• Collect and preprocess Twitter data, including geocoded tweets, retweets, favorites, and user timelines, focusing on nighttime (midnight–5am) and early morning periods.
• Implement a data augmentation strategy that filters local residents from noisy users (e.g., tourists), infers their home locations, and extends tweet data using non-geocoded activity (e.g., retweets, likes).
• Apply a neural network-based sentiment analysis model to detect abnormal local events (e.g., concerts, games) by identifying deviations in tweet sentiment from baseline.
• Extract traffic incident reports from tweets using natural language processing and keyword matching to identify planned or unplanned disruptions.
• Construct a multi-source predictive pipeline with three components: (1) road-level traffic descriptor, (2) segment-level traffic classifier (binary congestion), and (3) segment-level traffic regressor (congestion start time, duration, and planning index).
• Use a clustered learning structure that leverages ordered spatio-temporal congestion patterns along road segments to improve generalization and prediction accuracy.

Experimental results

Research questions

• RQ1 Can Twitter data collected by midnight provide reliable signals for predicting next-day morning traffic congestion?
• RQ2 How do people’s sleep-wake patterns, as inferred from nighttime tweeting, correlate with morning congestion levels?
• RQ3 To what extent do abnormal local events—indicated by extreme tweet sentiment—predict reduced morning travel demand and lower congestion?
• RQ4 How do tweet-derived features compare in predictive power to traditional data sources (e.g., traffic incidents, weather) across different road segments?
• RQ5 Does the integration of social media data improve prediction performance most significantly in areas with high day-to-day congestion variability, such as upstream of bottlenecks?

Key findings

• The earlier people go to sleep—indicated by reduced nighttime tweeting activity—the more congested the roads are likely to be the next morning.
• Tweeting activity in the night before and early morning (by 5am) is statistically associated with higher morning congestion, suggesting a direct link between late-night behavior and next-day traffic patterns.
• Abnormal local events, such as sports games or concerts, detected via sentiment deviation, are associated with lower-than-normal travel demand and reduced morning congestion.
• The tweet2traffic framework significantly outperforms baseline models without social media features, especially for road segments upstream of bottlenecks with high day-to-day congestion variation.
• Social media data improve prediction accuracy most notably during days with special events, when people delay or reduce morning travel due to evening activities.
• The model remains robust even with increased forecasting lag, demonstrating that data collected by midnight are sufficient for reliable next-day morning traffic prediction.