[Paper Review] Big Data application in congestion detection and classification using Apache spark

Atousa Zarindast, Anuj Sharma|arXiv (Cornell University)|Jan 16, 2021

Traffic Prediction and Management Techniques17 references4 citations

TL;DR

This paper proposes a scalable, Spark-based big data framework for spatio-temporal congestion detection and recurrent congestion (RC) classification on Iowa's interstates using 76 GB of probe vehicle data. Leveraging Apache Spark’s in-memory processing and HDFS storage, the method reduces computation time by 99.88% while maintaining 90% accuracy, enabling near real-time traffic management and pattern identification across 3,017 freeway segments.

ABSTRACT

With the era of big data, an explosive amount of information is now available. This enormous increase of Big Data in both academia and industry requires large-scale data processing systems. A large body of research is behind optimizing Spark's performance to make it state of the art, a fast and general data processing system. Many science and engineering fields have advanced with Big Data analytics, such as Biology, finance, and transportation. Intelligent transportation systems (ITS) gain popularity and direct benefit from the richness of information. The objective is to improve the safety and management of transportation networks by reducing congestion and incidents. The first step toward the goal is better understanding, modeling, and detecting congestion across a network efficiently and effectively. In this study, we introduce an efficient congestion detection model. The underlying network consists of 3017 segments in I-35, I-80, I-29, and I-380 freeways with an overall length of 1570 miles and averaged (0.4-0.6) miles per segment. The result of congestion detection shows the proposed method is 90% accurate while has reduced computation time by 99.88%.

Motivation & Objective

• To develop a scalable, high-performance big data framework for detecting congestion and recurrent congestion (RC) across large-scale freeway networks.
• To address the limitations of prior studies that are restricted to small geographic areas or short time periods by enabling comprehensive spatio-temporal analysis.
• To reduce computation time and cost while maintaining high detection accuracy for real-time traffic management applications.
• To leverage Apache Spark’s in-memory processing and HDFS for efficient, distributed data processing of large-scale transportation data.
• To provide a generalizable, computationally efficient solution applicable to real-time traffic monitoring and policy development.

Proposed method

• The framework uses Apache Spark’s Resilient Distributed Datasets (RDD) model to process 76 GB of probe vehicle speed data from Iowa’s interstates (I-35, I-80, I-29, I-380) across 3,017 segments.
• Data is stored and managed using the Hadoop Distributed File System (HDFS), enabling scalable and fault-tolerant data access.
• The method applies a three-phase pipeline: (1) data preprocessing, (2) Bayesian change point detection for congestion identification, and (3) classification of recurrent vs. non-recurrent congestion.
• Speed profiles are analyzed per segment and time of day to detect congestion events, with recurrent congestion defined as consistent congestion during peak hours.
• The system uses PySpark for high-level programming and leverages Spark optimizations such as in-memory computation, partitioning, and broadcast variables to enhance performance.
• Accuracy is evaluated using standard metrics: true positive (TP), true negative (TN), false positive (FP), and false negative (FN), with accuracy calculated as (TP + TN) / (TP + TN + FP + FN).

Experimental results

Research questions

• RQ1 How can a big data framework be designed to detect congestion and recurrent congestion patterns across large-scale, multi-segment freeway networks?
• RQ2 To what extent can Apache Spark reduce computation time and cost compared to traditional data-driven methods in congestion detection?
• RQ3 What is the trade-off between detection accuracy and computational efficiency in a scalable big data processing pipeline for traffic congestion analysis?
• RQ4 Can a distributed, in-memory processing system like Spark achieve high accuracy while processing terabyte-scale transportation data in near real time?
• RQ5 How does the proposed method compare in performance and accuracy to a recently published data-driven congestion detection method on the same dataset?

Key findings

• The proposed framework achieved 90% congestion detection accuracy, significantly outperforming the baseline method in terms of computational efficiency.
• Computation time was reduced by 99.88%, from 151 hours (baseline) to just 10 minutes, using the same hardware resources.
• The total processing cost was reduced from $488 to $1.34 on Amazon Web Services (AWS), demonstrating substantial cost savings.
• The method successfully identified recurrent congestion patterns, with Des Moines Loop showing consistent congestion during peak hours, as confirmed by visualizations in Figures 5 and 6.
• A trade-off analysis showed that a 10% decrease in accuracy could yield a 99.88% reduction in processing time, highlighting strong scalability and efficiency.
• The framework processed 76 GB of data across 3,017 freeway segments (1,570 miles) in 10 minutes, demonstrating high scalability and real-time feasibility.