[Paper Review] Nonlinear Traffic Prediction as a Matrix Completion Problem with Ensemble Learning

Wenqing Li, Chuhan Yang+1 more|arXiv (Cornell University)|Jan 8, 2020

Traffic Prediction and Management TechniquesEngineering67 references22 citations

TL;DR

This paper proposes a novel matrix completion framework with block-coordinate descent and ensemble learning for high-resolution (second-by-second) traffic prediction at signalized intersections. By modeling traffic states as a low-rank matrix and leveraging historical data through adaptive boosting, the method achieves sub-linear convergence and state-of-the-art accuracy, outperforming RNNs in computational efficiency while maintaining comparable prediction quality on real-world Abu Dhabi data.

ABSTRACT

This paper addresses the problem of short-term traffic prediction for signalized traffic operations management. Specifically, we focus on predicting sensor states in high-resolution (second-by-second). This contrasts with traditional traffic forecasting problems, which have focused on predicting aggregated traffic variables, typically over intervals that are no shorter than 5 minutes. Our contributions can be summarized as offering three insights: first, we show how the prediction problem can be modeled as a matrix completion problem. Second, we employ a block-coordinate descent algorithm and demonstrate that the algorithm converges in sub-linear time to a block coordinate-wise optimizer. This allows us to capitalize on the "bigness" of high-resolution data in a computationally feasible way. Third, we develop an ensemble learning (or adaptive boosting) approach to reduce the training error to within any arbitrary error threshold. The latter utilizes past days so that the boosting can be interpreted as capturing periodic patterns in the data. The performance of the proposed method is analyzed theoretically and tested empirically using both simulated data and a real-world high-resolution traffic dataset from Abu Dhabi, UAE. Our experimental results show that the proposed method outperforms other state-of-the-art algorithms.

Motivation & Objective

• To address the challenge of short-term traffic prediction at second-by-second resolution for adaptive signal control systems.
• To model high-resolution traffic prediction as a matrix completion problem to exploit temporal and spatial correlations in sensor data.
• To develop a computationally efficient block-coordinate descent algorithm with sub-linear convergence for large-scale traffic data.
• To reduce training error to arbitrary thresholds using ensemble learning that captures periodic patterns from historical data.
• To validate the method empirically on real-world high-resolution traffic data from Abu Dhabi, UAE, and compare it with state-of-the-art baselines.

Proposed method

• Formulate traffic prediction as a low-rank matrix completion problem, where observed sensor states across time and space form a partially observed matrix.
• Apply a block-coordinate descent (BCD) algorithm to optimize the matrix completion objective, ensuring convergence in sub-linear time.
• Introduce an ensemble learning (adaptive boosting) framework that iteratively improves predictions by leveraging past days’ data as predictors.
• Use kernel regression and sparse approximation within the BCD framework to model nonlinear traffic dynamics.
• Incorporate periodicity by using historical days as weak learners in the boosting process, enhancing pattern capture.
• Optimize the matrix completion objective using a Schur complement-based update rule to ensure computational scalability.

Experimental results

Research questions

• RQ1 Can high-resolution traffic prediction be effectively modeled as a matrix completion problem to exploit spatiotemporal correlations?
• RQ2 Does the proposed block-coordinate descent algorithm achieve sub-linear convergence for large-scale traffic data?
• RQ3 Can ensemble learning with historical data improve training error and capture periodic traffic patterns in second-by-second predictions?
• RQ4 How does the proposed method compare in accuracy and computational efficiency to state-of-the-art models like RNNs and SVR on real-world high-resolution data?
• RQ5 To what extent does the method generalize to unseen data, and how close are test errors to training errors?

Key findings

• The proposed B-BCD method achieved a testing accuracy of 0.8858 ± 0.0213 (1 − dM1) and 0.9091 ± 0.0211 (1 − ϵMAE) on real-world Abu Dhabi data, outperforming BCD, SVR, TS-GLM, and nearly matching RNNs.
• The block-coordinate descent algorithm demonstrated sub-linear convergence, with training and testing accuracy curves converging rapidly, indicating strong generalization.
• Paired t-tests confirmed that B-BCD significantly outperformed BCD, SVR, and TS-GLM under both 1 − dM1 and 1 − ϵMAE metrics (p < 0.05), except when compared to RNN.
• The ensemble learning approach reduced training error to within arbitrary thresholds, effectively capturing periodic patterns through historical data as predictors.
• The method achieved comparable accuracy to RNNs but with significantly lower computational cost, making it suitable for real-time adaptive traffic control.
• Empirical results showed that out-of-sample errors differed from training errors by no more than 10%, suggesting good generalization performance.