[Paper Review] Towards Fine Grained Network Flow Prediction

Patrick Jahnke, Emmanuel Stapf+3 more|arXiv (Cornell University)|Aug 20, 2018

Network Traffic and Congestion Control31 references3 citations

TL;DR

This paper presents the Frequency-based Kernel Kalman Filter (FKKF), a novel method for fine-grained, per-flow network traffic prediction by transforming time-series flow data into the frequency domain using Short-Time Fourier Transform (STFT), extracting key patterns via PCA, and applying a kernelized Kalman filter to predict bursty traffic peaks. The approach achieves an average prediction error of 6.43% across 17 out of 20 flow groups, significantly outperforming coarse-grained methods that exhibit 77% error.

ABSTRACT

One main challenge for the design of networks is that traffic load is not generally known in advance. This makes it hard to adequately devote resources such as to best prevent or mitigate bottlenecks. While several authors have shown how to predict traffic in a coarse grained manner by aggregating flows, fine grained prediction of traffic at the level of individual flows, including bursty traffic, is widely considered to be impossible. This paper shows, to the best of our knowledge, the first approach to fine grained per flow traffic prediction. In short, we introduce the Frequency-based Kernel Kalman Filter (FKKF), which predicts individual flows' behavior based on measurements. Our FKKF relies on the well known Kalman Filter in combination with a kernel to support the prediction of non linear functions. Furthermore we change the operating space from time to frequency space. In this space, into which we transform the input data via a Short-Time Fourier Transform (STFT), the peak structures of flows can be predicted after gleaning their key characteristics, with a Principal Component Analysis (PCA), from past and ongoing flows that stem from the same socket-to-socket connection. We demonstrate the effectiveness of our approach on popular benchmark traces from a university data center. Our approach predicts traffic on average across 17 out of 20 groups of flows with an average prediction error of 6.43% around 0.49 (average) seconds in advance, whilst existing coarse grained approaches exhibit prediction errors of 77% at best.

Motivation & Objective

• To address the challenge of predicting bursty, individual network flows in real time, which is critical for proactive congestion control.
• To overcome the limitations of coarse-grained aggregation methods that fail to capture short-term, high-variance traffic bursts.
• To enable accurate, early prediction of peak traffic in individual flows using only past and ongoing data from the same socket-to-socket connection.
• To design a scalable, decentralized prediction system suitable for deployment on commodity network switches.
• To support proactive network management by enabling timely rerouting or resource allocation before congestion occurs.

Proposed method

• Transform raw network flow data into the frequency domain using Short-Time Fourier Transform (STFT) to reveal stable spectral patterns.
• Apply Principal Component Analysis (PCA) to extract dominant frequency components from past and ongoing flows sharing the same socket-to-socket connection.
• Use a kernelized Kalman filter (FKKF) to model non-linear dynamics in the frequency domain, enabling prediction of future peak structures.
• Operate in a decentralized manner, with FKKF instances running on network devices to predict individual flows in real time.
• Integrate the FKKF with a monitoring system that ensures high-precision polling at 10ms intervals to maintain prediction accuracy.
• Optimize computational load to allow prediction of over 200 flows per switch core, enabling scalability in large networks.

Experimental results

Research questions

• RQ1 Can fine-grained per-flow traffic prediction be achieved for bursty network flows using only historical and ongoing data from the same connection?
• RQ2 Does transforming flow data into the frequency domain enable more accurate prediction of short-term traffic peaks than time-domain methods?
• RQ3 Can a kernelized Kalman filter effectively model non-linear, high-variance traffic dynamics in the frequency domain?
• RQ4 Is the proposed FKKF approach scalable and efficient enough for deployment on commodity network hardware?
• RQ5 Can the FKKF predict traffic peaks early enough and accurately enough to enable proactive congestion control?

Key findings

• The FKKF achieved an average prediction error of 6.43% across 17 out of 20 flow groups, demonstrating high accuracy in predicting bursty traffic peaks.
• The method predicted traffic on average 0.49 seconds in advance, providing sufficient lead time for proactive network adaptation.
• In contrast, traditional methods like ARIMA and GARCH failed to predict peaks, often predicting zeros or oscillating values due to their inability to model non-linear bursts.
• The FKKF required only 3.65ms on average per 1000 predictions on a single CPU core, enabling real-time prediction of over 200 flows per switch.
• The monitoring system achieved 99% polling accuracy with only 50% CPU utilization when monitoring 100 flows, confirming feasibility on commodity hardware.
• The approach successfully identified recurring patterns in socket-to-socket connections, enabling inter-flow learning and robust prediction even with limited initial data.