[Paper Review] Reinforcement Learning of Graph Neural Networks for Service Function Chaining

DongNyeong Heo, Do‐Young Lee+3 more|arXiv (Cornell University)|Nov 17, 2020

Software-Defined Networks and 5G10 references4 citations

TL;DR

This paper proposes a reinforcement learning (RL) framework to train graph neural networks (GNNs) for service function chaining (SFC) in dynamic network topologies, enabling generalization without labeled data. By using sparse rewards based on path validity and delay, the RL-trained GNNs achieve comparable performance to supervised learning on fixed topologies but significantly outperform it on unseen, randomly generated topologies, demonstrating robustness and adaptability.

ABSTRACT

In the management of computer network systems, the service function chaining (SFC) modules play an important role by generating efficient paths for network traffic through physical servers with virtualized network functions (VNF). To provide the highest quality of services, the SFC module should generate a valid path quickly even in various network topology situations including dynamic VNF resources, various requests, and changes of topologies. The previous supervised learning method demonstrated that the network features can be represented by graph neural networks (GNNs) for the SFC task. However, the performance was limited to only the fixed topology with labeled data. In this paper, we apply reinforcement learning methods for training models on various network topologies with unlabeled data. In the experiments, compared to the previous supervised learning method, the proposed methods demonstrated remarkable flexibility in new topologies without re-designing and re-training, while preserving a similar level of performance.

Motivation & Objective

• To address the limitation of supervised learning in SFC, which requires labeled data and fails on new topologies.
• To develop a flexible SFC module that generalizes across diverse, dynamic network topologies without retraining.
• To enable end-to-end training of GNN-based SFC models using only environment-provided rewards, eliminating dependency on labeled paths.
• To improve robustness of SFC path generation under changing VNF locations and network structures.

Proposed method

• Trained a GNN-based SFC model using the REINFORCE algorithm with a custom reward function that combines path validity and delay penalty.
• Designed a sparse reward signal: +1 for successful path generation, -1 for failure, and a delay penalty scaled by hyperparameter λ.
• Introduced two topology augmentation strategies during training: random node/edge addition/removal and random VNF repositioning to simulate diverse network states.
• Used the same GG-RNN architecture as prior work but fine-tuned via RL instead of supervised learning.
• Applied ε-greedy exploration with ε = 0.01 to balance exploration and exploitation during training.
• Trained models on 100 randomly generated topologies per episode, ensuring exposure to structural diversity.

Experimental results

Research questions

• RQ1 Can reinforcement learning train GNNs for SFC without relying on labeled paths from expensive ILP solvers?
• RQ2 How does RL-based training affect generalization to unseen network topologies compared to supervised learning?
• RQ3 Does incorporating delay as a penalty in the reward function improve path quality in RL-trained models?
• RQ4 How do different topology augmentation strategies during training affect robustness to structural changes?
• RQ5 Can RL-trained SFC models maintain high performance on new topologies while preserving performance on original topologies?

Key findings

• On the original 'internet2' topology, the RL method achieved a failure ratio of 0.00% and a delay ratio of 1.01, matching the performance of the supervised baseline.
• On randomly changed topologies (CS1), the RL model reduced failure ratio to 0.00% compared to 1.00% for the supervised method, demonstrating superior generalization.
• On topologies with both structural changes and repositioned VNFs (CS2), the RL model maintained 0.00% failure ratio, while the supervised method failed in 1.00% of cases.
• The model trained with λ = 1 (delay penalty) achieved a 1.01 delay ratio, slightly worse than λ = 0 (1.00), but significantly improved path quality in dynamic settings.
• The RL model trained with strategy CS2 showed greater robustness than CS1 in the third test, indicating that richer topology variation during training enhances adaptability.
• The deterioration rate metric confirmed that RL models were far less sensitive to topology shifts than supervised models, validating their flexibility.