[Paper Review] GeNet: A Multimodal LLM-Based Co-Pilot for Network Topology and Configuration

Beni Ifland, Elad Duani+10 more|arXiv (Cornell University)|Jul 11, 2024

Service-Oriented Architecture and Web Services4 citations

TL;DR

GeNet is a multimodal large language model (LLM)-based co-pilot that automates enterprise network design by interpreting network topology images and textual intents to update both network topologies and device configurations. It achieves 72% accuracy in configuring newly added components and demonstrates that precise topology understanding is critical for topology-modifying intents.

ABSTRACT

Communication network engineering in enterprise environments is traditionally a complex, time-consuming, and error-prone manual process. Most research on network engineering automation has concentrated on configuration synthesis, often overlooking changes in the physical network topology. This paper introduces GeNet, a multimodal co-pilot for enterprise network engineers. GeNet is a novel framework that leverages a large language model (LLM) to streamline network design workflows. It uses visual and textual modalities to interpret and update network topologies and device configurations based on user intents. GeNet was evaluated on enterprise network scenarios adapted from Cisco certification exercises. Our results demonstrate GeNet's ability to interpret network topology images accurately, potentially reducing network engineers' efforts and accelerating network design processes in enterprise environments. Furthermore, we show the importance of precise topology understanding when handling intents that require modifications to the network's topology.

Motivation & Objective

• Address the gap in network engineering automation by supporting both topology and configuration updates, not just configuration synthesis.
• Enable network engineers to specify high-level intents using natural language, with GeNet interpreting visual topology diagrams to guide updates.
• Reduce manual effort and accelerate network design by automating intent-driven topology and configuration changes.
• Investigate the impact of LLM temperature settings on topology understanding and intent fulfillment accuracy.
• Demonstrate that accurate topology interpretation is essential for tasks requiring structural network changes.

Proposed method

• GeNet employs a two-module pipeline: topology understanding (TIU) and intent implementation (II), with the former processing visual topology images and the latter generating configuration updates.
• The topology understanding module uses a vision-language LLM to extract structural information from network topology diagrams, enabling context-aware interpretation.
• Intent implementation leverages the LLM to generate device configurations and topology modifications based on user-provided natural language intents.
• The system uses a chain-of-thought prompting strategy to guide reasoning over topology and configuration changes, improving consistency and correctness.
• Performance is evaluated on 250 scenarios adapted from Cisco certification exercises, measuring success via S3-Z scores and topology understanding accuracy.
• Temperature hyperparameter tuning is used to assess the impact of LLM creativity on output reliability, with lower temperatures yielding better results.

Experimental results

Research questions

• RQ1 To what extent can a multimodal LLM accurately interpret network topology images to support intent-based network updates?
• RQ2 How does the performance of topology understanding affect the success of intent implementation in network design tasks?
• RQ3 What is the impact of LLM temperature settings on the reliability and accuracy of topology and configuration updates?
• RQ4 How does the complexity of topology changes (e.g., adding multiple components) compare to configuration-only updates in terms of LLM performance?
• RQ5 Is precise topology understanding necessary for successful fulfillment of intents that require structural network modifications?

Key findings

• GeNet successfully configured 79% of all expected network components across all test cases, with 92% of reconfiguration tasks completed correctly.
• GeNet achieved 72% accuracy in configuring newly added components not present in the initial input, indicating challenges in generating configurations from scratch.
• The topology understanding module showed a strong positive correlation (r = 0.55) with intent implementation success in topology scenarios, highlighting its critical role.
• Increasing the LLM temperature parameter significantly degraded intent fulfillment performance, with S3-Z scores decreasing, indicating that deterministic outputs are essential for network tasks.
• Configuration update tasks were less dependent on precise topology understanding than topology modification tasks, as evidenced by a weak negative correlation (r = -0.21) between topology understanding and success in configuration scenarios.
• The results confirm that accurate topology interpretation is essential for intents requiring structural changes, such as adding new switches or reorganizing network segments.