[Paper Review] Towards Message Brokers for Generative AI: Survey, Challenges, and Opportunities

Alaa Saleh, Morabito, Roberto+3 more|arXiv (Cornell University)|Dec 22, 2023

Scientific Computing and Data Management4 citations

TL;DR

This survey proposes a specialized message broker framework tailored for Generative AI (GenAI) workloads, analyzing traditional and modern pub/sub brokers across critical criteria like scalability, fault tolerance, and data persistence. It identifies key limitations in existing systems and outlines a robust, adaptable architecture to meet GenAI's dynamic, high-volume data communication needs.

ABSTRACT

In today's digital world, Generative Artificial Intelligence (GenAI) such as Large Language Models (LLMs) is becoming increasingly prevalent, extending its reach across diverse applications. This surge in adoption has sparked a significant increase in demand for data-centric GenAI models, highlighting the necessity for robust data communication infrastructures. Central to this need are message brokers, which serve as essential channels for data transfer within various system components. This survey aims to delve into a comprehensive analysis of traditional and modern message brokers, offering a comparative study of prevalent platforms. Our study considers numerous criteria including, but not limited to, open-source availability, integrated monitoring tools, message prioritization mechanisms, capabilities for parallel processing, reliability, distribution and clustering functionalities, authentication processes, data persistence strategies, fault tolerance, and scalability. Furthermore, we explore the intrinsic constraints that the design and operation of each message broker might impose, recognizing that these limitations are crucial in understanding their real-world applicability. Finally, this study examines the enhancement of message broker mechanisms specifically for GenAI contexts, emphasizing the criticality of developing a versatile message broker framework. Such a framework would be poised for quick adaptation, catering to the dynamic and growing demands of GenAI in the foreseeable future. Through this dual-pronged approach, we intend to contribute a foundational compendium that can guide future innovations and infrastructural advancements in the realm of GenAI data communication.

Motivation & Objective

• To analyze the suitability of existing message brokers for Generative AI (GenAI) workloads, especially those based on large language models (LLMs).
• To identify critical limitations in current message broker designs that hinder effective support for GenAI's high-throughput, low-latency data exchange requirements.
• To propose a novel, adaptable message broker framework optimized for the evolving demands of GenAI applications in distributed, dynamic computing environments.
• To guide future research and infrastructure development by highlighting key challenges and opportunities in building resilient, scalable, and efficient GenAI-native messaging systems.

Proposed method

• Conduct a comprehensive comparative analysis of 12 prominent message brokers—including Apache Kafka, RabbitMQ, Amazon MQ, Google Cloud Pub/Sub, Azure Storage Queue, Amazon Kinesis, and IronMQ—evaluating them across 14 key criteria.
• Systematically assess each broker on features such as clustering, monitoring, authentication, message prioritization, fault tolerance, and support for both push and pull messaging patterns.
• Identify architectural and operational constraints (e.g., message size limits, connection caps, lack of guaranteed delivery) that restrict GenAI applicability.
• Synthesize findings into a proposed framework that integrates best practices from top-performing brokers, emphasizing extensibility, horizontal scalability, and real-time diagnostics.
• Propose algorithmic and architectural enhancements for message routing, load balancing, and state management tailored to GenAI’s data-intensive pipelines.
• Integrate principles from MLOps and Continuous Diagnostics and Mitigation (CDM) to enable runtime monitoring and adaptive configuration in GenAI messaging stacks.

Experimental results

Research questions

• RQ1 What are the key architectural and operational limitations of existing message brokers when applied to GenAI workloads?
• RQ2 How do different message brokers compare across critical criteria such as scalability, fault tolerance, message persistence, and support for parallel processing in GenAI contexts?
• RQ3 What design principles and components are essential for a message broker framework to effectively support the dynamic, high-volume data flows of GenAI applications?
• RQ4 How can message broker systems be enhanced to support real-time diagnostics, monitoring, and adaptive resource management in GenAI pipelines?
• RQ5 What role do pub/sub paradigms and brokerless alternatives play in shaping the future of scalable, resilient data communication for GenAI systems?

Key findings

• Existing message brokers exhibit significant limitations for GenAI workloads, including strict message size caps (e.g., 10MB in Google Cloud Pub/Sub, 64KB in Azure Storage Queue), connection limits (up to 1,000 in Amazon MQ), and lack of guaranteed delivery in certain configurations.
• Brokers like Apache Kafka and RabbitMQ demonstrate strong support for clustering, fault tolerance, and persistent storage, making them more suitable for production GenAI pipelines than managed cloud services with hidden scaling constraints.
• Proprietary brokers such as Google Cloud Pub/Sub and Amazon Kinesis face limitations in large-scale GenAI deployments due to data size restrictions and rate-limiting behaviors under high load.
• Message brokers with native support for multi-protocol communication, distributed tracing, and real-time stream analytics (e.g., Apache Kafka, Amazon Kinesis) show higher adaptability for complex GenAI data pipelines.
• The absence of standardized, extensible frameworks for message broker interoperability and dynamic configuration remains a key gap in current GenAI infrastructure.
• A proposed message broker framework must integrate advanced features such as dynamic message routing, support for both push and pull models, and real-time monitoring to ensure high Quality of Service (QoS) in GenAI systems.