[논문 리뷰] Diffusion-based Reinforcement Learning for Edge-enabled AI-Generated Content Services

Hongyang Du, Zonghang Li외 5명|arXiv (Cornell University)|2023. 03. 23.

Privacy-Preserving Technologies in Data인용 수 9

한 줄 요약

논문은 DRL에 D2SAC로 통합된 확산모델 기반 최적화기 AGOD를 제시하며, 메타버스를 위한 엣지 기반 AIGC 서비스의 ASP 선택을 최적화한다. 실험에서 D2SAC가 여러 DRL 베이스라인을 능가함을 보여준다.

ABSTRACT

As Metaverse emerges as the next-generation Internet paradigm, the ability to efficiently generate content is paramount. AIGenerated Content (AIGC) emerges as a key solution, yet the resource intensive nature of large Generative AI (GAI) models presents challenges. To address this issue, we introduce an AIGC-as-a-Service (AaaS) architecture, which deploys AIGC models in wireless edge networks to ensure broad AIGC services accessibility for Metaverse users. Nonetheless, an important aspect of providing personalized user experiences requires carefully selecting AIGC Service Providers (ASPs) capable of effectively executing user tasks, which is complicated by environmental uncertainty and variability. Addressing this gap in current research, we introduce the AI-Generated Optimal Decision (AGOD) algorithm, a diffusion model-based approach for generating the optimal ASP selection decisions. Integrating AGOD with Deep Reinforcement Learning (DRL), we develop the Deep Diffusion Soft Actor-Critic (D2SAC) algorithm, enhancing the efficiency and effectiveness of ASP selection. Our comprehensive experiments demonstrate that D2SAC outperforms seven leading DRL algorithms. Furthermore, the proposed AGOD algorithm has the potential for extension to various optimization problems in wireless networks, positioning it as a promising approach for future research on AIGC-driven services. The implementation of our proposed method is available at: https://github.com/Lizonghang/AGOD.

연구 동기 및 목표

• 무선 엣지 서버에서 AIGC 모델을 배치하여 보편적 AIGC 서비스를 가능하게 하는 AIGC-as-a-Service (AaaS) 아키텍처를 도입한다.
• 환경 불확실성 하에서 최적의 이산 결정을 생성하기 위한 확산 모델 기반 최적화기 AGOD를 개발한다.
• ASP( AIGC Service Provider) 선택을 위해 AGOD를 DRL과 통합하여 Deep Diffusion Soft Actor-Critic (D2SAC) 알고리즘을 만든다.
• D2SAC가 다수의 DRL 베이스라인보다 우수하다는 것을 보여주고 AGOD의 확장 가능성을 다른 무선 네트워크 최적화 문제에 대해 논의한다.]
• method2=None
• method1 : [
• Formulate ASP selection as a resource-constrained, online, discrete optimization problem with human-aware utility.
• Develop the AGOD algorithm that uses a diffusion process to generate discrete decision distributions conditioned on environment states.
• Describe the forward probability noising process and the reverse denoising process to obtain optimal decision distributions.
• Embed AGOD into the DRL SAC framework to produce the D2SAC algorithm.
• Use a human-aware utility function based on a content quality assessment (e.g., BRISQUE) and diffusion model-assisted decision making.
• Provide implementation and evaluation comparing D2SAC to seven DRL baselines across ASP selection and control tasks.

제안 방법

• Formulate ASP selection as a resource-constrained, online, discrete optimization problem with human-aware utility.
• Develop the AGOD algorithm that uses a diffusion process to generate discrete decision distributions conditioned on environment states.
• Describe the forward probability noising process and the reverse denoising process to obtain optimal decision distributions.
• Embed AGOD into the DRL SAC framework to produce the D2SAC algorithm.
• Use a human-aware utility function based on a content quality assessment (e.g., BRISQUE) and diffusion model-assisted decision making.
• Provide implementation and evaluation comparing D2SAC to seven DRL baselines across ASP selection and control tasks.

Figure 1: The architecture of AIGC-as-a-Service in wireless edge networks. Part A is demo of AIGC service based on Stable Diffusion v1.5 as an example of deployable AIGC model for edge servers; Part B is network architecture of ASPs employing edge servers to deploy AIGC models for providing AaaS to

실험 결과

연구 질문

• RQ1 How can diffusion models be used to generate optimal discrete decisions for ASP selection under resource constraints and uncertainty?
• RQ2 Can a diffusion-based augmentation of DRL (D2SAC) outperform standard DRL algorithms for edge-based AIGC service provisioning?
• RQ3 What is the impact of human-aware content quality metrics on ASP selection and user utilities?
• RQ4 How well does the proposed approach generalize to other optimization problems in wireless networks?

주요 결과

• D2SAC outperforms seven representative DRL algorithms on ASP selection and standard control tasks.
• AGOD generates discrete decision distributions by reversing a diffusion process conditioned on environment states.
• The AaaS architecture enables edge deployment of AIGC models to provide ubiquitous services with human-centric optimization.
• The human-aware utility function links AIGC model output quality to resource-aware decision making.
• The approach is extensible to broader wireless-network optimization problems beyond ASP selection.

Figure 2: Energy cost versus diffusion steps for stable-diffusion-v1-4 model inference.