[Paper Review] Task-Based Information Compression for Multi-Agent Communication Problems with Channel Rate Constraints.

Arsham Mostaani, Thang X. Vu+2 more|arXiv (Cornell University)|May 28, 2020

Distributed Control Multi-Agent Systems50 references4 citations

TL;DR

This paper proposes task-based information compression for multi-agent systems under communication rate constraints, using two schemes: Learning-based Information Compression (LBIC) via reinforcement learning and State Aggregation for Information Compression (SAIC) with analytical design. The methods minimize reward loss by compactly communicating observations, with SAIC conditionally achieving optimal performance, outperforming benchmarks in a rendezvous task.

ABSTRACT

A collaborative task is assigned to a multiagent system (MAS) in which agents are allowed to communicate. The MAS runs over an underlying Markov decision process and its task is to maximize the averaged sum of discounted one-stage rewards. Although knowing the global state of the environment is necessary for the optimal action selection of the MAS, agents are limited to individual observations. The inter-agent communication can tackle the issue of local observability, however, the limited rate of the inter-agent communication prevents the agent from acquiring the precise global state information. To overcome this challenge, agents need to communicate their observations in a compact way such that the MAS compromises the minimum possible sum of rewards. We show that this problem is equivalent to a form of rate-distortion problem which we call the task-based information compression. We introduce two schemes for task-based information compression (i) Learning-based information compression (LBIC) which leverages reinforcement learning to compactly represent the observation space of the agents, and (ii) State aggregation for information compression (SAIC), for which a state aggregation algorithm is analytically designed. The SAIC is shown, conditionally, to be capable of achieving the optimal performance in terms of the attained sum of discounted rewards. The proposed algorithms are applied to a rendezvous problem and their performance is compared with two benchmarks; (i) conventional source coding algorithms and the (ii) centralized multiagent control using reinforcement learning. Numerical experiments confirm the superiority of the proposed algorithms.

Motivation & Objective

• To address the challenge of limited inter-agent communication rates in multi-agent systems (MAS) with partial observability.
• To minimize the loss in discounted sum of rewards due to compressed communication in MAS under channel rate constraints.
• To develop communication strategies that compactly represent agent observations while preserving task-relevant global state information.
• To design a theoretically grounded method (SAIC) that can achieve optimal performance under specific conditions.
• To evaluate and compare the proposed schemes against conventional source coding and centralized RL benchmarks.

Proposed method

• Formulates the communication problem as a task-based rate-distortion problem, where distortion is measured by reward loss.
• Introduces Learning-based Information Compression (LBIC), using deep reinforcement learning to learn compact representations of agent observations.
• Proposes State Aggregation for Information Compression (SAIC), which analytically groups states to minimize information loss under rate constraints.
• Applies SAIC under the assumption that the underlying MDP allows for sufficient state aggregation to preserve task-relevant information.
• Uses a Markov decision process framework to model the MAS, with agents maximizing the sum of discounted rewards.
• Employs a centralized training with decentralized execution (CTDE) paradigm to train LBIC and evaluate performance.

Experimental results

Research questions

• RQ1 Can task-based information compression reduce reward loss in multi-agent systems under communication rate constraints?
• RQ2 How does LBIC, trained via reinforcement learning, compare to conventional source coding in communication efficiency and reward performance?
• RQ3 Under what conditions can SAIC achieve optimal performance in terms of discounted reward?
• RQ4 How does the performance of the proposed schemes compare to centralized multi-agent reinforcement learning?
• RQ5 What is the trade-off between communication rate and reward loss in multi-agent coordination tasks?

Key findings

• The proposed task-based information compression framework effectively reduces reward loss under communication rate constraints.
• SAIC conditionally achieves optimal performance in terms of the sum of discounted rewards, under specific structural assumptions of the MDP.
• LBIC outperforms conventional source coding algorithms in both communication efficiency and reward attainment.
• Both LBIC and SAIC significantly outperform centralized multi-agent reinforcement learning in terms of communication efficiency and reward performance under rate limits.
• Numerical experiments on a rendezvous task confirm the superiority of the proposed schemes over benchmarks in minimizing reward loss.
• The rate-distortion formulation of the problem enables a principled trade-off between communication cost and task performance.