University of Oulu

M. Krouka, A. Elgabli, C. B. Issaid and M. Bennis, "Communication-Efficient and Federated Multi-Agent Reinforcement Learning," in IEEE Transactions on Cognitive Communications and Networking, vol. 8, no. 1, pp. 311-320, March 2022, doi: 10.1109/TCCN.2021.3130993

Communication-efficient and federated multi-agent reinforcement learning

Saved in:
Author: Krouka, Mounssif1; Elgabli, Anis1; Issaid, Chaouki Ben1;
Organizations: 1Centre of Wireless Communications, University of Oulu, 90014 Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 22.2 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe2022012811205
Language: English
Published: IEEE Communications Society, 2022
Publish Date: 2022-01-28
Description:

Abstract

In this paper, we consider a distributed reinforcement learning setting where agents are communicating with a central entity in a shared environment to maximize a global reward. A main challenge in this setting is that the randomness of the wireless channel perturbs each agent’s model update while multiple agents’ updates may cause interference when communicating under limited bandwidth. To address this issue, we propose a novel distributed reinforcement learning algorithm based on the alternating direction method of multipliers (ADMM) and “over air aggregation” using analog transmission scheme, referred to as A-RLADMM. Our algorithm incorporates the wireless channel into the formulation of the ADMM method, which enables agents to transmit each element of their updated models over the same channel using analog communication. Numerical experiments on a multi-agent collaborative navigation task show that our proposed algorithm significantly outperforms the digital communication baseline of A-RLADMM (DRLADMM), the lazily aggregated policy gradient (RL-LAPG), as well as the analog and the digital communication versions of the vanilla FL, (A-FRL) and (D-FRL) respectively.

see all

Series: IEEE transactions on cognitive communications and networking
ISSN: 2372-2045
ISSN-E: 2332-7731
ISSN-L: 2372-2045
Volume: 8
Issue: 1
Pages: 311 - 320
DOI: 10.1109/TCCN.2021.3130993
OADOI: https://oadoi.org/10.1109/TCCN.2021.3130993
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Subjects:
Funding: This work is supported by Academy of Finland 6G Flagship (grant no. 318927) and project SMARTER, projects EU-ICT IntellIoT and EUCHISTERA LearningEdge, and CONNECT, Infotech-NOOR, and NEGEIN.
Academy of Finland Grant Number: 318927
Detailed Information: 318927 (Academy of Finland Funding decision)
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.