University of Oulu

A. Elgabli, J. Park, S. Ahmed and M. Bennis, "L-FGADMM: Layer-Wise Federated Group ADMM for Communication Efficient Decentralized Deep Learning," 2020 IEEE Wireless Communications and Networking Conference (WCNC), Seoul, Korea (South), 2020, pp. 1-6, doi: 10.1109/WCNC45663.2020.9120758

L-FGADMM : layer-wise federated group ADMM for communication efficient decentralized deep learning

Saved in:
Author: Elgabli, Anis1; Park, Jihong1; Ahmed, Sabbir1;
Organizations: 1University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.8 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2020
Publish Date: 2020-06-19


This article proposes a communication-efficient decentralized deep learning algorithm, coined layer-wise federated group ADMM (L-FGADMM). To minimize an empirical risk, every worker in L-FGADMM periodically communicates with two neighbors, in which the periods are separately adjusted for different layers of its deep neural network. A constrained optimization problem for this setting is formulated and solved using the stochastic version of GADMM proposed in our prior work. Numerical evaluations show that by less frequently exchanging the largest layer, L-FGADMM can significantly reduce the communication cost, without compromising the convergence speed. Surprisingly, despite less exchanged information and decentralized operations, intermittently skipping the largest layer consensus in L-FGADMM creates a regularizing effect, thereby achieving the test accuracy as high as federated learning (FL), a baseline method with the entire layer consensus by the aid of a central entity.

see all

Series: IEEE Wireless Communications and Networking Conference
ISSN: 1525-3511
ISSN-E: 1558-2612
ISSN-L: 1525-3511
ISBN: 978-1-7281-3106-1
ISBN Print: 978-1-7281-3107-8
Pages: 1 - 6
Article number: 19710664
DOI: 10.1109/WCNC45663.2020.9120758
Host publication: 2020 IEEE Wireless Communications and Networking Conference (WCNC)
Conference: IEEE Wireless Communications and Networking Conference
Type of Publication: A4 Article in conference proceedings
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This research was supported in part by Academy of Finland (Grant Nr. 294128), in part by the 6Genesis Flagship (Grant Nr. 318927), in part by the Kvantum Institute Strategic Project (SAFARI), and in part by the Academy of Finland thorough the MISSION Project (Grant Nr. 319759).
Academy of Finland Grant Number: 294128
Detailed Information: 294128 (Academy of Finland Funding decision)
318927 (Academy of Finland Funding decision)
319759 (Academy of Finland Funding decision)
Copyright information: © 2020 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.