University of Oulu

M. Krouka, A. Elgabli, C. B. Issaid and M. Bennis, "Communication-Efficient Second-Order Newton-Type Approach for Decentralized Learning," 2023 IEEE Wireless Communications and Networking Conference (WCNC), Glasgow, United Kingdom, 2023, pp. 1-7, doi: 10.1109/WCNC55385.2023.10118646

Communication-efficient second-order Newton-type approach for decentralized learning

Saved in:
Author: Krouka, Mounssif1; Elgabli, Anis1; Issaid, Chaouki Ben1;
Organizations: 1Centre for Wireless Communications (CWC), University of Oulu, Finland
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 0.9 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2023
Publish Date: 2023-10-03


In this paper, we propose a decentralized Newton-type approach to solve the problem of decentralized federated learning (FL). Notably, our proposed algorithm leverages the fast convergence of the second-order methods while avoid sending the hessian matrix at each iteration. Therefore, the proposed approach significantly reduces the communication cost and preserves the privacy. Specifically, we alternate between two problems. The inner problem approximates the inverse Hessian-gradient product which is formulated as a quadratic optimization problem and approximately solved in a decentralized manner using one step of the group alternating direction method of multipliers (GADMM) method. The outer problem learns the model, which is solved by performing one decentralized Newton step at every iteration. Moreover, to reduce the communication-overhead per iteration, a quantized version (leveraging stochastic quantization) is also proposed. Simulation results illustrate that our algorithm outperforms the baselines of GADMM, Q-GADMM, Newton tracking, and Decentralized SGD, and provides energy and communication-efficient solutions for bandwidth-limited systems under different SNR regimes.

see all

Series: IEEE Wireless Communications and Networking Conference
ISSN: 1525-3511
ISSN-E: 1558-2612
ISSN-L: 1525-3511
ISBN: 978-1-6654-9122-8
ISBN Print: 978-1-6654-9123-5
DOI: 10.1109/WCNC55385.2023.10118646
Host publication: 2023 IEEE Wireless Communications and Networking Conference (WCNC) : proceedings
Conference: IEEE Wireless Communications and Networking Conference
Type of Publication: A4 Article in conference proceedings
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This work is supported by the European Union’s Horizon Europe program through the project ADROIT6G, CENTRIC, and DESIRE6G.
Copyright information: © 2023 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.