University of Oulu

M. Krouka, A. Elgabli, C. B. Issaid and M. Bennis, "Communication-Efficient Federated Learning: A Second Order Newton-Type Method With Analog Over-the-Air Aggregation," in IEEE Transactions on Green Communications and Networking, vol. 6, no. 3, pp. 1862-1874, Sept. 2022, doi: 10.1109/TGCN.2022.3173420

Communication-efficient federated learning : a second order Newton-type method with analog over-the-air aggregation

Saved in:
Author: Krouka, Mounssif1; Elgabli, Anis1; Issaid, Chaouki Ben1;
Organizations: 1Center of Wireless Communications, University of Oulu, 90014 Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 1.3 MB)
Persistent link: http://urn.fi/urn:nbn:fi-fe2019080123354
Language: English
Published: Institute of Electrical and Electronics Engineers, 2022
Publish Date: 2022-10-03
Description:

Abstract

Owing to their fast convergence, second-order Newton-type learning methods have recently received attention in the federated learning (FL) setting. However, current solutions are based on communicating the Hessian matrices from the devices to the parameter server, at every iteration, incurring a large number of communication rounds; calling for novel communication-efficient Newton-type learning methods. In this article, we propose a novel second-order Newton-type method that, similarly to its first-order counterpart, requires every device to share only a model-sized vector at each iteration while hiding the gradient and Hessian information. In doing so, the proposed approach is significantly more communication-efficient and privacy-preserving. Furthermore, by leveraging the over-the-air aggregation principle, our method inherits privacy guarantees and obtains much higher communication efficiency gains. In particular, we formulate the problem of learning the inverse Hessian-gradient product as a quadratic problem that is solved in a distributed way. The framework alternates between updating the inverse Hessian-gradient product using a few alternating direction method of multipliers (ADMM) steps, and updating the global model using Newton’s method. Numerical results show that our proposed approach is more communication-efficient and scalable under noisy channels for different scenarios and across multiple datasets.

see all

Series: IEEE transactions on green communications and networking
ISSN: 2473-2400
ISSN-E: 2473-2400
ISSN-L: 2473-2400
Volume: 6
Issue: 3
Pages: 1862 - 1874
DOI: 10.1109/tgcn.2022.3173420
OADOI: https://oadoi.org/10.1109/tgcn.2022.3173420
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Subjects:
Funding: This work was supported in part by the Academy of Finland 6G Flagship under Grant 318927; in part by Project SMARTER; in part by Projects EU-ICT IntellIoT and EUCHISTERA LearningEdge; and in part by CONNECT, Infotech-NOOR, and NEGEIN
Academy of Finland Grant Number: 318927
Detailed Information: 318927 (Academy of Finland Funding decision)
Copyright information: © 2022 The Authors. This work is licensed under a Creative Commons Attribution 4.0 License. For more information, see https://creativecommons.org/licenses/by/4.0/.
  https://creativecommons.org/licenses/by/4.0/