University of Oulu

Ben Issaid, C., Elgabli, A., & Bennis, M. (2022). DR-DSGD: A distributionally robust decentralized learning algorithm over graphs. Transactions on Machine Learning Research, 2022(8), 1-25.

DR-DSGD : a distributionally robust decentralized learning algorithm over graphs

Saved in:
Author: Ben Issaid, Chaouki1; Elgabli, Anis1; Bennis, Mehdi1
Organizations: 1Centre for Wireless Communications, University of Oulu, Finland
Format: article
Version: published version
Access: open
Online Access: PDF Full Text (PDF, 1.7 MB)
Persistent link:
Language: English
Published: Transactions on Machine Learning Research, 2022
Publish Date: 2023-06-14


In this paper, we propose to solve a regularized distributionally robust learning problem in the decentralized setting, taking into account the data distribution shift. By adding a Kullback-Liebler regularization function to the robust min-max optimization problem, the learning problem can be reduced to a modified robust minimization problem and solved efficiently. Leveraging the newly formulated optimization problem, we propose a robust version of Decentralized Stochastic Gradient Descent (DSGD), coined Distributionally Robust Decentralized Stochastic Gradient Descent (DR-DSGD). Under some mild assumptions and provided that the regularization parameter is larger than one, we theoretically prove that DR-DSGD achieves a convergence rate of O(1/√KT + K/T), where K is the number of devices and T is the number of iterations. Simulation results show that our proposed algorithm can improve the worst distribution test accuracy by up to 10%. Moreover, DR-DSGD is more communication-efficient than DSGD since it requires fewer communication rounds (up to 20 times less) to achieve the same worst distribution test accuracy target. Furthermore, the conducted experiments reveal that DR-DSGD results in a fairer performance across devices in terms of test accuracy.

see all

Series: Transactions on machine learning research
Volume: 2022
Issue: 8
Pages: 1 - 25
Type of Publication: D1 Article in a trade journal
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This work was supported in part by the Academy of Finland 6G Flagship under grant No. 318927, in part by project SMARTER, in part by projects EU-ICT IntellIoT under grant No. 957218, EUCHISTERA LearningEdge, CONNECT, Infotech-NOOR, and NEGEIN.
EU Grant Number: (957218) IntellIoT - Intelligent, distributed, human-centered and trustworthy IoT environments
Academy of Finland Grant Number: 318927
Detailed Information: 318927 (Academy of Finland Funding decision)
Copyright information: TMLR makes all published content immediately available to the public free of charge.