DR-DSGD : a distributionally robust decentralized learning algorithm over graphs
|Author:||Ben Issaid, Chaouki1; Elgabli, Anis1; Bennis, Mehdi1|
1Centre for Wireless Communications, University of Oulu, Finland
|Online Access:||PDF Full Text (PDF, 1.7 MB)|
|Persistent link:|| http://urn.fi/urn:nbn:fi-fe2023061454708
Transactions on Machine Learning Research,
|Publish Date:|| 2023-06-14
In this paper, we propose to solve a regularized distributionally robust learning problem in the decentralized setting, taking into account the data distribution shift. By adding a Kullback-Liebler regularization function to the robust min-max optimization problem, the learning problem can be reduced to a modified robust minimization problem and solved efficiently. Leveraging the newly formulated optimization problem, we propose a robust version of Decentralized Stochastic Gradient Descent (DSGD), coined Distributionally Robust Decentralized Stochastic Gradient Descent (DR-DSGD). Under some mild assumptions and provided that the regularization parameter is larger than one, we theoretically prove that DR-DSGD achieves a convergence rate of O(1/√KT + K/T), where K is the number of devices and T is the number of iterations. Simulation results show that our proposed algorithm can improve the worst distribution test accuracy by up to 10%. Moreover, DR-DSGD is more communication-efficient than DSGD since it requires fewer communication rounds (up to 20 times less) to achieve the same worst distribution test accuracy target. Furthermore, the conducted experiments reveal that DR-DSGD results in a fairer performance across devices in terms of test accuracy.
Transactions on machine learning research
|Pages:||1 - 25|
|Type of Publication:||
D1 Article in a trade journal
|Field of Science:||
213 Electronic, automation and communications engineering, electronics
This work was supported in part by the Academy of Finland 6G Flagship under grant No. 318927, in part by project SMARTER, in part by projects EU-ICT IntellIoT under grant No. 957218, EUCHISTERA LearningEdge, CONNECT, Infotech-NOOR, and NEGEIN.
|EU Grant Number:||
(957218) IntellIoT - Intelligent, distributed, human-centered and trustworthy IoT environments
|Academy of Finland Grant Number:||
318927 (Academy of Finland Funding decision)
TMLR makes all published content immediately available to the public free of charge.