University of Oulu

X. Wang, R. Li, C. Wang, X. Li, T. Taleb and V. C. M. Leung, "Attention-Weighted Federated Deep Reinforcement Learning for Device-to-Device Assisted Heterogeneous Collaborative Edge Caching," in IEEE Journal on Selected Areas in Communications, vol. 39, no. 1, pp. 154-169, Jan. 2021, doi: 10.1109/JSAC.2020.3036946

Attention-weighted federated deep reinforcement learning for device-to-device assisted heterogeneous collaborative edge caching

Saved in:
Author: Wang, Xiaofei1; Li, Ruibin1; Wang, Chenyang1;
Organizations: 1College of Intelligence and Computing, Tianjin University, Tianjin, 300072 China
2Key Laboratory of Dependable Service Computing in Cyber Physical Society (Chongqing University), Ministry of Education, China
3School of Big Data & Software Engineering, Chongqing University, Chongqing, 401331 China
4Department of Communications and Networking, School of Electrical Engineering, Aalto University, 02150 Espoo, Finland
5Information Technology and Electrical Engineering, Oulu University, 90570 Oulu, Finland
6Department of Computer and Information Security, Sejong University, Seoul, 05006 South Korea
7College of Computer Science & Software Engineering, Shenzhen University, Shenzhen, 518060 China
8Department of Electrical and Computer Engineering, The University of British Columbia, Vancouver, V6T 1Z4 Canada
Format: article
Version: accepted version
Access: open
Online Access: PDF Full Text (PDF, 13.5 MB)
Persistent link:
Language: English
Published: Institute of Electrical and Electronics Engineers, 2021
Publish Date: 2021-03-29


In order to meet the growing demands for multimedia service access and release the pressure of the core network, edge caching and device-to-device (D2D) communication have been regarded as two promising techniques in next generation mobile networks and beyond. However, most existing related studies lack consideration of effective cooperation and adaptability to the dynamic network environments. In this article, based on the flexible trilateral cooperation among user equipment, edge base stations and a cloud server, we propose a D2D-assisted heterogeneous collaborative edge caching framework by jointly optimizing the node selection and cache replacement in mobile networks. We formulate the joint optimization problem as a Markov decision process, and use a deep Q-learning network to solve the long-term mixed integer linear programming problem. We further design an attention-weighted federated deep reinforcement learning (AWFDRL) model that uses federated learning to improve the training efficiency of the Q-learning network by considering the limited computing and storage capacity, and incorporates an attention mechanism to optimize the aggregation weights to avoid the imbalance of local model quality. We prove the convergence of the corresponding algorithm, and present simulation results to show the effectiveness of the proposed AWFDRL framework in reducing average delay of content access, improving hit rate and offloading traffic.

see all

Series: IEEE journal on selected areas in communications
ISSN: 0733-8716
ISSN-E: 1558-0008
ISSN-L: 0733-8716
Volume: 39
Issue: 1
Pages: 154 - 169
DOI: 10.1109/JSAC.2020.3036946
Type of Publication: A1 Journal article – refereed
Field of Science: 213 Electronic, automation and communications engineering, electronics
Funding: This work is supported in part by the National Key R & D Program of China through Grants No. 2019YFB2101901, 2018YFC0809803 and 2018YFF0214700, National NSFC through Grants No. 62072332, 61902044, 62002260, 61672117 and 62072060, Chongqing Research Program of Basic Research and Frontier Technology (Grant No. cstc2019jcyj-msxmX0589), Fundamental Research Funds for the Central Universities (Grant No. 2020CDJQY-A022), the Academy of Finland Project CSN - under Grant Agreement 311654 and the 6Genesis project under Grant No. 318927, Chinese National Engineering Laboratory for Big Data System Computing Technology at Shenzhen University, and Canadian Natural Sciences and Engineering Research Council.
Academy of Finland Grant Number: 311654
Detailed Information: 311654 (Academy of Finland Funding decision)
318927 (Academy of Finland Funding decision)
Copyright information: © 2021 IEEE. Personal use of this material is permitted. Permission from IEEE must be obtained for all other uses, in any current or future media, including reprinting/republishing this material for advertising or promotional purposes, creating new collective works, for resale or redistribution to servers or lists, or reuse of any copyrighted component of this work in other works.