JultikaUniversity of Oulu repository

Abstract

This dissertation offers a unique federated continual learning setup for anomaly detection in the fast growing 5G Open Radio Access Network (O-RAN) environment. Conventional AI techniques frequently fall short of meeting the security automation needs of 5G networks, owing to their outstanding latency, dependability, and bandwidth demands. As a result, the thesis provides an anomaly detection system that does not only use federated learning (FL) to solve inherent privacy problems and resource constraints but also incorporates replay buffer concept in the training phase of the model to eradicate catastrophic forgetting. To allow the intended federated learning architecture, anomaly detectors are incorporated into the Near-real time RIC, while aggregation servers are installed within the Non-real time RIC. The configuration was carefully tested using the 5G NIDD Dataset, revealing a considerable boost in detection accuracy by reaching close to 99% for almost all datasets after including the continual learning process. The thesis also investigates the notion of transfer learning, in which pre-trained local models are evaluated against a hybrid Application layer DDoS dataset that includes benign samples from the CICIDS 2017 dataset and attack flows generated in proprietary SDN environment. The captured results show almost over 99% of accuracy, confirming the suggested system’s efficacy and flexibility. The study represents a significant step forward in the development of a more secure, efficient, and privacy-protecting 5G network architecture.