GSOC2024RLUsability5GFinalReport
Project Overview
- Project Name: Enhancement of RL Approach Accessibility in NR
- Student: Hyerin Kim
- Mentors: Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
- Project Goals: In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
- Project Proposal: https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
- Project Wiki: GSOC2024RLUsability5G
Merge Requests and Commits
I maintained a single branch for all work during GSoC: gsoc24-nr-usability
Merge Requests
All the following activities can be easily reviewed on the following MR:
| No. | Name | Status |
|---|---|---|
| [1] | Draft: GSoC2024: RL-based Scheduler | Draft |
Milestones
During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability" by organizing them into milestones and issues
| No. | Name | Period |
|---|---|---|
| [1] | Design Scenario | Jun 10, 2024–Jun 17, 2024 |
| [2] | Drafting an AI scheduler | Jul 11, 2024–Jul 17, 2024 |
| [3] | Develop an RL-based scheduler | Jul 18, 2024–Jul 24, 2024 |
| [4] | Update the RL-based Scheduler (Code Refactoring) | Jul 24, 2024–Jul 31, 2024 |
| [5] | Create test | Jul 29, 2024–Aug 11, 2024 |
| [6] | Develop Gym Interface in "cttc-nr-rl-based-sched" Example | Aug 12, 2024–Aug 25, 2024 |
| [7] | Develop Gym Python Scripts | Aug 21, 2024–Oct 2, 2024 |
| [8] | Resolve comments in MR: cttc-lena/nr!166 | Sep 2, 2024–Oct 6, 2024 |
Project Details
Phase 1: Design example
During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and cttc-nr-demo tutorial, and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and NR module documentation.
Based on this studies, I designed draft scenario which be applied RL-based scheduler.
Activities
- Familiar with 5g-lena (Week 1, Week 2)
- Design Scenario (Week 3) (Milestone 1)
Phase 2: Design RL based Scheduler
During this phrase, I designed RL-based scheduler which can be user-friendly and easy to reuse. Also, set parameters for the number of traffic and traffic types on the example to provide various scenarios to users.
The goal of scheduler is decided based on the problems of the QoS scheduler with QoS LC assignment that the non-GBR flow with DC-GRB flow in a UE have benefit and a single non-GBR flow suffers from starvation due to this benefit. Also, we considered the structure of scheduler is easy to understand.
After setting the goal of RL-base scheduler, I designed how to implement the designed RL-based scheduler in 5g-lena and ns3-gym. To design the RL process, I drew the UML diagram that represents the relationship between scheduler classes and the sequences among methods in the classes. While proceeding this work, I refactored the code based on the review on MR 1.
I developed the process of AI schedulers side of 5G-LENA, that processes data from active UEs and sends it via Notify callback to Ns3-Gym. To verify the code, I also created the unit test which verifies the functionality of the callback used for invoking the ns3-gym module during the resource assigning process.
Activities
- Design Scheduler (Week 4-5)
- Design RL Process (Week 6)
- Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
- Refactor code (Week 9) (Milestone 4)
- Create Unit Test (Week 10-11) (Milestone 5)
Phase 3: RL Integration
During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface 'NrMacSchedulerAiNs3GymEnv' which convert the data from the AI scheduler to ns3-gym formatted data and transfer it to RL model through 'OpenGymInterface'. This interface also converted the selected action from the RL model to the structure 'Weight' defined in 'NrMacSchedulerUeInfoAi'.
After developing the interface, I developed the Python scripts which train RL models using observation and reward from the AI scheduler. I provided two examples: the simple test example which uses the default Ns3Env for training the model, and the PPO test example which uses the Proximal Policy Optimization (PPO) model under the Ns3Env.
Activities
- Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
- Develop a python gym script for a simple test (Week 13) (Milestone 7)
- Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)
Phase 4: Refine Code and Evaluation
In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.
Activities
- Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
- Conduct Simulation Campaign (Week 17-18)
- Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
- Squash commits (Week 19)