GSOC2024RLUsability5GFinalReport
Project Overview
- Project Name: Enhancement of RL Approach Accessibility in NR
- Student: Hyerin Kim
- Mentors: Katerina Koutlia, Amir Ashtari, Biljana Bojovic, Gabriel Ferreira
- Project Goals: This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
- Project Proposal: https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
- Project Wiki: GSOC2024RLUsability5G
Merge Requests and Commits
I maintained a single branch for all work during GSoC: gsoc24-nr-usability
Merge Requests
All the following activities can be easily reviewed on the following MR:
| No. | Name | Status |
|---|---|---|
| [1] | Draft: GSoC2024: RL-based Scheduler | Draft |
Milestones
During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability, by organizing them into milestones and issues, as defined during the planning of the project (Project Details).
| No. | Name | Period |
|---|---|---|
| [1] | Design Scenario | Jun 10, 2024–Jun 17, 2024 |
| [2] | Drafting an AI scheduler | Jul 11, 2024–Jul 17, 2024 |
| [3] | Develop an RL-based scheduler | Jul 18, 2024–Jul 24, 2024 |
| [4] | Update the RL-based Scheduler (Code Refactoring) | Jul 24, 2024–Jul 31, 2024 |
| [5] | Create test | Jul 29, 2024–Aug 11, 2024 |
| [6] | Develop Gym Interface in "cttc-nr-rl-based-sched" Example | Aug 12, 2024–Aug 25, 2024 |
| [7] | Develop Gym Python Scripts | Aug 21, 2024–Oct 2, 2024 |
| [8] | Resolve comments in MR: cttc-lena/nr!166 | Sep 2, 2024–Oct 6, 2024 |
Project Details
You can find detailed information about all activities on the wiki at GSOC2024RLUsability5G
Bonding Period - Phase 1: Design Example
During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and cttc-nr-demo tutorial, and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the NR module documentation.
Based on these studies, I designed a draft scenario to apply an RL-based scheduler.
Activities
- Familiar with 5G-LENA (Week 1-2)
- Design Scenario (Week 3) (Milestone 1)
Phase 2: Design RL based Scheduler
During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.
The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.
- The analysis of the QoS scheduler referenced the following paper: Katerina Koutlia, Sandra Lagen, and Biljana Bojovic. 2023. Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA. In Proceedings of the 2023 Workshop on ns-3 (WNS3 '23). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3592149.3592159
After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from MR.
I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.
Activities
- Design Scheduler (Week 4-5)
- Design RL Process (Week 6)
- Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
- Refactor code (Week 9) (Milestone 4)
- Create Unit Test (Week 10-11) (Milestone 5)
Phase 3: RL Integration
During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.
After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.
Activities
- Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
- Develop a python gym script for a simple test (Week 13) (Milestone 7)
- Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)
Phase 4: Refine Code and Evaluation
In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA", using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.
Activities
- Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
- Conduct Simulation Campaign (Week 17-18)
- Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
- Squash commits (Week 19)
My Experience
Acknowledgements
During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.
Challenges Faced
Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.
Suggestions for Future Work
It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.