GSOC2024RLUsability5G

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

Back to GSoC 2024 projects

Project Overview

• Project Name: Enhancement of RL Approach Accessibility in NR
• Student: Hyerin Kim
• Mentors: Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
• Google page: https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
• Project Goals: In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
• Repository: https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
• About Me: I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

Milestones

The planned milestones are outlined below

Phase1. Design example (3 weeks)

• Familiar with 5g-lena (2 weeks)
• Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) (1 week)

- Define Assumption (e.g., delay, TDMA/OFDMA, …)

Phase2. Design RL based Scheduler (6 +1 weeks)

• Design scheduler (2 weeks)

- input/output

- goal of optimization

• Design RL process (1 week)

- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity

• Implementation of RL based scheduler in 5g lena (3 +1 weeks)

- Create the test

- Create documentation

- Create MR to 5g lena (Milestone 1)

Phase3. RL Integration (4 weeks)

• Design RL framework (1 week)

- Define RL technique

• Develop gym scripts (3 weeks)

- Develop gym python scripts

- Develop ns3 gym interface in RL 5g lena example

- Validate RL process of the example

- Create MR to 5g lena (Milestone 2)

Phase4. Evaluation (3 +1 weeks)

• Evaluate the result of example compared with other schedulers

- Write simulation campaign scripts

- Execute scripts

- Plotting python scripts

• Address review comment of the MR 1 and 2
• Update MR 1 and 2 with necessary modification
• Create brief description of the work and the results for 5g lena blog (Milestone 3)

Weekly Report

Week 1 [May 27 - Jun. 02]

Familiar with 5g-lena (1)

• Studied cttc-nr-demo example with cttc-nr-demo tutorial

৹ quasi-ideal assumption

৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)

৹ How to configure the scenario, EPC, physical layer, and traffic(application)

৹ (DOUBT) What do I consider when I configure bands, carrier component, BWP?

Week 2 [Jun. 03 - Jun. 09]

Familiar with 5g-lena (2)

• Organized the concepts of Band, CC, and BWP to address the issue from the previous week

৹ Page: Band, Carrier Component (CC), Bandwidth Part (BWP)

• Studied schedulers in 5g-lena

৹ Studied MAC Layer section related to schedulers in NR module documentation

৹ Studied cttc-nr-simple-qos-sched example

Week 3 [Jun. 10 - Jun. 16]

Design Scenario

• Analyzed the QoS Scheduler in 5g-lena

৹ Page: QoS Scheduler Simulation Results Analysis

৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

• Created an initial example code

৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'

৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively

৹ Commit: Add new example cttc-nr-initial-my-sched

Week 4 [Jun. 17 - Jun. 23]

Design Scheduler (1)

• Modified the initial example

৹ Added a new parameter `numTrafficProfile`

‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3

‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic

‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.

৹ Commit: Add a parameter for the number of traffic types

• Evaluated the QoS scheduler in various scenarios

৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week

৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic

৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.

৹ Page: QoS Scheduler Evaluation

• Designed a new scheduler

৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)

৹ Decided the goals of the scheduler

‣ Option 1: Minimize the total delay of the gNB

‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

Week 5 [Jun. 24 - Jun. 30]

Design Scheduler (2)

• The scheduler is designed to be user-friendly and easy to reuse

৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable

৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE

‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic

‣ To address this issue, the designed scheduler focuses on minimizing the total delay

৹ Page: Scheduler Design

৹ TBD: the reward function

Week 6 [Jul. 01 - Jul. 07]

Design RL Process

• Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym

৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`

৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process

‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE

‣ However, the RL-based scheduler should transfer data, including data about all UEs

‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs

৹ Diagram: AI Scheduler Diagram (Google Docs, draw.io)

Week 7 [Jul. 08 - Jul. 14]

Implementation of RL-based scheduler in 5g lena (1)

• Created a MR to share the development progress
• Developed the skeleton of the RL-based scheduler on the 5g-lena side

৹ Created the class skeleton

‣ (THOUGHT) The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?

৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class

৹ Managed a week's work as a milestone. The issues pertaining to this milestone are as follows:

‣ Create class skeleton

‣ Develop AssignDLRBG function

• Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`

৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.

৹ (DISCUSSION): When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

Week 8 [Jul. 15 - Jul. 21]

Implementation of RL-based scheduler in 5g lena (2)

• Developed the RL-based scheduler on the 5g-lena side

৹ Developed a class for UE information, including the data needed for RL-based scheduling

৹ Developed classes for RL-based resource allocation in OFDMA and TDMA

৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler

• Managed a week's work as a milestone. The issues pertaining to this milestone are as follows:

৹ Develop NrMacSchedulerUeInfoAI class

৹ Develop NrMacSchedulerOfdmaAI class

৹ Develop NrMacSchedulerTdmaAI class

৹ Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"

• Created draft documentation corresponding to the developed code

৹ Commits: Add draft descriptions for RL-based scheduler, Revise descriptions for RL-based scheduler

• (DISCUSSION): Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

Week 8 [Jul. 22 - Jul. 26]

Implementation of RL-based scheduler in 5g lena (3)

Code Refactoring based on the comments in MR

• Remove duplicate functions

৹ Remove functions that are duplicated with the classes related to QoS

• Improve the clarity and detail of structures

৹ Change the structure of observation

৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated

• Improve the Doxygen documentation
• Fix CI/CD errors