GSOC2024RLUsability5G

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

Back to GSoC 2024 projects

Project Overview

• Project Name: Enhancement of RL Approach Accessibility in NR
• Student: Hyerin Kim
• Mentors: Amir Ashtari, Katerina Koutlia, Bijana Bojovic, Gabriel Ferreira
• Google page: https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
• Project Goals: In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
• Repository: https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
• About Me: I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

Milestones

The planned milestones are outlined below

Phase1. Design example (3 weeks)

• Familiar with 5g-lena (2 weeks)
• Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) (1 week)

- Define Assumption (e.g., delay, TDMA/OFDMA, …)

Phase2. Design RL based Scheduler (6 +1 weeks)

• Design scheduler (2 weeks)

- input/output

- goal of optimization

• Design RL process (1 week)

- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity

• Implementation of RL based scheduler in 5g lena (3 +1 weeks)

- Create the test

- Create documentation

- Create MR to 5g lena (Milestone 1)

Phase3. RL Integration (4 weeks)

• Design RL framework (1 week)

- Define RL technique

• Develop gym scripts (3 weeks)

- Develop gym python scripts

- Develop ns3 gym interface in RL 5g lena example

- Validate RL process of the example

- Create MR to 5g lena (Milestone 2)

Phase4. Evaluation (3 +1 weeks)

• Evaluate the result of example compared with other schedulers

- Write simulation campaign scripts

- Execute scripts

- Plotting python scripts

• Address review comment of the MR 1 and 2
• Update MR 1 and 2 with necessary modification
• Create brief description of the work and the results for 5g lena blog (Milestone 3)

Weekly Report

Week 1 [May 27 - Jun. 02]

Familiar with 5g-lena (1)

• Studied cttc-nr-demo example with cttc-nr-demo tutorial

৹ quasi-ideal assumption

৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)

৹ How to configure the scenario, EPC, physical layer, and traffic(application)

৹ (DOUBT) What do I consider when I configure bands, carrier component, BWP?

Week 2 [Jun. 03 - Jun. 09]

Familiar with 5g-lena (2)

• Organized the concepts of Band, CC, and BWP

৹ Page: Band, Carrier Component (CC), Bandwidth Part (BWP)

• Studied schedulers in 5g-lena

৹ Studied MAC Layer section related to schedulers in NR module documentation

৹ Studied cttc-nr-simple-qos-sched example

Week 3 [Jun. 10 - Jun. 16]

Design Scenario

• Analyzed the QoS Scheduler in 5g-lena

৹ Page: QoS Scheduler Simulation Results Analysis

৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

• Created an initial example code

৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'

৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively

৹ Commit: Add new example cttc-nr-initial-my-sched

Week 4 [Jun. 17 - Jun. 23]

Design Scheduler (1)

• Modified the initial example

৹ Added a new parameter `numTrafficProfile`

‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3

‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic

‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.

৹ Commit: Add a parameter for the number of traffic types

• Evaluated the QoS scheduler in various scenarios

৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week

৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic

৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.

৹ Page: QoS Scheduler Evaluation

• Designed a new scheduler

৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)

৹ Decided the goals of the scheduler

‣ Option 1: Minimize the total delay of the gNB

‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

Week 5 [Jun. 24 - Jun. 30]

Design Scheduler (2)

• The scheduler is designed to be user-friendly and easy to reuse

৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable

৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE

‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic

‣ To address this issue, the designed scheduler focuses on minimizing the total delay

৹ Page: Scheduler Design

৹ TBD: the reward function

Week 6 [Jul. 01 - Jul. 07]

Design RL Process

• Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym

৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`

৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process

‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE

‣ However, the RL-based scheduler should transfer data, including data about all UEs

‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs

৹ Diagram: AI Scheduler Diagram (Google Docs, draw.io)

Week 7 [Jul. 08 - Jul. 14]

Implementation of RL-based scheduler in 5g lena (1)

• Created a MR to share the development progress
• Developed the skeleton of the RL-based scheduler on the 5g-lena side

৹ Created the class skeleton

‣ (THOUGHT) The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?

৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class

৹ Issues:

‣ Create class skeleton

‣ Develop AssignDLRBG function