Nsnam - User contributions [en]

GSOC2024RLUsability5GFinalReport

2024-10-16T06:55:36Z

Mye280c37: /* Project Overview */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Biljana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

You can find detailed information about all activities on the wiki at [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

* The analysis of the QoS scheduler referenced the following paper: '''Katerina Koutlia, Sandra Lagen, and Biljana Bojovic. 2023. Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA. In Proceedings of the 2023 Workshop on ns-3 (WNS3 '23). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3592149.3592159'''

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T15:11:30Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

You can find detailed information about all activities on the wiki at [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

* The analysis of the QoS scheduler referenced the following paper: '''Katerina Koutlia, Sandra Lagen, and Biljana Bojovic. 2023. Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA. In Proceedings of the 2023 Workshop on ns-3 (WNS3 '23). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3592149.3592159'''

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T15:11:00Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

ou can find detailed information about all activities on the wiki at []

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

* The analysis of the QoS scheduler referenced the following paper: '''Katerina Koutlia, Sandra Lagen, and Biljana Bojovic. 2023. Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA. In Proceedings of the 2023 Workshop on ns-3 (WNS3 '23). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3592149.3592159'''

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T15:06:41Z

Mye280c37: /* Phase 2: Design RL based Scheduler */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

* The analysis of the QoS scheduler referenced the following paper: '''Katerina Koutlia, Sandra Lagen, and Biljana Bojovic. 2023. Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA. In Proceedings of the 2023 Workshop on ns-3 (WNS3 '23). Association for Computing Machinery, New York, NY, USA, 45–51. https://doi.org/10.1145/3592149.3592159'''

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T15:05:19Z

Mye280c37: /* Phase 2: Design RL based Scheduler */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

* The analysis of the QoS scheduler referenced the following paper: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. 'Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA.' Proceedings of the 2023 Workshop on ns-3. 2023.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T15:02:56Z

Mye280c37: /* Phase 4: Refine Code and Evaluation */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the example used in the paper [https://dl.acm.org/doi/abs/10.1145/3592149.3592159 "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA"], using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:59:27Z

Mye280c37: /* Phase 4: Refine Code and Evaluation */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR)
using the developed scenario known as 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:58:10Z

Mye280c37: /* Phase 3: RL Integration */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the action selected by the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi, which stores data for active UEs and their active flows. It then calculates weights to allocate resources and assesses the reward for resource assignment.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:54:23Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5G-LENA (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and ns3-gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to ns3-gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into ns3-gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:51:32Z

Mye280c37: /* Phase 1: Design example */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Bonding Period - Phase 1: Design Example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:50:59Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) ('''Milestone 1''')

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) ('''Milestone 2, 3''')
* Refactor code (Week 9) ('''Milestone 4''')
* Create Unit Test (Week 10-11) ('''Milestone 5''')

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) ('''Milestone 6''')
* Develop a python gym script for a simple test (Week 13) ('''Milestone 7''')
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) ('''Milestone 7''')

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) ('''Milestone 8''')
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) ('''Milestone 7, 8''')
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:50:05Z

Mye280c37: /* Milestones */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, '''as defined during the planning of the project (Project Details)'''.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:48:38Z

Mye280c37: /* Project Overview */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new Reinforcement Learning (RL)-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, as defined during the planning of the project (Project Details).

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:48:17Z

Mye280c37: /* Phase 2: Design RL based Scheduler */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new RL-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, as defined during the planning of the project (Project Details).

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the RL-based scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:47:52Z

Mye280c37: /* Milestones */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new RL-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, as defined during the planning of the project (Project Details).

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:47:39Z

Mye280c37: /* Milestones */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new RL-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, as defined during the planning of the project (Project Details).

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an RL-based scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T14:47:00Z

Mye280c37: /* Project Overview */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' This project targets the design and implementation of a new RL-based Mac scheduler for the nr module (5G-LENA) of ns-3, involving the integration with ns3-gym module. Additionally, the usability of 5G-LENA will be enhanced in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues, as defined during the planning of the project (Project Details).

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T13:01:11Z

Mye280c37: /* Phase 1: Design example */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena (Week 1-2)
* Design Scenario (Week 3) (Milestone 1)

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T13:00:29Z

Mye280c37: /* Milestones */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I made over 170 commits and later squashed them to around 70 commits for the merge. I managed all commits in my personal repository, named [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability], by organizing them into milestones and issues.

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:59:41Z

Mye280c37: /* Phase 4: Refine Code and Evaluation */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted a simulation campaign to evaluate all types of schedulers (i.e., AI, QoS, PF, RR) using the 'gsoc-nr-rl-based-sched' example. While running the simulations, I also refined the PPO model to improve its performance and clarity. Additionally, to compare the results of the AI scheduler, I updated the example to align with the 'cttc-nr-multi-flow-qos-sched' example, using two UEs: one with a single non-GBR flow, and the other with multiple flows, including both non-GBR and DC-GBR flows. Furthermore, I enhanced the code by incorporating feedback from the MR review.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:59:01Z

Mye280c37: /* Phase 3: RL Integration */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the NS3-Gym module under the designed scenario. I developed the NS3-Gym interface, NrMacSchedulerAiNs3GymEnv, which converts data from the AI scheduler into NS3-Gym formatted data and transfers it to the RL model through the OpenGymInterface. This interface also converts the selected action from the RL model into the Weight structure defined in NrMacSchedulerUeInfoAi.

After developing the interface, I created Python scripts to train RL models using the observation and reward from the AI scheduler. I provided two examples: a simple test example that uses the default Ns3Env for training the model, and a PPO test example that uses the Proximal Policy Optimization (PPO) model within the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:58:33Z

Mye280c37: /* Phase 2: Design RL based Scheduler */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phase, I designed an RL-based scheduler that is user-friendly and easy to reuse. I also set parameters for traffic volume and traffic types in the example to provide users with various scenarios.

The goal of the scheduler was defined based on the issues with the QoS scheduler, particularly the QoS LC assignment problem, where a non-GBR flow benefits when grouped with a DC-GBR flow in a UE, while a single non-GBR flow suffers from starvation. Additionally, we aimed to make the scheduler's structure easy to understand.

After setting the goal for the RL-based scheduler, I designed its implementation in 5G-LENA and NS3-Gym. To structure the RL process, I created a UML diagram that illustrates the relationships between scheduler classes and the sequences of methods within those classes. While working on this, I refactored the code based on feedback from [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR].

I developed the process for the AI scheduler side of 5G-LENA, which processes data from active UEs and sends it via the Notify callback to NS3-Gym. To verify the implementation, I also created a unit test to ensure the callback functionality worked correctly during the resource allocation process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface 'NrMacSchedulerAiNs3GymEnv' which convert the data from the AI scheduler to ns3-gym formatted data and transfer it to RL model through 'OpenGymInterface'. This interface also converted the selected action from the RL model to the structure 'Weight' defined in 'NrMacSchedulerUeInfoAi'.

After developing the interface, I developed the Python scripts which train RL models using observation and reward from the AI scheduler. I provided two examples: the simple test example which uses the default Ns3Env for training the model, and the PPO test example which uses the Proximal Policy Optimization (PPO) model under the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:57:30Z

Mye280c37: /* Phase 1: Design example */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5G-LENA by studying the cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and I analyzed the existing schedulers in 5G-LENA (i.e., QoS, PF, RR) through the cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and the [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on these studies, I designed a draft scenario to apply an RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phrase, I designed RL-based scheduler which can be user-friendly and easy to reuse. Also, set parameters for the number of traffic and traffic types on the example to provide various scenarios to users.

The goal of scheduler is decided based on the problems of the QoS scheduler with QoS LC assignment that the non-GBR flow with DC-GRB flow in a UE have benefit and a single non-GBR flow suffers from starvation due to this benefit. Also, we considered the structure of scheduler is easy to understand.

After setting the goal of RL-base scheduler, I designed how to implement the designed RL-based scheduler in 5g-lena and ns3-gym. To design the RL process, I drew the UML diagram that represents the relationship between scheduler classes and the sequences among methods in the classes. While proceeding this work, I refactored the code based on the review on [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR 1].

I developed the process of AI schedulers side of 5G-LENA, that processes data from active UEs and sends it via Notify callback to Ns3-Gym. To verify the code, I also created the unit test which verifies the functionality of the callback used for invoking the ns3-gym module during the resource assigning process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface 'NrMacSchedulerAiNs3GymEnv' which convert the data from the AI scheduler to ns3-gym formatted data and transfer it to RL model through 'OpenGymInterface'. This interface also converted the selected action from the RL model to the structure 'Weight' defined in 'NrMacSchedulerUeInfoAi'.

After developing the interface, I developed the Python scripts which train RL models using observation and reward from the AI scheduler. I provided two examples: the simple test example which uses the default Ns3Env for training the model, and the PPO test example which uses the Proximal Policy Optimization (PPO) model under the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:56:16Z

Mye280c37:

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on this studies, I designed draft scenario which be applied RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phrase, I designed RL-based scheduler which can be user-friendly and easy to reuse. Also, set parameters for the number of traffic and traffic types on the example to provide various scenarios to users.

The goal of scheduler is decided based on the problems of the QoS scheduler with QoS LC assignment that the non-GBR flow with DC-GRB flow in a UE have benefit and a single non-GBR flow suffers from starvation due to this benefit. Also, we considered the structure of scheduler is easy to understand.

After setting the goal of RL-base scheduler, I designed how to implement the designed RL-based scheduler in 5g-lena and ns3-gym. To design the RL process, I drew the UML diagram that represents the relationship between scheduler classes and the sequences among methods in the classes. While proceeding this work, I refactored the code based on the review on [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR 1].

I developed the process of AI schedulers side of 5G-LENA, that processes data from active UEs and sends it via Notify callback to Ns3-Gym. To verify the code, I also created the unit test which verifies the functionality of the callback used for invoking the ns3-gym module during the resource assigning process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface 'NrMacSchedulerAiNs3GymEnv' which convert the data from the AI scheduler to ns3-gym formatted data and transfer it to RL model through 'OpenGymInterface'. This interface also converted the selected action from the RL model to the structure 'Weight' defined in 'NrMacSchedulerUeInfoAi'.

After developing the interface, I developed the Python scripts which train RL models using observation and reward from the AI scheduler. I provided two examples: the simple test example which uses the default Ns3Env for training the model, and the PPO test example which uses the Proximal Policy Optimization (PPO) model under the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

= My Experience =

== Acknowledgements ==

During GSoC, I had experiences I had never encountered before. I received reviews on my code (e.g., structure, details) from experts, which greatly improved my ability to design and develop code and classes. The communication with mentors and the skill development that came through these interactions were the best parts of GSoC for me. Since the work was publicly accessible, I had to be more deliberate about the quality of my code compared to when working on private projects. This opportunity, along with the mentors' guidance, was an excellent chance for me to grow.

== Challenges Faced ==

Since this project is publicly accessible, I needed to provide more detailed descriptions of my work than I initially thought. Determining how detailed the documentation should be and how to structure it was the most challenging part.

== Suggestions for Future Work ==

It seems necessary to improve the algorithms for beam assignment in OFDMA, as well as the weight calculation and resource allocation algorithms for UEs with multiple flows. Additionally, having an NR-specific RRC module would be great, as it could enable more complex features like handover examples.

GSOC2024RLUsability5GFinalReport

2024-10-07T12:53:11Z

Mye280c37:

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on this studies, I designed draft scenario which be applied RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phrase, I designed RL-based scheduler which can be user-friendly and easy to reuse. Also, set parameters for the number of traffic and traffic types on the example to provide various scenarios to users.

The goal of scheduler is decided based on the problems of the QoS scheduler with QoS LC assignment that the non-GBR flow with DC-GRB flow in a UE have benefit and a single non-GBR flow suffers from starvation due to this benefit. Also, we considered the structure of scheduler is easy to understand.

After setting the goal of RL-base scheduler, I designed how to implement the designed RL-based scheduler in 5g-lena and ns3-gym. To design the RL process, I drew the UML diagram that represents the relationship between scheduler classes and the sequences among methods in the classes. While proceeding this work, I refactored the code based on the review on [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR 1].

I developed the process of AI schedulers side of 5G-LENA, that processes data from active UEs and sends it via Notify callback to Ns3-Gym. To verify the code, I also created the unit test which verifies the functionality of the callback used for invoking the ns3-gym module during the resource assigning process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the AI scheduler with the ns3-gym module under the designed scenario. I developed the ns3-gym interface 'NrMacSchedulerAiNs3GymEnv' which convert the data from the AI scheduler to ns3-gym formatted data and transfer it to RL model through 'OpenGymInterface'. This interface also converted the selected action from the RL model to the structure 'Weight' defined in 'NrMacSchedulerUeInfoAi'.

After developing the interface, I developed the Python scripts which train RL models using observation and reward from the AI scheduler. I provided two examples: the simple test example which uses the default Ns3Env for training the model, and the PPO test example which uses the Proximal Policy Optimization (PPO) model under the Ns3Env.

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

In this phase, I conducted simulation campaign that evaluates all types of schedulers (i.e., AI, QoS, PF, RR) under the example 'gsoc-nr-rl-based-sched'. While conducting simulation campaign, I also refined the PPO model to enhance the performance and clarity. Additionally, to compare the result of the AI scheduler, I updated the example aligning with `cttc-nr-multi-flow-qos-sched' example with 2 UE, one UE with a single non-GBR flow, and the other UE with multiple flows: non-GBR and DC-GBR flows. Additionally, I enhanced the code by reflecting the comments on the MR.

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

GSOC2024RLUsability5GFinalReport

2024-10-07T12:33:33Z

Mye280c37:

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on this studies, I designed draft scenario which be applied RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3]) ([https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Milestone 1])

== Phase 2: Design RL based Scheduler ==

During this phrase, I designed RL-based scheduler which can be user-friendly and easy to reuse. Also, set parameters for the number of traffic and traffic types on the example to provide various scenarios to users.

The goal of scheduler is decided based on the problems of the QoS scheduler with QoS LC assignment that the non-GBR flow with DC-GRB flow in a UE have benefit and a single non-GBR flow suffers from starvation due to this benefit. Also, we considered the structure of scheduler is easy to understand.

After setting the goal of RL-base scheduler, I designed how to implement the designed RL-based scheduler in 5g-lena and ns3-gym. To design the RL process, I drew the UML diagram that represents the relationship between scheduler classes and the sequences among methods in the classes. While proceeding this work, I refactored the code based on the review on [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR 1].

I developed the process of AI schedulers side of 5G-LENA, that processes data from active UEs and sends it via Notify callback to Ns3-Gym. To verify the code, I also created the unit test which verifies the functionality of the callback used for invoking the ns3-gym module during the resource assigning process.

=== Activities ===

* Design Scheduler (Week 4-5)
* Design RL Process (Week 6)
* Implementation of RL-based scheduler in 5g lena (Week 7-9) (Milestone 2, 3)
* Refactor code (Week 9) (Milestone 4)
* Create Unit Test (Week 10-11) (Milestone 5)

== Phase 3: RL Integration ==

During this phase, I integrated the

=== Activities ===

* Develop the ns3-gym interface in the RL 5G-LENA example (Week 12) (Milestone 6)
* Develop a python gym script for a simple test (Week 13) (Milestone 7)
* Develop a python gym script for Proximal Policy Optimization (PPO) model (Week 14) (Milestone 7)

== Phase 4: Refine Code and Evaluation ==

=== Activities ===

* Address comments in MR 1 (Week 15-16, Week 19) (Milestone 8)
* Conduct Simulation Campaign (Week 17-18)
* Refine Code and RL Models (Week 17-18) (Milestone 7, 8)
* Squash commits (Week 19)

GSOC2024RLUsability5GFinalReport

2024-10-07T12:01:08Z

Mye280c37: /* Activities */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on this studies, I designed draft scenario which be applied RL-based scheduler.

=== Activities ===
* Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2])
* Design Scenario ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_3_%5BJun._10_-_Jun._16%5D Week 3])

== Phase 2: Design RL based Scheduler ==

== Phase 3: RL Integration ==

== Phase 4: Refine Code and Evaluation ==

GSOC2024RLUsability5GFinalReport

2024-10-07T11:59:52Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

During this phase, I became familiar with 5g-lena by studying cttc-nr-demo example and [https://cttc-lena.gitlab.io/nr/cttc-nr-demo-tutorial.pdf cttc-nr-demo tutorial], and analyzed the existing schedulers in 5g-lena (i.e., QoS, PF, RR) through cttc-nr-simple-qos-sched example, cttc-nr-multi-flow-qos-sched example, and [https://cttc-lena.gitlab.io/nr/nrmodule.pdf NR module documentation].

Based on this studies, I designed draft scenario which be applied RL-based scheduler.

=== Activities ===
* Week 1-2: Familiar with 5g-lena ([https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_1_%5BMay_27_-_Jun._02%5D Week 1], [https://www.nsnam.org/wiki/GSOC2024RLUsability5G#Week_2_%5BJun._03_-_Jun._09%5D Week 2]
* Week 3: Design Scenario

== Phase 2: Design RL based Scheduler ==

== Phase 3: RL Integration ==

== Phase 4: Refine Code and Evaluation ==

GSOC2024RLUsability5GFinalReport

2024-10-07T11:50:00Z

Mye280c37: /* Phase 1: Design example */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

== Phase 2: Design RL based Scheduler ==

== Phase 3: RL Integration ==

== Phase 4: Refine Code and Evaluation ==

GSOC2024RLUsability5GFinalReport

2024-10-07T11:49:21Z

Mye280c37: /* Merge Requests and Commits */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

=== Merge Requests ===

All the following activities can be easily reviewed on the following MR:

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

== Phase 2: Design RL based Scheduler ==

== Phase 3: RL Integration ==

== Phase 4: Refine Code and Evaluation ==

GSOC2024RLUsability5GFinalReport

2024-10-07T11:48:29Z

Mye280c37: /* Project Details */

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1: Design example ==

== Phase 2: Design RL based Scheduler ==

== Phase 3: RL Integration ==

== Phase 4: Refine Code and Evaluation ==

GSOC2024RLUsability5GFinalReport

2024-10-07T11:44:09Z

Mye280c37:

= Project Overview =

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Project Proposal:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Wiki:''' [https://www.nsnam.org/wiki/GSOC2024RLUsability5G GSOC2024RLUsability5G]

== Merge Requests and Commits ==

I maintained a single branch for all work during GSoC: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commits/gsoc24-nr-usability?ref_type=heads gsoc24-nr-usability]

{| class="wikitable"
|+ Merge Requests
|-
! No. || Name || Status
|-
| [1] || [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 Draft: GSoC2024: RL-based Scheduler] || Draft
|}

=== Milestones ===

During the project, I pushed 170+ commits and squashed them to about 70 commits for merge. I managed all commits during the work in my personal repository, named "[https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads, 5g-lena-integrated-with-ns-3-gym/gsoc24-nr-usability]" by organizing them into milestones and issues

{| class="wikitable"
|+ Milestones
|-
! No. || Name || Period
|-
| [1] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/1#tab-issues Design Scenario] || Jun 10, 2024–Jun 17, 2024
|-
| [2] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues Drafting an AI scheduler] || Jul 11, 2024–Jul 17, 2024
|-
| [3] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues Develop an RL-based scheduler] || Jul 18, 2024–Jul 24, 2024
|-
| [4] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues Update the RL-based Scheduler (Code Refactoring)] || Jul 24, 2024–Jul 31, 2024
|-
| [5] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues Create test] || Jul 29, 2024–Aug 11, 2024
|-
| [6] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues Develop Gym Interface in "cttc-nr-rl-based-sched" Example] || Aug 12, 2024–Aug 25, 2024
|-
| [7] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/7#tab-issues Develop Gym Python Scripts] || Aug 21, 2024–Oct 2, 2024
|-
| [8] || [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/8#tab-issues Resolve comments in MR: cttc-lena/nr!166] || Sep 2, 2024–Oct 6, 2024
|}

= Project Details =

== Phase 1 ==

== Phase 2 ==

== Phase 3 ==

== Phase 4 ==

Summer Projects

2024-10-07T11:10:42Z

Mye280c37: /* Google Summer of Code 2024 */

{{TOC}}

The project coordinates a few summer coding programs in which student developers are paired with mentors to produce code over the summer.

= Google Summer of Code 2024 =

ns-3 is mentoring three projects for 2024:

* '''Joao Albuquerque''', [[GSOC2024Channels5G | 5G NR Module Benchmark and Analysis for Distinct Channel Models]], mentored by Biljana Bojovic, Amir Ashtari, and Gabriel Ferreira: '''''[[GSOC2024Channels5GFinalReport | Final report]]'''''
* '''Kavya Bhat''', [[GSOC2024DHCPv6 | DHCPv6]], mentored by Tommaso Pecorella, Alberto Gallegos Ramonet, and Manoj Kumar Rana: '''''[[GSOC2024DHCPv6FinalReport | Final report]]'''''
* '''Hyerin Kim''', [[GSOC2024RLUsability5G | Enhancement of RL Approach Accessibility in NR]], mentored by Katerina Koutlia, Amir Ashtari, Biljana Bojovic, and Gabriel Ferreira: '''''[[GSOC2024RLUsability5GFinalReport | Final report]]'''''

* [[GSOC2024Projects | Project Ideas Page]]
* [[GSOC2024ContributorGuide | ns-3's GSoC Contributor Guide]]

= Google Summer of Code 2023 =

The organizational admins were Tommaso Pecorella, Mohit Tahiliani, and Tom Henderson.

We had three successful student projects for 2023; you can read their final reports below:

* '''Giovanni Grieco''', [[GSOC20235GUsability | IUNS-3 5G NR: Improving the Usability of ns-3's 5G NR Module]], mentored by Tom Henderson, Katerina Koutlia, and Biljana Bojovic: '''''[[GSOC20235GUsabilityFinalReport | Final report]]'''''
* '''Raghuram Kannan''', [[GSOC2023NetAnim | Dynamic device registration for NetAnim simulation animations]], mentored by Tommaso Pecorella and Manoj Kumar Rana: '''''[[GSOC2023NetAnimFinalReport | Final report]]'''''
* '''Muyuan Shen''', [[GSOC2023ns3-ai | ns3-ai enhancements]], mentored by Collin Brady and Hao Yin: '''''[[GSOC2023ns3-aiFinalReport | Final report]]'''''

We reviewed seven proposals, which were evaluated by a committee that includes most of the mentors listed on the Project Ideas page. For future reference, below is our ideas page and contributor's guide.

* [[GSOC2023Projects | Project Ideas Page]]
* [[GSOC2023ContributorGuide | ns-3's GSoC Contributor Guide]]

= Google Summer of Code 2022 =

The organizational admins were Tommaso Pecorella, Mohit Tahiliani, and Tom Henderson.

Two contributors completed projects with ns-3 in the 2022 Google Summer of Code.

* '''Matteo Pagin''', [[GSOC2022Channel | A simplified channel and beamforming model for ns-3]], mentored by Sandra Lagen, Biljana Bojovic, and Michele Polese '''''[https://pagmatt.github.io/blog/2022/gsoc2022 Final Report]'''''
* '''Zhiheng Dong''', [[GSOC2022PerfectArp | Perfect ARP and NDP]], mentored by Tommaso Pecorella, Ameya Deshpande,and Manoj Kumar Rana '''''[[GSOC2022NeighborCacheFinalReport | Final Report]]'''''

One additional project was started but did not progress past the midterm evaluation:

* '''Akash Mondal''', [[GSOC2022TCPMaximumSegmentSize | TCP maximum segment size (MSS) improvements]], mentored by Mohit Tahiliani, Bhaskar Kataria, and Vivek Jain

We received seven proposals, which were evaluated by a committee that includes most of the mentors listed on the Project Ideas page. For future reference, below is our ideas page and contributor's guide.

* [[GSOC2022Projects | Project Ideas Page]]
* [[GSOC2022ContributorGuide | ns-3's GSoC Contributor Guide]]

= ns-3 Summer of Code 2022 =

ns-3 Summer of Code (NSoC) is a program that runs in parallel to Google Summer of Code. Unlike GSoC, it is not funded, but we aim to operate it similarly to GSoC.

* '''Chandrakant Jena,''' ''Ping and Ping6 Enhancements for ns-3:'' '''''[[NSOC2022Ping | project wiki page ]]''''' mentored by Tommaso Pecorella and Tom Henderson

The project by Chandrakant was completed successfully in December 2022, with six commits made to ns-3-dev starting with [https://gitlab.com/nsnam/ns-3-dev/-/commit/da107e04eeb0c12631eadda6a30ad1a33b7f0931 commit da107e04]. The project added a '''[https://www.nsnam.org/docs/doxygen/d6/dbe/classns3_1_1_ping.html new Ping application]''', helper class, example program, unit test, and [https://www.nsnam.org/docs/models/html/internet-apps.html#ping documentation].

= Google Summer of Code 2021 =

Three students successfully completed projects in [https://summerofcode.withgoogle.com/organizations/4672908493848576/ Google Summer of Code 2021].

* '''Parth Pratim Chatterjee,''' ''Direct Code Execution Modernization:'' '''''[[GSOC2021DCE | project wiki page]]''''' '''--''' '''''[https://ns-3-dce-linux-upgrade.github.io/ Final report]'''''
* '''Ameya Deshpande,''' ''IPv6 Nix-Vector Routing:'' '''''[[GSOC2021NixVector | project wiki page]]''''' '''--''' '''''[https://www.nsnam.org/wiki/GSOC2021NixVectorFinalReport Final report]'''''
* '''Akshit Patel,''' ''Add logging support to Simulation Execution Manager (SEM):'' '''''[[GSOC2021SEM | project wiki page]]''''' '''--''' '''''[https://akshitpatel01.github.io/GSoC-2021-Report/ Final report]'''''

For reference, below were the 2021 project ideas and the 2021 student guide:

* [[GSOC2021StudentGuide | Student Guide]]
* [[GSOC2021Projects | Project Ideas Page]]

= ns-3 Summer of Code 2021 =

ns-3 Summer of Code (NSoC) is a program that runs in parallel to Google Summer of Code. Unlike GSoC, it is not funded, but we aim to operate it similarly to GSoC. The commitments are similar; students and mentors are expected to define and work towards a mergeable project goal by the end of the summer. The program is offered to 'honorable mention' GSoC proposals (i.e., proposals that we would have selected had we received more student slots from Google) and for other reasons such as a contributor's ineligibility for GSoC.

* '''Nitya Chandra,''' ''Enable IPv6 support for ad-hoc routing protocols in ns-3:'' '''''[[NSOC2021Ipv6 | project wiki page ]]''''' '''Note: project did not complete'''

= Google Summer of Code 2020 =

Four students successfully completed [https://summerofcode.withgoogle.com/ Google Summer of Code 2020] projects:

* '''Shivamani Patil,''' ''App Store Improvements:'' '''''[https://shivamanipatil.github.io/gsoc-2020-report/ final report], [[GSOC2020AppStore | project wiki page]]'''''
* '''Ananthakrishan S,''' ''NetDevice up/down consistency and event chain:'' '''''[https://ananthu-dev.github.io/net-device-consistency-gsoc-2020/ final report], [[GSOC2020NetDevice | project wiki page]]'''''
* '''Bhaskar Kataria,''' ''SCE AQMs and TCP along with CNQ-CodelAF and LFQ'' '''''[https://bhaskar792.github.io/GSoC-2020-Report/ final report], [[GSOC2020AQM | project wiki page]]'''''
* '''Deepak K,''' ''TCP Prague model for ns-3'', '''''[https://deepakkavoor.github.io/gsoc-2020-prague/ final report], [[GSOC2020Prague | project wiki page]]'''''

For reference, below were the 2020 project ideas and the 2020 student guide:

* [[GSOC2020StudentGuide | Student Guide]]
* [[GSOC2020Projects | Project Ideas Page]]

= ns-3 Summer of Code 2020 =

These projects are unfunded but are mentored in a manner similar to GSoC, at a lesser pace than the 12-week GSoC program.

* Muhammad Iqbal Rochman, [[NSOC2020WifiPHY | Wi-Fi PHY Restructure]] '''Note:''' This project successfully completed.
* Harsha Sharma, [[NSOC2020L4SEvaluation | L4S evaluation framework]] '''Note:''' This project continued through fall 2020 but did not yet merge.
* Rahul Bothra, [[NSOC2020Routing | Routing for community wireless networks]] '''Note:''' Project discontinued in August 2020.

= Google Summer of Code 2019 =

ns-3 participated in Google Summer of Code 2019 with four student projects:

* Apoorva Bhargava, [[GSOC2019TCPTestingAndAlignment | Testing and Alignment of ns-3 TCP with Linux TCP]]
* Mishal Shah, [[GSOC2019AppStore | Improving the ns-3 AppStore and linking with bake]]
* Tommaso Zugno, [[GSOC2019ThreeGPPChannel | Integration of the 3GPP TR 38.901 channel model in the ns-3 spectrum module]]
* Liangcheng Yu, [[GSOC2019DCN | Framework of Studying Flow Completion Time Minimization for Data Center Networks in ns-3]]

Below are project ideas and the 2019 student guide:

* [[GSOC2019StudentGuide | ns-3 GSoC student guide]]
* [[GSOC2019Projects | Project Ideas Page]]

= European Space Agency Summer of Code in Space (SOCIS) 2019 =

ns-3 ultimately was not selected for funding for SOCIS 2019. Below is an archive of our student guide, for future reference.

* [[SOCIS2019 | ns-3 SOCIS student guide]]

= Google Summer of Code 2018 =

ns-3 participated in the 2018 edition of Google Summer of Code, with five students:

* WenYing Dai, [[GSOC2018AccECN_ECN++ | Implementation of AccECN and ECN++ in ns-3]]
* Muhammad Iqbal CR, [[GSOC2018Coexistence | Merging and Improvement of LTE and Wi-Fi Coexistence Module]]
* Sourabh Jain, [[GSoC2018_DCE_Upgrade | Direct Code Execution upgrade]]
* Davide Magrin, [[GSoC2018:_A_Simulation_Execution_Manager_for_ns-3 | A simulation execution manager for ns-3]]
* Jude Niroshan, [[GSoC2018:Trust-based_routing_protocols_framework | Trust-based routing protocols framework]]

* [[GSOC2018Projects | Project Ideas Page]]
* [[GSOC2018StudentGuide | Student Application Guide]]

= European Space Agency Summer of Code in Space (SOCIS) 2017 =

ns-3 has been accepted to the 2017 ESA Summer of Code in Space, with student Pasquale Imputato (mentored by Tommaso Pecorella). The project successfully completed in October 2017 (details in the below wiki project page).

* [[SOCIS2017 | project page]]
* [https://codereview.appspot.com/330220043/ Final code review]

The original project ideas page is posted below.

* [[SOCIS2017Projects#Project_Ideas | Project Ideas page]]

= Google Summer of Code 2017 =

ns-3 was fortunate to mentor five outstanding students for the 2017 edition of [https://developers.google.com/open-source/gsoc/ Google Summer of Code].

* [[GSOC2017AcceptedProjects | Accepted Projects]]
== Final reports ==
* [http://mailman.isi.edu/pipermail/ns-developers/2017-August/013916.html ns-3 App Store] by Abhijith Anilkumar
* [https://www.nsnam.org/wiki/GSOC2017Lte#Project_summary Enabling LTE CA handover to secondary cell] by Alexander Krotov
* [http://mailman.isi.edu/pipermail/ns-developers/2017-September/013929.html TCP Prague] by Shravya Ks
* [http://mailman.isi.edu/pipermail/ns-developers/2017-September/013918.html LTE and IPv6 support] by Manoj Kumar Rana
* [http://mailman.isi.edu/pipermail/ns-developers/2017-September/013921.html TBF and HHF] by Surya Seetharaman

== Phase 2 reports ==
* [http://mailman.isi.edu/pipermail/ns-developers/2017-July/014038.html BCube and FatTree topology helpers (component of TCP Prague project)]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-August/014054.html Implementation of TBF and HHF]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-July/014047.html Enabling LTE CA handover to secondary cell, Phase 2]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-July/014042.html ns-3 App Store]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-July/014049.html Mobile IPv6 implementation with LTE support (report)]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-August/014058.html Mobile IPv6 implementation with LTE support (review request)]
== Phase 1 reports ==
* [http://mailman.isi.edu/pipermail/ns-developers/2017-June/013980.html Data Center TCP (component of TCP Prague project)]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-June/013982.html Implementation of TBF and HHF traffic control]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-June/013983.html Enabling LTE CA handover to secondary cell, Phase 1]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-June/013985.html ns-3 App Store]
* [http://mailman.isi.edu/pipermail/ns-developers/2017-June/013987.html Mobile IPv6 implementation with LTE support]
== Background ==
Below is some information that was used during the application phase.

* [[GSOC2017Projects | Project Ideas Page]]
* [[GSOC2017StudentGuide | Student Application Guide]]

= European Space Agency Summer of Code in Space (SOCIS) 2016 =

ns-3 had one student (Michael Di Perna) successfully complete the 2016 [http://sophia.estec.esa.int/socis/ ESA Summer of Code in Space].

* [[SOCIS2016 | Project page]] for Optical Satellite Systems project
* [[SOCIS2016Projects#Project_Ideas | Project Ideas page]]

= Mentored summer projects 2016 =

ns-3 maintainers will mentor additional summer projects (that students will work on using their own sources of funding) on a best-effort basis. Students interested in this option should review the GSoC or SOCIS ideas page, or propose their own.

* See [[MentoredProjects2016]]

= Google Summer of Code 2016 =

ns-3 was not selected for the 2016 [https://developers.google.com/open-source/gsoc/ Google Summer of Code]. We mentored two summer projects outside of GSoC. Below were our materials prepared for our GSoC organizational application.
* [[GSOC2016Projects | Project ideas page]]
* [[GSOCStudentGuide | Student guide]]

= Google Summer of Code 2015 =

ns-3 was selected to participate in the 2015 [http://www.google-melange.com/gsoc/homepage/google/gsoc2015 Google Summer of Code]. More information can be found on our Project Ideas page and our Student Guide.

* [[GSOC2015AcceptedProjects | Accepted projects]]
* [[GSOC2015Projects | Project ideas page]]
* [[GSOC2015StudentGuide | Student guide]]

This year's students were announced on April 27, and all four successfully completed the program:

* Melchiorre Danilo Abrignani, [[GSOC2015LTECA | Carrier Aggregation support for the LTE module]]
* Matthieu Coudron, [[GSOC2015MpTcpImplementation | Implementing multipath TCP (MPTCP) in ns3]]
* Natale Patriciello, [[GSOC2015TCPTest | TCP layer refactoring with automated test on RFC compliance and validation]]
* Vishwesh Rege, [[GSOC2015LrWpanMac | 802.15.4 realistic MAC and Energy Model]]

= European Space Agency Summer of Code in Space (SOCIS) 2015 =

ns-3 has been accepted to the 2015 [http://sophia.estec.esa.int/socis2015/ ESA Summer of Code in Space]. The ns-3 project had one student in SOCIS in each of 2013, 2014 and 2015. However, the satellite channel models project for 2015 [[SOCIS2015 | Satellite channel models]] did not successfully complete.

* [[SOCIS2015Projects | Project ideas page]] (for reference)

= Mentored summer projects =

ns-3 maintainers will mentor additional summer projects (that students will work on using their own sources of funding) on a best-effort basis. Students interested in this option should review the GSoC or SOCIS ideas page, or propose their own.

We have one such mentored project in 2015:

* Saswat Mishra, [[NeighborDiscoveryProject | Neighbor Discovery enhancements]]

= Past summer projects =

* [[GSOC2014AcceptedProjects | GSoC 2014 Accepted Projects]]
* [[SOCIS2014TCP | SOCIS 2014 Accepted Project]]
* [[MentoredProjects2014 | 2014 Mentored Projects]]
* [[SOCIS2013BundleProtocolProject | SOCIS 2013 Accepted Project]]
* [[GSOC2013AcceptedProjects | GSoC 2013 Accepted Projects]]
* [[GSOC2012AcceptedProjects |GSoC 2012 Accepted Projects]]
* [[NSOC2011AcceptedProjects |NSoC 2011 Accepted Projects]]
* [[GSOC2010AcceptedProjects |GSoC 2010 Accepted Projects]]
* [[GSOC2009AcceptedProjects |GSoC 2009 Accepted Projects]]
* [https://developers.google.com/open-source/soc/2008/?csw=1#ns3 GSoC 2008 Accepted Projects]

GSOC2024RLUsability5GFinalReport

2024-10-07T11:10:31Z

Mye280c37: Created page with "= Project Overview = == Merge Requests and Project Details =="

= Project Overview =

== Merge Requests and Project Details ==

GSOC2024RLUsability5G

2024-10-07T07:46:15Z

Mye280c37: /* Project Overview */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''MR:''' [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 cttc-lena/nr!166]
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system, to accommodate the potential increase in the number of flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]
* Adjusted the action range for active flows to avoid zero, ensuring it is distinguishable from the default action value of zero for non-active flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4331d17333de1ce1b415895d6a1e8ce589ccb427 Adjust the action range to avoid zero]
* Resolved the documentation-related threads in the [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/cdd1cc8f00b3c0745b446b4b81ef0ecc8b620e90 Modify the description of gsoc-nr-rl-based-sched]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d2bd589f8116b6ab0e60e05f2dd31f84b7160151 Add a description for nr-test-scheduler-ai]

=== Week 19 [Sep. 30 - Oct. 06]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] and squash commits

* Relocated and renamed ns3-gym environment files to improve user accessibility when constructing their own examples using the ns3-gym environment
::৹ Renamed files to 'nr-mac-scheduler-ai-ns3-gym-env.*' and class name to 'NrMacSchedulerAiNs3GymEnv'
::৹ Moved files to 'model/' directory
* Add Doxygen to the ns3-gym environment per each functions to enhance users' understanding and enable more effective utilization of the class
* Squashed commits for merge

GSOC2024RLUsability5G

2024-10-07T07:39:59Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system, to accommodate the potential increase in the number of flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]
* Adjusted the action range for active flows to avoid zero, ensuring it is distinguishable from the default action value of zero for non-active flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4331d17333de1ce1b415895d6a1e8ce589ccb427 Adjust the action range to avoid zero]
* Resolved the documentation-related threads in the [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/cdd1cc8f00b3c0745b446b4b81ef0ecc8b620e90 Modify the description of gsoc-nr-rl-based-sched]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d2bd589f8116b6ab0e60e05f2dd31f84b7160151 Add a description for nr-test-scheduler-ai]

=== Week 19 [Sep. 30 - Oct. 06]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] and squash commits

* Relocated and renamed ns3-gym environment files to improve user accessibility when constructing their own examples using the ns3-gym environment
::৹ Renamed files to 'nr-mac-scheduler-ai-ns3-gym-env.*' and class name to 'NrMacSchedulerAiNs3GymEnv'
::৹ Moved files to 'model/' directory
* Add Doxygen to the ns3-gym environment per each functions to enhance users' understanding and enable more effective utilization of the class
* Squashed commits for merge

GSOC2024RLUsability5G

2024-10-07T07:35:53Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system, to accommodate the potential increase in the number of flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]
* Adjusted the action range for active flows to avoid zero, ensuring it is distinguishable from the default action value of zero for non-active flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4331d17333de1ce1b415895d6a1e8ce589ccb427 Adjust the action range to avoid zero]
* Resolved the documentation-related threads in the [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/cdd1cc8f00b3c0745b446b4b81ef0ecc8b620e90 Modify the description of gsoc-nr-rl-based-sched]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d2bd589f8116b6ab0e60e05f2dd31f84b7160151 Add a description for nr-test-scheduler-ai]

=== Week 19 [Sep. 30 - Oct. 06]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] and squash commits

* Relocate and rename ns3-gym environment files
::৹ Rename files to 'nr-mac-scheduler-ai-ns3-gym-env.*' and class name to 'NrMacSchedulerAiNs3GymEnv' to improve user accessibility when constructing their own examples using the ns3-gym environment

GSOC2024RLUsability5G

2024-10-02T07:56:12Z

Mye280c37: /* Week 18 [Sep. 23 - Sep. 29] */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system, to accommodate the potential increase in the number of flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]
* Adjusted the action range for active flows to avoid zero, ensuring it is distinguishable from the default action value of zero for non-active flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4331d17333de1ce1b415895d6a1e8ce589ccb427 Adjust the action range to avoid zero]
* Resolved the documentation-related threads in the [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/cdd1cc8f00b3c0745b446b4b81ef0ecc8b620e90 Modify the description of gsoc-nr-rl-based-sched]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d2bd589f8116b6ab0e60e05f2dd31f84b7160151 Add a description for nr-test-scheduler-ai]

GSOC2024RLUsability5G

2024-10-02T07:44:17Z

Mye280c37: /* Week 18 [Sep. 23 - Sep. 29] */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system, to accommodate the potential increase in the number of flows
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]

GSOC2024RLUsability5G

2024-10-02T07:41:08Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluated the AI scheduler against other schedulers: QoS, RR, PF
::৹ Ran tests with various simulation parameters: TDMA/OFDMA, RR LC/QoS LC Assignment
::৹ Tested with different reward functions
* Fixed and reordered state based on (RNTI, LC ID) order to prevent state shape changes when the number of active flows changes during the simulation
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/a3d3f842a01675337cc981676fb2d6969de34f2a Fix state shape bug caused by active flows]
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/8ea00773e73f71a89d6072f0e0de290c067bd988 Support fixed format state representation]
* Adjusted the action range to continuous values
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/74262f1042bbcdcce89ec0ae03a251e3a8e9cdcc Adjust to a continuous action range]

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]

GSOC2024RLUsability5G

2024-10-02T07:31:02Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluate the AI scheduler against other schedulers: QoS, RR, PF
::৹ Run tests with various simulation parameters: TDMA/OFDMA, RRLCScheduler/QoSLCScheduler
::৹ Run tests with different reward functions

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* Evaluated the schedulers (AI, QoS, PF, and RR) in a scenario where a single gNB with a single cell is deployed with 2 UEs, each having a single flow:
::৹ One UE has a non-GBR flow, and the other UE has a Delay-Critical (DC)-GBR flow
::৹ OFDMA was configured with QoS LC assignment
::৹ You can view the simulation results here: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]
* Enabled CUDA support when users set the 'enableCuda' parameter to 1, provided that CUDA is available on the system.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/d8cb48b5c71748e52d7b32d3f0fa7f17ef5fc32a Enable torch processes on cuda]

GSOC2024RLUsability5G

2024-10-01T14:46:04Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models (1)

* Evaluate the AI scheduler against other schedulers: QoS, RR, PF
::৹ Run tests with various simulation parameters: TDMA/OFDMA, RRLCScheduler/QoSLCScheduler
::৹ Run tests with different reward functions

=== Week 18 [Sep. 23 - Sep. 29]===
Conduct Simulation Campaigns and Refine Models (2)

* [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/wikis/Simulation-Campaigns Simulation Campaigns]

GSOC2024RLUsability5G

2024-09-25T09:00:35Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

=== Week 17 [Sep. 16 - Sep. 22]===
Conduct Simulation Campaigns and Refine Models

* Evaluate the AI scheduler against other schedulers: QoS, RR, PF
::৹ Run tests with various simulation parameters: TDMA/OFDMA, RRLCScheduler/QoSLCScheduler
::৹ Run tests with different reward functions

GSOC2024RLUsability5G

2024-09-16T18:52:10Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

=== Week 16 [Sep. 09 - Sep. 15]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Revised the UE configuration in the example to align with the multi-flow qos example
* Resolved comments on the python gym scripts
::৹ Changed the structure of the Memory class in test-ppo.py to use a dequeue
::৹ Added descriptions for each function and class
::৹ Added a simple debug print function
::৹ Fixed minor issues related to variables
* Resolved other comments:
::৹ Removed empty destructor
::৹ Removed mask-related code
::৹ Added descriptions to the MyGymEnv class
::৹ Added an installation guide for the rl-based example

GSOC2024RLUsability5G

2024-09-16T18:39:57Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
::৹ Fixed minor doxygen details in the scheduler class
::৹ Enhanced documentation details in the .rst file
* Resolved comments on the scheduler's unit test
::৹ Added documentation
::৹ Removed notching-related code

GSOC2024RLUsability5G

2024-09-16T18:10:15Z

Mye280c37: /* Week 15 [Sep. 02 - Sep. 08] */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Updated Doxygen documentation
* Resolved comments on the scheduler's unit test

GSOC2024RLUsability5G

2024-09-16T18:09:40Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for Proximal Policy Optimization (PPO) model

* Develop a python gym script for PPO model that aligns with the observations and actions of the AI scheduler
::৹ Developed a PPO class containing the Actor-Critic component
::৹ Using the Actor-Critic class, the PPO model selects actions and evaluates the selected actions
::৹ Developed a Memory class that stores states, actions, and log probabilities for future updates during each action selection iteration
::৹ The stored memory is periodically updated
::৹ Defined arguments using 'argparse' to allow users to adjust parameters for a simple test
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Update Doxygen documentation
::৹ Resolve comments on the scheduler's unit test

GSOC2024RLUsability5G

2024-09-10T10:24:50Z

Mye280c37: /* Weekly Report */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Aug. 26 - Sep. 01]===
Develop a python gym script for PPO model

* Develop a python gym script for PPO model
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]

=== Week 15 [Sep. 02 - Sep. 08]===
Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Update Doxygen documentation
::৹ Resolve comments on the scheduler's unit test

GSOC2024RLUsability5G

2024-09-10T08:09:06Z

Mye280c37: /* Week 114 [Sep. 02 - Sep. 08] */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 14 [Sep. 02 - Sep. 08]===
Develop a python gym script for PPO model and Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Develop a python gym script for PPO model
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]
* Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Update Doxygen documentation
::৹ Resolve comments on the scheduler's unit test

GSOC2024RLUsability5G

2024-09-10T08:08:56Z

Mye280c37: /* Week 114 [Sep. 02 - Sep. 08] */

{{TOC}}

Back to [[Summer_Projects#Google_Summer_of_Code_2024 | GSoC 2024 projects]]

== Project Overview ==

* '''Project Name:''' Enhancement of RL Approach Accessibility in NR
* '''Student:''' Hyerin Kim
* '''Mentors:''' Katerina Koutlia, Amir Ashtari, Bijana Bojovic, Gabriel Ferreira
* '''Google page:''' https://summerofcode.withgoogle.com/programs/2024/projects/vPuZgTe1
* '''Project Goals:''' In this project, I will design a new RL based MAC scheduler of NR and implement it in 5g-lena integrating with ns3-gym. Additionally, I will enhance the usability of 5G-lena in terms of RL approach by providing an example using the designed RL based scheduler.
* '''Repository:''' https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/tree/gsoc24-nr-usability?ref_type=heads (fork nr GitLab)
* '''About Me:''' I am currently pursuing a Master's degree in Computer Science and Engineering at Seoul National University, South Korea. My research at the Mobile Computing & Communications Laboratory focuses on resource allocation methods in NR V2X Sidelink. As an undergraduate, I conducted research on improving spatial reuse in dense Wi-Fi environments and implemented a Reinforcement Learning (RL)-based modified OBSS/PD algorithm using ns3 gym. I believe that participating in GSoC 2024 presents an excellent opportunity for me to contribute to enhancing the usability of 5G and RL experiences on ns-3, while also deepening my understanding of 5G technology, mechanisms, and system architecture.

== Milestones ==

The planned milestones are outlined below

=== Phase1. Design example (3 weeks) ===

* Familiar with 5g-lena ''(2 weeks)''
* Design Scenario (e.g., UEs deployment, UEs speed, cell configuration, …) ''(1 week)''
::- Define Assumption (e.g., delay, TDMA/OFDMA, …)

=== Phase2. Design RL based Scheduler (6 +1 weeks) ===

* Design scheduler ''(2 weeks)''
::- input/output
::- goal of optimization
* Design RL process ''(1 week)''
::- Define suitable RL techniques considering optimization objective of the scheduler and computational complexity
* Implementation of RL based scheduler in 5g lena ''(3 +1 weeks)''
::- Create the test
::- Create documentation
::- Create MR to 5g lena '''''(Milestone 1)'''''

=== Phase3. RL Integration (4 weeks) ===

* Design RL framework ''(1 week)''
::- Define RL technique
* Develop gym scripts ''(3 weeks)''
::- Develop gym python scripts
::- Develop ns3 gym interface in RL 5g lena example
::- Validate RL process of the example
::- Create MR to 5g lena '''''(Milestone 2)'''''

=== Phase4. Evaluation (3 +1 weeks)===

* Evaluate the result of example compared with other schedulers
::- Write simulation campaign scripts
::- Execute scripts
::- Plotting python scripts
* Address review comment of the MR 1 and 2
* Update MR 1 and 2 with necessary modification
* Create brief description of the work and the results for 5g lena blog '''''(Milestone 3)'''''

== Weekly Report ==

=== Week 1 [May 27 - Jun. 02] ===
Familiarity with 5g-lena (1)

* Studied cttc-nr-demo example with cttc-nr-demo tutorial
::৹ quasi-ideal assumption
::৹ RAN Lifecycle (i.e. downlink packet flow from gNB to UE)
::৹ How to configure the scenario, EPC, physical layer, and traffic(application)
::৹ ''(DOUBT)'' What do I consider when I configure bands, carrier component, BWP?

=== Week 2 [Jun. 03 - Jun. 09] ===
Familiar with 5g-lena (2)

* Organized the concepts of Band, CC, and BWP to address the issue from the previous week
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-01/ Band, Carrier Component (CC), Bandwidth Part (BWP)]
* Studied schedulers in 5g-lena
::৹ Studied MAC Layer section related to schedulers in NR module documentation
::৹ Studied cttc-nr-simple-qos-sched example

=== Week 3 [Jun. 10 - Jun. 16]===
Design Scenario

* Analyzed the QoS Scheduler in 5g-lena
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-02/ QoS Scheduler Simulation Results Analysis]
::৹ Reference: Koutlia, Katerina, Sandra Lagén, and Biljana Bojovic. "Enabling QoS Provisioning Support for Delay-Critical Traffic and Multi-Flow Handling in ns-3 5G-LENA." Proceedings of the 2023 Workshop on ns-3. 2023.

* Created an initial example code
::৹ Created the example based on the example named 'cttc-nr-simple-qos-sched'
::৹ Altered the default UE count to 3, assigning each UE a single QoS flow represented by 5QI indices 1, 80, and 87, which denote GBR, non-GBR, and Delay Critical GBR resource types, respectively
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/80523b0b9baf3199554e1d0b72a4682ef560b62e Add new example cttc-nr-initial-my-sched]

=== Week 4 [Jun. 17 - Jun. 23]===
Design Scheduler (1)

* Modified the initial example
::৹ Added a new parameter `numTrafficProfile`
::::‣ Specified the flow profile based on the number of traffic types. Possible values are 2 and 3
::::‣ If the `numTrafficProfile` is 2, two types of traffic (5QI 80, 87), same as ‘cttc-nr-multi-flow-qos-sched’, is installed in 3 UEs. Two UE have 5QI 80 traffic and the other has 5QI 87 traffic
::::‣ If the `numTrafficProfile` is 3, three types of traffic (5QI 1, 80, 87), as defined initially, is installed in 3 UEs repectively.
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7eafd7d5cd173a1d3e25603c53065e50966945cb Add a parameter for the number of traffic types]

* Evaluated the QoS scheduler in various scenarios
::৹ Evaluated the results when non-GBR and DC GBR are installed in different UEs to verify the simulation results analysis from the previous week
::৹ As a result of this evaluation, I observed that all non-GBR traffic experienced more severe delays compared to DC GBR traffic
::৹ From my analysis of the simulation results in the QoS scheduler paper, non-GBR traffic benefits when the UE with non-GBR traffic also has DC GBR traffic in a multi-flow configuration.
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-03/ QoS Scheduler Evaluation]

* Designed a new scheduler
::৹ Made the assumption that all UEs have a single flow for simplicity (extend it further to multi-flow after achieving the first goal)
::৹ Decided the goals of the scheduler
::::‣ Option 1: Minimize the total delay of the gNB
::::‣ Option 2: The scheduler schedules UEs to meet their PDB requirements (primary goal). Among DC-GBR and other types of traffic, DC-GBR traffic has higher priority in scheduling (secondary goal)

=== Week 5 [Jun. 24 - Jun. 30]===
Design Scheduler (2)

* The scheduler is designed to be user-friendly and easy to reuse
::৹ To facilitate ease of reuse, the scheduler's design should be simple and easily understandable
::৹ For simplicity, the goal of the scheduler is to minimize the total delay of the gNB while considering the priority of each attached UE
::::‣ As analyzed before, one of the issues in the QoS scheduler is the unfairness of delay between DC-GBR and other types of traffic
::::‣ To address this issue, the designed scheduler focuses on minimizing the total delay
::৹ Page: [https://mye280c37.github.io/posts/gsoc2024-04/ Scheduler Design]
::৹ '''TBD:''' the reward function

=== Week 6 [Jul. 01 - Jul. 07]===
Design RL Process

* Decided how to implement the designed RL-based scheduler in 5g-lena and ns3-gym
::৹ Visualized the sequence UML diagram of the scheduling process among `NrMacSchedulerTdma`, `NrMacSchedulerTdmaQos`, and `NrMacSchedulerUeInfoQos`
::৹ Based on the communication logic between 5g-lena and ns3-gym, designed the sequence UML diagram and the class UML diagram for the RL process
::::‣ The existing TDMA and OFDMA scheduler classes assign resources to UEs, calling their methods per UE
::::‣ However, the RL-based scheduler should transfer data, including data about all UEs
::::‣ Thus, `NrMacSchedulerTdmaAI` and `NrMacSchedulerOfdmaAI` will inherit from `NrMacSchedulerTdma` and `NrMacSchedulerOfdma`, respectively, and override `AssignDLRBG()` and `AssignULRBG()`. They will then perform the RL process by calling the `Notify()` method of the `OpenGymEnv` in the ns3-gym module, transferring the data about all UEs
::৹ Diagram: AI Scheduler Diagram ([https://docs.google.com/document/d/1n6u4pbRRItiNC5UgIYZiVjhuwZZl4bwsOz3BELxme4Q/edit?usp=sharing Google Docs], [https://drive.google.com/file/d/1NZMoSk6l0MX3IGXuHlEmaVupeOneHciU/view?usp=sharing draw.io])

=== Week 7 [Jul. 08 - Jul. 14]===
Implementation of RL-based scheduler in 5g lena (1)

* Created a [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR] to share the development progress
* Developed the skeleton of the RL-based scheduler on the 5g-lena side
::৹ Created the class skeleton
::::‣ ''(THOUGHT)'' The functions used for scheduling resources to UEs and the functions used for communicating with ns3-gym are the same in both TDMA and OFDMA schedulers. How about separating these functions into a class such as `NrMacSchedulerAIAlgorithm`?
::৹ Developed the logic for calling the ns3-gym environment in the `AssignDLRBG` function of the `NrMacSchedulerOfdmaAI` class
::৹ Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/2#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/2 Create class skeleton]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/3 Develop AssignDLRBG function]
* Modified the UML diagram to reflect the characteristics of a constant function in a class and that of the variable `ueVector`
::৹ In the AssignDLRBG function, the vector used for getting observations is assigned to a local variable named `ueVector`. Due to the characteristics of a constant function, `ueVector` can only be passed to other constant functions through input. Therefore, by defining a function named `CallNotifyFn()` that takes `ueVector` as an input, the function first transfers the information, including the observation, to `OpenGymEnv` in ns3-gym. It then calls the `Notify` function of `OpenGymEnv` to transfer this information to the Python gym through `OpenGymInterface`.
::৹ ''(DISCUSSION)'' When getting observations of all UEs, the function `GetUeObservation()` in `NrMacSchedulerUeInfoAI` is called for each UE. This can be implemented in either `NrMacSchedulerOfdmaAI`/`NrMacSchedulerTdmaAI` or `OpenGymEnv`. In the former case, the clarity of the classes is maintained, but the sequence diagram becomes more complex. In the latter case, the sequence diagram can remain simple. Which approach is better?

=== Week 8 [Jul. 15 - Jul. 21]===
Implementation of RL-based scheduler in 5g lena (2)

* Developed the RL-based scheduler on the 5g-lena side
::৹ Developed a class for UE information, including the data needed for RL-based scheduling
::৹ Developed classes for RL-based resource allocation in OFDMA and TDMA
::৹ Modified functions for assigning RBGs in OFDMA and TDMA scheduler classes to implement the RL-based scheduler
* Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/3#tab-issues milestone]. The issues pertaining to this milestone are as follows:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/4 Develop NrMacSchedulerUeInfoAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/5 Develop NrMacSchedulerOfdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/6 Develop NrMacSchedulerTdmaAI class]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/7 Modify the functions "AssignDLRBG" and "AssignULRBG" in the classes "NrMacSchedulerOfdma" and "NrMacSchedulerTdma"]
* Created draft documentation corresponding to the developed code
::৹ Commits: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/4cec70cd550ad010d2cbe931581366fb3f8a6a62 Add draft descriptions for RL-based scheduler], [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/5408fd5ae0d4621db9c16bf82bf6ecd50e4a2b2f Revise descriptions for RL-based scheduler]
* ''(DISCUSSION)'': Our developed classes call an RL model for each available symbol. How about adding functions to call the RL model for each slot, including information about available symbols in a slot as an observation?

=== Week 9 [Jul. 22 - Jul. 28]===
Implementation of RL-based scheduler in 5g lena (3) : Code Refactoring based on the comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
(Managed a week's work as a [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/5#tab-issues milestone].)

* Remove duplicate functions
::৹ Remove functions that are duplicated with the classes related to QoS
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/8 Remove duplicate functions]
* Improve the clarity and detail of structures
::৹ Change the structure of observation
::::‣ Create the LcObservation structure which represents the observation of a active flow
::::‣ The observation of a UE becomes a vector of LcObservation
::::‣ The observation of a scheduler becomes a vector of LcObservation
::৹ Add the details of the comparison weights function to call the functions of the QoS when AI is not activated
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/9 Improve the clarity and detail of structures]
* Improve the Doxygen documentation
::৹ Expand the doxygen of the classes related with the RL-based scheduler
::৹ Expand the descriptions about the RL-based scheduler in nr-module.rst
::৹ Fix the documentation in the 'cttc-nr-rl-based-sched' example
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/10 Improve the Doxygen documentation]
* Fix CI/CD errors
::৹ Fix check-style-clang-format errors
::৹ Fix clang-tidy errors
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/11 Fix CI/CD errors]

=== Week 10 [Jul. 29 - Aug. 04]===
Implementation of RL-based scheduler in 5g lena (4): Familiarity with the test code

* Discussed the test code for the RL-based scheduler
::৹ Decided to create unit test code for the function `CallNotifyDl`
::::‣ The test will check the first call of the function `CallNotifyDl`
::৹ If the test succeeds, it satisfies the following conditions:
::::‣ The observations transferred by the scheduler correspond to the QoS profiles of all UEs and the initial configuration
::::‣ The transferred reward should be zero, indicating no reward on the first call
::::‣ The function for updating weights from the scheduler is called successfully

* Studied the reference test code
::৹ By referring to “nr-test-notching.cc”, I can create a simplified SapProvider for testing, set up LC and parameters, and then call the functions that need to be tested (e.g., `CreateMac`, `CreateScheduler`, and the functions in the scheduler)
::৹ Alternatively, by referring to “nr-test-sched.cc”, I can use a system test, trace a specific observation time, and compare it with the UE’s observation

=== Week 11 [Aug. 05 - Aug. 11]===
Implementation of RL-based scheduler in 5g lena (5): Create the test and refactor the code

* Refactored the code while creating the test code
::৹ Removed redundant code
::৹ Clarified the logic for setting the function that calls the gym class
::::‣ The conversion error occurred while using `std::bind` because the `this` pointer was continually being recognized as an input while passing the weight update function of the `NrMacSchedulerUeInfoAi` class as a function argument
::::‣ This happened because the function was not declared as `const`
::৹ Clarified the definition of structures sent to the AI model
::৹ Fixed minor details in the definitions of functions and parameters
::৹ Progressed according to the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/4#tab-issues milestone], and the issue related to refactoring is as follows:
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/13 Refactor the code]
::::‣ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/12 Conversion error when passing member functions to CallNotifyDl and CallNotifyUl using std::bind]

* Created the unit test code according to the design discussed last week
::৹ Issue: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/14 Create the test code]

=== Week 12 [Aug. 12 - Aug. 18]===
Develop the ns3-gym interface in the RL 5G-LENA example

* To implement RL models using the AI scheduler, we need to define a custom gym environment by inheriting from the `OpenGymEnv` class in the ns3-gym module and then apply this in the example
* The following steps were taken to achieve this:
::৹ Defined a custom gym environment inheriting from the `OpenGymEnv` class in the ns3-gym module
::::‣ Defined OpenGymSpace and OpenGymDataContainer to align with the observation structure and weight of the scheduler
::::‣ Defined the Notify function, which will be set as a callback for the scheduler
::::‣ Defined a variable to implement the weights update function of the scheduler, which is passed as an input to the Notify function
::::‣ Overrode the functions to ensure the proper execution of the preceding tasks
::৹ Modified the "cttc-nr-rl-based-sched" example to apply the custom gym environment
::৹ Modified CMakeLists.txt to build the files needed for implementing RL models and the example
* The details can be found in the [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/milestones/6#tab-issues milestone], including the following issues:
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/16 Develop Custom Gym Env]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/17 Apply the Gym Env to "cttc-nr-rl-based-sched"]
::৹ [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/issues/15 Modify CMakeList.txt]
* ''(CHALLENGE)'' Modified CMake so that the necessary files are built only if the user has the ns-3 gym module when running the python gym script example. While there are no errors in clang-tidy, build, or execution, an error occurs in the CMake format pipeline

=== Week 13 [Aug. 19 - Aug. 25]===
Develop a python gym script for a simple test
* Developed a simple example using only the ns-3 gym environment without implementing a specific RL Model
::৹ Developed a simple python example that includes the arguments required to implement "Ns3Env," as well as the parameters corresponding to the "cttc-nr-rl-based-sched" example
::৹ Executed a basic loop that samples an action for each received data point from the custom C++-based Gym environment, which consists of observation, reward, game-over status, and extra information
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/30c1361d87868d1c1d969e29523eb2ac6b554036 Add a simple test implementation]
* Added the parameters from python gym environment in the "cttc-nr-rl-based-sched" example
::৹ To execute the "cttc-nr-rl-based-sched" example through the python gym script, the example must include the parameters "openGymPort" and "simSeed," which are passed by the python-based "Ns3Env"
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/332346480afcdf689e4c687a6f0bd8fabbf261e5, Apply the parameters used in the gym env]

=== Week 114 [Sep. 02 - Sep. 08]===
Develop a python gym script for PPO model and Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]

* Develop a python gym script for PPO model
::৹ Commit: [https://gitlab.com/mye280c37/5g-lena-integrated-with-ns-3-gym/-/commit/7e2fc471a47c516676c0326f6e4d2774719787a6 Add draft script for PPO model]
* Address comments in [https://gitlab.com/cttc-lena/nr/-/merge_requests/166 MR]
::৹ Update Doxygen documentation
::৹ Resolve comments on the scheduler's unit test