Difference between revisions of "GSOC2021DCE"

From Nsnam
Jump to: navigation, search
(Pre-Coding Phase - Report of work done during : 28th May to 2nd June(7 days))
(Artifacts for Workshop on ns-3 publication)
 
(30 intermediate revisions by 2 users not shown)
Line 8: Line 8:
 
* '''Student:''' [mailto:parth27official@gmail.com Parth Pratim Chatterjee]
 
* '''Student:''' [mailto:parth27official@gmail.com Parth Pratim Chatterjee]
 
* '''Mentors:''' Tom Henderson, Apoorva Bhargava, Vivek Jain
 
* '''Mentors:''' Tom Henderson, Apoorva Bhargava, Vivek Jain
* '''Project Goals:''' DCE currently makes use of net-next-nuse to extend the Linux kernel internals like the networking stack to host applications but over the years the project hasn't been updated with the latest releases of the Linux kernel. As Linux progressed with newer releases, a major part of the source code changed, making previous glue code incompatible with the newer implementations of the network stack as some of the init calls and function usage changed significantly making migration to newer releases non-trivial. This project aims at enabling support for latest Linux kernel features and toolchains in the DCE environment with support for the socket networking stack, sysctl interfaces, system call access, etc. without any changes to the user APIs currently being used by host applications. The project aims at incorporating the LKL(Linux Kernel Library) into the DCE environment for host applications to effortlessly make use of Linux kernel stacks with minimum to no change in existing simulation scripts.
+
* '''Project Goals:''' DCE currently makes use of net-next-nuse to extend the Linux kernel internals like the networking stack to host applications but over the years the project hasn't been updated with the latest releases of the Linux kernel. As Linux progressed with newer releases, a major part of the source code changed, making previous glue code incompatible with the newer implementations of the network stack as some of the init calls and function usage changed significantly making migration to newer releases non-trivial. This project aims at enabling support for latest Linux kernel features and toolchains in the DCE environment with support for the socket networking stack, sysctl interfaces, system call access, etc. without any changes to the user APIs currently being used by host applications. The project aims at incorporating an upgraded net-next-nuse port for Linux +v5.10 or the LKL(Linux Kernel Library) into the DCE environment for host applications to effortlessly make use of Linux kernel stacks with minimum to no change in existing simulation scripts.
 
* '''Repository:'''  
 
* '''Repository:'''  
 +
** net-next-nuse-5.10.27 Repo : https://github.com/ParthPratim/net-next-nuse-5.10.47
 +
** DCE Repo: https://github.com/ParthPratim/ns-3-dce
 
** LKL Repo : https://github.com/ParthPratim/linux
 
** LKL Repo : https://github.com/ParthPratim/linux
** DCE Repo: https://github.com/ParthPratim/ns-3-dce
 
 
* '''About Me:''' I'm a freshman Computer Science undergraduate student at Kalinga Institute of Industrial Technology, Bhubaneshwar, India. I have a keen interest in Linux internals and computer networking. I was a grand prize winner at Google Code-In, 2018 for ns-3 organization, which helped me initially get introduced to DCE. I have an aptitude for Competitive Programming and heavily make use of C/C++, STL and other OOP concepts in solving algorithmic puzzles. I have an experience with C/C++ and Python of more than 3 years, working on projects for numerous Hackathons.
 
* '''About Me:''' I'm a freshman Computer Science undergraduate student at Kalinga Institute of Industrial Technology, Bhubaneshwar, India. I have a keen interest in Linux internals and computer networking. I was a grand prize winner at Google Code-In, 2018 for ns-3 organization, which helped me initially get introduced to DCE. I have an aptitude for Competitive Programming and heavily make use of C/C++, STL and other OOP concepts in solving algorithmic puzzles. I have an experience with C/C++ and Python of more than 3 years, working on projects for numerous Hackathons.
  
 
= Milestones and Deliverables =
 
= Milestones and Deliverables =
* '''Phase 1'''
+
The overall project goal is to update DCE such that the latest Linux systems are supported and the latest Linux kernel code could be used.  
** Add links to kernel exported functions in KernelHandle, used in DCE.
+
* [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit?usp=sharing Detailed Project Plan] (will be continuously updated throughout the GSoC program duration)
** Fill up DceHandle struct with function references to DCE exported functions.
+
* '''Phase 1'''
** Implement all the host functions, link it to custom written mutex, semaphore and pthread functions, create the lkl_host_ops structure, initialize kernel with lkl_start_kernel.  
+
** Ubuntu 20.04 support
** Implement the sim_init method.
+
*** Goal is that most capabilities presently available for Ubuntu 16 DCE will be available for Ubuntu 20.04 (native)
 
+
*** Also contact glibc maintainers about a non-patched solution
* '''Phase 2'''  
+
*** More about this [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=id.8ttuw5i22nza here]
** Add support for socket networking and netdevices interface functions which will be exported to DCE.
+
** Upgrade net-next-nuse Linux kernel support to recent kernel
 
+
*** Focus is on the Google BBRv2 kernel (5.10 base): https://github.com/google/bbr
* '''Phase 3'''
+
*** Borrow from net-next-nuse-4.4.0 and LKL as appropriate to try to get a new version of net-next-nuse
** Add support for the new struct naming conventions.  
+
*** Review existing tests and define/write new tests
** Add support for liblkl by default in bake, ns-3-dce wscript, custom gcc parameters like -fpermissive(without which the header files won’t compile).
+
*** More about this [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=id.ixtd8h2ia3hf here]
** Align the sysctl usage in LinuxSocketFdFactory according to LKL.
+
** Investigate SMP architecture for LKL by querying the LKL developers list
** Modify existing examples to load liblkl.so. Fix errors and bugs that might appear in this stage.
+
*** More about this (also includes Technical specifications and reasons for the problems) [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=kix.h6e9x86v84qy here]
 
+
* '''Phase 2'''
= Pre-Coding Phase =
+
** Re-Implement Python Bindings build in DCE
* I thought of testing the scheduler fix for LKL which I thought of previously.
+
** Adding --apiscan feature to DCE
* Idea : The accept network system call depends on the following working principle. For AF_INET sockets it executes the inet_accept() function call which further sets up a few locks and other stuff and ultimately controls goes to inet_wait_for_connect(). This function is responsible for putting the current thread to TASK_INTERRUPTIBLE state and issues a schedule_timeout() which sets up a mod_timer to call back the called thread after particular amount of jiffies have passed, meanwhile it also schedules other tasks in the run queue so that while the thread which issued accept call is waiting for the call to return back with a file descriptor of the newly opened socket, we can run other tasks. This is pretty straight forward on any operating system, assume having two application, one server and one client, so while the server waits on accept, the linux scheduler schedules the client to issue a connect() call and then switches back to the server where the accept() returns back with a file descriptor on which send and recv operations could be done(in the same way as above). But, this isn't that straight forward with DCE.
+
** Debug dce script failures from Phase 1
 
+
** Implement and test fopencookie based FILE stream redirection
* First and foremost, DCE doesn't work on actual threads, rather it uses fibres(LWP) and creates contexts and switches back and forth between them using TaskManager and Fibre Switch handlers. A few important functions are TaskManager::(TaskStart/TaskWakeup/TaskWait/Schedule/TaskSwitch...). Now, since we are loading LKL as a shared library the kernel scheduler inside the LKL space doesn't have access to the native threads we work with in DCE and thus when any system call calls the scheduler, LKL keeps scheduling the tasks(ksoftirqd,tasklets,workqueues,kernel_threads,IRQ handlers) created inside the LKL environment and never reaches the DCE threads keeping the DCE execution at a standstill on blocking operations such as accept/connect/send/recv. Apart from normal syscalls, socket functions like send register a namespace skb destructor which gets pushed to a work_queue for later execution when sk_free is called. workqueue ultimately depends on schedule to schedule the rescuer kernel threads created to dequeue all the tasks pushed to the workqueue.
+
* '''Phase 3'''
 
+
** Introduce Github Actions CI to ns-3-dce
* Solution 1  : I though if the problem is, not reaching the DCE threads/fibres, then let's overwrite the LKL scheduler to first schedule the DCE threads first and then schedule the LKL tasks because LKL being a uniprocessor system(there goes another problem....i'll talk about that later in the report too) won't make much a difference(ummm...at least that's what seemed to me). So, I overwrote the schedule_timeout and schedule function to put the current DCE thread to sleep for given timeout and start the DCE scheduler with TaskManager::Schedule (private function, so had to create some public functions to run them), and then ran the scheduler inside LKL.
+
** Collaborate with mentors and project maintainers on resolving existing issues
 
+
** Produce Docker image and documentation to ease installation process
* Report 1 (did not pass) : I didn't notice the fact that some of the init calls under start_kernel defined in init/main.c required the scheduler. The problem with this is, we need to first call lkl_start_kernel(...) which calls the init functions and since we have already prioritized the DCE threads over the kernel threads, the kernel didn't initialize and simply jumped to the other DCE thread which started with creating the socket even when the network interface wasn't even initialized, and the funny part is the program didn't crash rather it got stuck again.
+
** Produce final DCE releases
 
+
* Solution 2 : I thought it was a problem with the scheduler and decided to ignore it and try to make the patch more specific by making scheduler calls within inet_wait_for_connect and avoiding changes in the universal LKL scheduler.
+
  
* Report 2 : Now, the init calls worked as they used to before and the system came to a point where after the accept call the system switched over to the other DCE task. But, the socket(...) function got stuck the same way as it did in Report 1.
+
= Weekly Reports =
 +
* '''Community Bonding Period''' (May 17 - June 7)
 +
** Figured out possible Scheduling bottlenecks in LKL in blocking network calls. [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=kix.h6e9x86v84qy Section 4.5]
 +
** Developed a beta docker port for ns-3-dce. [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=kix.qn5nf4lqt7uj Section 1.4]
 +
** Discussed the bright side of porting the latest Linux kernel using the net-next-nuse architecture.
 +
** Discussed possible regression tests for verifying both performance and results.
 +
* '''Week 1''' (June 7 - June 14)
 +
** Implemented the first Linux Kernel-5.12 port for DCE. [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=id.ixtd8h2ia3hf Section 5]
 +
** Passed 5 tests/examples. [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=kix.pqkknl5dxvb2 Section 5.12]
 +
** Initiate talks with LKL team to get reviews on a possible SMP port of LKL. [https://docs.google.com/document/d/1o3xsukgDN9e4-q8n6KbLDX2c9fTxKIhRhimDsn4ivr8/edit#bookmark=kix.h6e9x86v84qy Section 4.5]
 +
* '''Week 2''' (June 14 - June 21)
 +
** Opened PR to integrate Github Actions Workflow for DCE [https://github.com/direct-code-execution/ns-3-dce/pull/118 #118]
 +
** Opened PR to the DCE repo for adding support for custom Glibc build [https://github.com/direct-code-execution/ns-3-dce/pull/117 #117]
 +
** Opened PR to Bake to support pulling and building required dependencies [https://gitlab.com/nsnam/bake/-/merge_requests/9 !9]
 +
** Got patches on pyViz dependency checks and configure_arguments attribute for depends_on field, merged into upstream bake : [https://gitlab.com/nsnam/bake/-/merge_requests/8 !8] and [https://gitlab.com/nsnam/bake/-/merge_requests/7 !7]
 +
** Initiated discussions on issues with current Bake environment and fixes to problems like regexp based file lookups
 +
** Debugging Linux kernel timekeeping inconsistencies, net-device xmit packet loss and untimely socket connection request timeout over custom(P2P,Csma,Wifi etc.) registered net-device
 +
* '''Week 3''' (June 21 - June 28)
 +
** Integrated ns-3-dev way of generating Python Bindings for DCE. (PR yet to be made)
 +
** Identified possible positions where packets are being dropped in the IP Layer for the UDP protocol, in the Linux Kernel.
 +
* '''Week 4''' (June 28 - July 5)
 +
** Figured out specific setuptools(50.3.2) and setuptools_scm(5.0.0) versions which supports building python bindings on Ubuntu-16.04
 +
** Identified Linux kernel [https://git.kernel.org/pub/scm/linux/kernel/git/torvalds/linux.git/commit/?id=031a414b80a9cbfa0a6d3f334e7299f6bfb1654b commit] which valides packets with CHECKSUM_PARTIAL (possibly linked to hardware offloading)
 +
** Linux-4.9.273 (LTS release with BBR support) : 35 tests passed
 +
** Linux-5.10.47 (LTS release) : Fixed timeout issues, spotted inconsistent results and one-way traffic due to probable packet drops.
 +
** Repos :
 +
*** Linux-4.9.273 : https://github.com/ParthPratim/net-next-nuse-4.9.273
 +
*** Linux-5.10.47 : https://github.com/ParthPratim/net-next-nuse-5.10.47
 +
* '''Week 5''' (July 5 - July 12)
 +
** DCE-Linux-5.10.47 : Fixed NULL current task pointer : [https://github.com/ParthPratim/ns-3-dce/commit/36e0b6453117c8da6bf37b733b9aafb566fdb99c commit]
 +
** Linux-5.10.47 : Dropped down to net-next-nuse's iterative sysctl interface, Fixed failure on encountering link in table
 +
** Linux-5.10.47 : Setup ethtool , Tested offloading and checksum operations and fetched kernel device properties
 +
** Linux-5.10.47 : Patch for HW checksum BUG_ON and xmit skb_checksum_help : [https://github.com/ParthPratim/net-next-nuse-5.10.47/commit/f528d9c556ea6023bc7fdd4cf720c4910f62f0fa commit]
 +
** Python --apiscan argument for DCE : Repo for [https://github.com/ParthPratim/ns-3-dce/tree/pybind-apiscan pybind-apiscan]
 +
** Python Bindings for DCE : Pull Request [https://github.com/direct-code-execution/ns-3-dce/pull/120 #120]
 +
* '''Week 6''' (July 12 - July 19)
 +
** Updated issue trackers with current issues and patches for each
 +
*** ns-3-dce : https://github.com/ParthPratim/ns-3-dce/issues
 +
*** net-next-nuse-5.10.47 : https://github.com/ParthPratim/net-next-nuse-5.10.47/issues
 +
** Implemented an fopencookie based stdio Read/Write/Close/Seek callback redirection : [https://drive.google.com/file/d/1KfzLcaqe6wOzeByOJuaJC2ryt2SnVo4X/view?usp=sharing patch]
 +
** Fixed invalid I/O stream selection in fopecoolie implementation
 +
* '''Week 7''' (July 19 - July 26)
 +
** Fixed bug caused by this [https://patchwork.ozlabs.org/project/glibc/patch/20190312130235.8E82C89CE49C@oldenburg2.str.redhat.com/ glibc commit for 2.30+], by erasing the DF_1_PIE bit, set in the flags entry of the Dynamic Linking Section of the ELF header of the probable PIE executable : [https://github.com/ParthPratim/ns-3-dce/commit/c9a8b401034bca2d0242a9eaf80adf98432428a0 commit]
 +
** PIE flag removal is implemented to be automatic and will be initiated only when a file is being copied into elf-cache/0 and it's a valid ELF valid(starts with \177ELF\002) with valid elf header.
 +
** dce-umip-nenmo : Identified that the socket polling events are being enqueued and dequeued from the rr task scheduler m_active queue, making it never return, but only return for specific cases like RngRun=2, but never returns back on gdb no matter what the conditions are.
 +
** Also, the LinuxSockImpl::Poll(...) which should probably be called whenever the polling event returns back with an event(POLLIN, POLLOUT, etc.), which would further call applications like PacketSink to increase the number of RX bytes, is never being called.
 +
* '''Week 8''' (July 26 - Aug 2)
 +
** Discussed with mentors on DCE releses and the order of priority of each.
 +
** Started work on DCE-1.11 release with Tom Sir
 +
** Design Decisions and discussions with mentors in progress for docker based DCE setup
 +
** Tom Sir and I discussed the bug in dce-umip-nemo, and DCE's LTE script failures, and possible causes behind it.
 +
** Started work on the possible fix suggested by Tom Sir to DCE's LTE script failures [https://github.com/direct-code-execution/ns-3-dce/issues/122 #122]
 +
* '''Week 9''' (Aug 2 - Aug 9)
 +
** Debugged dce-iperf script failures when using the Linux stack. Ipv4Linux implementation in DCE, works differently as compared to ns-3's Ipv4L3Protocol. More technical details about why the script fails can be found [https://github.com/direct-code-execution/ns-3-dce/issues/125 here]
 +
** Tom Sir suggested a patch for the dce-iperf example script to avoid static node index references while using GetAddress(). PR [https://github.com/direct-code-execution/ns-3-dce/pull/126 #126]
 +
** Tom Sir and I debugged the LTE script issues with Mac48Address usage in DCE(constraint due to net-next-nuse's implementation). [https://github.com/direct-code-execution/ns-3-dce/issues/122 #122]
 +
** Discussed DCE docker build design and implemented a protoype. [https://github.com/ParthPratim/dce-docker-beta/tree/main Docker Repo]
 +
** We tried to close all (known) issues for a possible DCE-1.11 release
 +
** Started work on getting a DCE-1.12 release for Ubuntu-20.04
 +
* '''Week 10''' (Aug 9 - Aug 16)
 +
** Participated in code review in a new PR on fopencookie based FILE stream callback redirection [https://github.com/direct-code-execution/ns-3-dce/pull/128 #128]
 +
** Figured out an execve based edge case where the implementation failed
 +
** Also, tried to provide technical specifications and reasons for the failure above. [https://github.com/direct-code-execution/ns-3-dce/pull/128#discussion_r687856219 Details here]
 +
** Opened PR for adding --apiscan feature to DCE [https://github.com/direct-code-execution/ns-3-dce/pull/129 #129]
 +
** Followed up with possible fixes on suggestions from Tom Sir on my PR #129
 +
= Final Project Evaluation Report =
 +
The final project evaluation report is hosted on : https://ns-3-dce-linux-upgrade.github.io/
  
* Why did this happen?
+
Please read through the report which covers details of all the work done during GSoC 2021 on this project.
  
* Turns out it's not our fault, it's more like how LKL was designed to work. Remember, I spoke about LKL being a uniprocessor system. So when we initialize LKL with lkl_start_kernel it basically does a bunch of things including running init calls such as setting the thread and cpu mutex locks and semaphores. One of the most important things is lkl_cpu_change_owner(lkl_pthread_t) which operates on the cpu variable of type lkl_cpu. Now the lkl_cpu has a lot of fields, some of the important once :
+
= Artifacts for Workshop on ns-3 publication =
** lock : a mutex lock to decide which host thread currently holds access
+
** owner : thread id of the cpu locker owner
+
** count : no of times the current thread acquired the lock
+
Now, the lkl_cpu_change_owner function checks if the count is not > 1 i.e the lock has not been provided to any other thread(including itself) for more than once and then changes the owner to itself keeping the count intact.
+
  
* So, when our socket(...) system call is run using the lkl syscall APIs, it enters the lkl_syscall(...) function which first requires it to get the cpu lock using lkl_cpu_get(...), now since for example Thread 1 ran lkl_start_kernel and the first syscall(namely accept) and acquired a lock already, and while it was processing, the scheduler decided to switch to another Thread 2, which issued another syscall, which in our case in socket(...) it will be waiting on a mutex for the first function to return so that it calls lkl_cpu_put() and the lock is freed for Thread 2 to acquire.  
+
A [https://dl.acm.org/doi/10.1145/3532577.3532606 Workshop on ns-3 paper] based on the work in this project was published in June 2022This section of the wiki page describes how to reproduce the simulation data presented in that paper.
  
* Now, there's something else to keep a check to. Even if we ran lkl_cpu_change_owner for every syscall, it would be inconsistent and would obviously lead to failures but even if it doesn't fail, LKL would guarantee it would fail. LKL works on the idea that only one host thread acquires the lock and thus when we manually make the previous thread drop the lock using lkl_cpu_put, make the current thread as host using set_thread_flag(TIF_HOST_THREAD) and then change the cpu owner, the subsequent syscalls would fail the cpu.count keeps increasing which has an upper bound of 1.
+
Figures 4, 5, and 6 all provide performance results that are dependent on both how the software was compiled (optimizations) and on the underlying machine CPU.  It will be difficult to reproduce precisely the same results on a different machine, but similar trends in the data should be reproducible.
  
* There's one legal way of changing cpu owners and that is through the default LKL scheduler function schedule(...) which makes context_switch(...) and calls the __switch_to(prev,next) function defined in arch/lkl/threads.c which legally changes the cpu owner from the prev to next, where prev and next are of type struct task_struct.
+
It should also be noted that in Figures 4 and 5, the curves labeled 'Tazaki' are experimental data points that were copied from a previous paper by the author and not reproduced by us.
  
* I then decided to work simultaneously on the writing an SMP interface of LKL based on x86 and arm64 architectures and porting net-next-nuse to Linux-5.12.  
+
Figure 4 and 5 data were generated on a machine running Ubuntu 16.04 with an Intel Core i7-4770 3.40 GHz CPU.  Figure 6 data was generated on a machine running Ubuntu 20.04 with an Intel Core i7-1065G7 1.30 GHz CPU.
  
* Opened a thread on LKL developers lists to get expert suggestions on design and implementation of the project integration in DCE : [https://www.freelists.org/post/linux-kernel-library/LKL-Integration-into-DCE LKL Integration into DCE]
+
Both ns-3-dce and ns-3.34 were build in optimized mode, and other software (net-next-nuse-4.4.0 and ELF loader) were built with the standard Bake configuration.
  
* net-next-nuse port : Linux-4.5 Successful Build and Linux-5.12 defconfig generation report :
+
DCE software used is found on the [https://github.com/tomhenderson/ns-3-dce/tree/performance performance] branch of Tom Henderson's ns-3-dce repository.  This branch as of commit a13b443 (Feb. 18, 2022) was used. This branch is consistent with the DCE 1.11 release, except that additional Bash scripts were added to generate each curve, and the program `example/dce-udp-perf.cc` was slightly modified as compared with the DCE 1.11 release.
** I could get Linux-4.5 to work(tested it with DCE too) with some work. To be specific I had to do the following things :  
+
*** The net-next-nuse Makefile under arch/lib uses objcopy and nm to rename some symbols exported by the kernel using the EXPORT_SYMBOL(...) call and maps it to rumpns_<symbol-name>. These symbols are usually rewritten by net-next-nuse as only a few to_keep files are compiled to lower the size of the shared library and also to get control over certain important parts of the kernel such as the paging service, scheduler, and proc_sysctl interface. Some functions in Linux-4.5 had been written in a different way as compared to Linux-4.4, for example rather than exporting put_page(responsible for handling compound or single pages) has been replaced with __put_page, so had to rewrite the implementation a bit.
+
*** It also introduces slib under memory management which is a memory page slab allocator making use of some dce/nuse routines too, much inspiration has been drawn from the slob allocator(not linked in the library).  
+
*** Had to modify some of the include/linux/slab.h to include the CONFIG_SLIB macro so that only the functions rewritten in mm/slib.c (custom written) could be included in the final build, as it has already enabled in Kconfig.
+
*** Setup the proc_sysctl interface based on the following commit
+
*** I think moving over to Linux-4.6 wouldn't be much tough too, though I haven't tried it yet.
+
** I was a little too curious to get my hands on the latest Linux kernel, so I tested Linux-5.12 too, and with the following changes mentioned below could get the first step(of the two steps) needed to generate the net-next-nuse library to complete successfully (i.e. make defconfig ARCH=lib)
+
*** Identified changes in defconfig process, so had to change the order of defconfig storage which should specifically be under arch/$(ARCH)/configs and also how defconfig were being called on the Kconfig linux build configuration file.
+
*** Kconfig scripting language has evolved quite a bit since net-next-nuse-4.4 release so had to modify the way Kconfig was using environment variables according to this commit by linux
+
*** A significant amount of work is yet to be done in order to make (make library ARCH=lib) work. I'm working on it too.
+
* I also worked on creating a Docker setup of ns-3-dce which would add the following advantages
+
** Lower down the current 12+ GB disk usage to 7GB (just 720 MB more than the previous release). This additional 1 GB is due to a patch I made that enables ns-3-dce to build using a custom Glibc-2.31 setup, so that features like vtable hijacking used by dce to redirect functions like fopen and fseek to custom dce defined implementations)
+
** In the dce-docker-beta folder(my git repo) there is a bake directory and any changes made there gets synced with the docker internal bake directory, so users can both run simulation scripts, as well as make development changes to all the projects in the current bake installation.
+
* My beta docker ns-3-dce test image can be tested using the following commands(on any machine which can run docker and docker-compose) :
+
  
Note : Currently docker has to be run under sudo, but there are ways to avoid this (by creating a docker user group and adding the current user to the group) and a init script for this could be created for this which has to be run just once.
+
For DCE 1.11 on Ubuntu 16.04, the ELF loader (elf-loader) must be compiled and used for some curves.  ELF loader is not available for Ubuntu 20.04 at the time of this writing.
  
<code>
+
== Figure 4 ==
git clone https://github.com/ParthPratim/dce-docker-beta.git
+
  
cd dce-docker-beta
+
The curve labeled 'DCE-1.11 ELF' (which uses DCE kernel mode) can be reproduced using the 'figure3-elf.sh' script, which will generate an output file called 'output-fig3-elf'.  This figure used to be numbered Figure 3 in an earlier version of the paper.  The data points plotted are generated by dividing the values in column 4 (Packets) by the values in column 5 (seconds).  The curve labeled 'Tazaki' was transcribed from the referenced publication.
  
sudo docker-compose up -d
+
== Figure 5 ==
  
sudo docker exec -w / -it ns-3-dce ./setup
+
Figure 5 provides four curves of generated data, plus a curve labeled 'Tazaki' which was transcribed from the referenced publication.  A Bash script was used for each curve.  The curve labeled 'ns-3' was generated using the 'run-rate-vs-time-ns3.sh' script, yielding a 'rate-vs-time.ns3.dat' file; the curve plotted is the plot of column 3 (rate) vs. column 5 (execution time).  Similarly, the curve labeled 'DCE-1.11 ELF' was generated using the 'run-rate-vs-time-elf-kernel.sh' script, the curve labeled 'DCE-1.11 ELF user' was generated using the 'run-rate-vs-time-elf-user.sh' script, and the curve 'DCE-1.11' was generated using the 'run-rate-vs-time-kernel.sh' script.
  
sudo docker exec -w /bake/source/ns-3-dce -it ns-3-dce ./waf --run dce-linux-simple
+
== Figure 6 ==
  
</code>
+
Figure 6 was generated similarly to Figure 5 but on a different machine (Ubuntu 20.04 with different hardware) and with software labelled DCE 1.12 which is a precursor to the actual DCE 1.12 release.  The scripts used to generate Figure 5 data can also be slightly modified to generate Figure 6 data.
  
To stop the docker instance
+
Unlike Figures 4 and 5, which were based on software in Tom Henderson's 'performance' branch, Figure 6 is based on software staged in Parth Pratim Chatterjee's repositories, which can be built as follows.
  
<code>
+
* '''DCE-1.12 (Native Build, Linux Kernel-4.4.0)'''
sudo docker-compose down
+
<div style="margin-left:5%;">
</code>
+
<code> git clone https://gitlab.com/ParthPratim1/bake.git</code><br>
* I also worked on creating fixes for the circleci interface of ns-3-dce and created a PR for the same too. I'll keep working on it based on Matt Sir's suggestions : https://github.com/direct-code-execution/ns-3-dce/pull/115
+
<code>cd bake</code><br>
 +
<code>git checkout dce-1.12</code><br>
 +
<code>./bake.py configure -e dce-linux-1.12</code><br>
 +
<code>./bake.py download</code><br>
 +
<code>./bake.py build</code>
 +
</div>
 +
* '''DCE-1.12 with net-next-nuse-5.10 (Native Build, Linux Kernel-5.10.0)'''
 +
<div style="margin-left:5%;">
 +
<code> git clone https://gitlab.com/ParthPratim1/bake.git</code><br>
 +
<code>cd bake</code><br>
 +
<code>git checkout dce-1.12-linux-5.10</code><br>
 +
<code>./bake.py configure -e dce-linux-1.12</code><br>
 +
<code>./bake.py download</code><br>
 +
<code>./bake.py build</code>
 +
</div>
 +
* '''DCE-1.12 with net-next-nuse-5.10 on Docker'''
 +
** To avoid running docker as root, follow the first sub-section from the [https://docs.docker.com/engine/install/linux-postinstall/ Docker post-installation article]
 +
<div style="margin-left:5%;">
 +
<code>git clone https://github.com/ParthPratim/dce-docker-beta.git</code><br>
 +
<code>cd dce-docker-beta</code><br>
 +
<code>sudo docker-compose up -d</code><br>
 +
<code>sudo docker exec -it ns-3-dce /bin/bash</code><br>
 +
<code>cd /home/bake</code><br>
 +
<code>git init</code><br>
 +
<code>git remote add origin https://gitlab.com/ParthPratim1/bake.git</code><br>
 +
<code>git fetch origin</code><br>
 +
<code>git checkout docker-new</code><br>
 +
<code>./bake.py configure -e dce-linux-1.12</code><br>
 +
<code>./bake.py download</code><br>
 +
<code>./bake.py build</code>
 +
</div>

Latest revision as of 23:35, 21 June 2022

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Back to GSoC 2021 projects

Project Overview

  • Project Name: Direct Code Execution Modernization
  • Student: Parth Pratim Chatterjee
  • Mentors: Tom Henderson, Apoorva Bhargava, Vivek Jain
  • Project Goals: DCE currently makes use of net-next-nuse to extend the Linux kernel internals like the networking stack to host applications but over the years the project hasn't been updated with the latest releases of the Linux kernel. As Linux progressed with newer releases, a major part of the source code changed, making previous glue code incompatible with the newer implementations of the network stack as some of the init calls and function usage changed significantly making migration to newer releases non-trivial. This project aims at enabling support for latest Linux kernel features and toolchains in the DCE environment with support for the socket networking stack, sysctl interfaces, system call access, etc. without any changes to the user APIs currently being used by host applications. The project aims at incorporating an upgraded net-next-nuse port for Linux +v5.10 or the LKL(Linux Kernel Library) into the DCE environment for host applications to effortlessly make use of Linux kernel stacks with minimum to no change in existing simulation scripts.
  • Repository:
  • About Me: I'm a freshman Computer Science undergraduate student at Kalinga Institute of Industrial Technology, Bhubaneshwar, India. I have a keen interest in Linux internals and computer networking. I was a grand prize winner at Google Code-In, 2018 for ns-3 organization, which helped me initially get introduced to DCE. I have an aptitude for Competitive Programming and heavily make use of C/C++, STL and other OOP concepts in solving algorithmic puzzles. I have an experience with C/C++ and Python of more than 3 years, working on projects for numerous Hackathons.

Milestones and Deliverables

The overall project goal is to update DCE such that the latest Linux systems are supported and the latest Linux kernel code could be used.

  • Detailed Project Plan (will be continuously updated throughout the GSoC program duration)
  • Phase 1
    • Ubuntu 20.04 support
      • Goal is that most capabilities presently available for Ubuntu 16 DCE will be available for Ubuntu 20.04 (native)
      • Also contact glibc maintainers about a non-patched solution
      • More about this here
    • Upgrade net-next-nuse Linux kernel support to recent kernel
      • Focus is on the Google BBRv2 kernel (5.10 base): https://github.com/google/bbr
      • Borrow from net-next-nuse-4.4.0 and LKL as appropriate to try to get a new version of net-next-nuse
      • Review existing tests and define/write new tests
      • More about this here
    • Investigate SMP architecture for LKL by querying the LKL developers list
      • More about this (also includes Technical specifications and reasons for the problems) here
  • Phase 2
    • Re-Implement Python Bindings build in DCE
    • Adding --apiscan feature to DCE
    • Debug dce script failures from Phase 1
    • Implement and test fopencookie based FILE stream redirection
  • Phase 3
    • Introduce Github Actions CI to ns-3-dce
    • Collaborate with mentors and project maintainers on resolving existing issues
    • Produce Docker image and documentation to ease installation process
    • Produce final DCE releases

Weekly Reports

  • Community Bonding Period (May 17 - June 7)
    • Figured out possible Scheduling bottlenecks in LKL in blocking network calls. Section 4.5
    • Developed a beta docker port for ns-3-dce. Section 1.4
    • Discussed the bright side of porting the latest Linux kernel using the net-next-nuse architecture.
    • Discussed possible regression tests for verifying both performance and results.
  • Week 1 (June 7 - June 14)
    • Implemented the first Linux Kernel-5.12 port for DCE. Section 5
    • Passed 5 tests/examples. Section 5.12
    • Initiate talks with LKL team to get reviews on a possible SMP port of LKL. Section 4.5
  • Week 2 (June 14 - June 21)
    • Opened PR to integrate Github Actions Workflow for DCE #118
    • Opened PR to the DCE repo for adding support for custom Glibc build #117
    • Opened PR to Bake to support pulling and building required dependencies !9
    • Got patches on pyViz dependency checks and configure_arguments attribute for depends_on field, merged into upstream bake : !8 and !7
    • Initiated discussions on issues with current Bake environment and fixes to problems like regexp based file lookups
    • Debugging Linux kernel timekeeping inconsistencies, net-device xmit packet loss and untimely socket connection request timeout over custom(P2P,Csma,Wifi etc.) registered net-device
  • Week 3 (June 21 - June 28)
    • Integrated ns-3-dev way of generating Python Bindings for DCE. (PR yet to be made)
    • Identified possible positions where packets are being dropped in the IP Layer for the UDP protocol, in the Linux Kernel.
  • Week 4 (June 28 - July 5)
    • Figured out specific setuptools(50.3.2) and setuptools_scm(5.0.0) versions which supports building python bindings on Ubuntu-16.04
    • Identified Linux kernel commit which valides packets with CHECKSUM_PARTIAL (possibly linked to hardware offloading)
    • Linux-4.9.273 (LTS release with BBR support) : 35 tests passed
    • Linux-5.10.47 (LTS release) : Fixed timeout issues, spotted inconsistent results and one-way traffic due to probable packet drops.
    • Repos :
  • Week 5 (July 5 - July 12)
    • DCE-Linux-5.10.47 : Fixed NULL current task pointer : commit
    • Linux-5.10.47 : Dropped down to net-next-nuse's iterative sysctl interface, Fixed failure on encountering link in table
    • Linux-5.10.47 : Setup ethtool , Tested offloading and checksum operations and fetched kernel device properties
    • Linux-5.10.47 : Patch for HW checksum BUG_ON and xmit skb_checksum_help : commit
    • Python --apiscan argument for DCE : Repo for pybind-apiscan
    • Python Bindings for DCE : Pull Request #120
  • Week 6 (July 12 - July 19)
  • Week 7 (July 19 - July 26)
    • Fixed bug caused by this glibc commit for 2.30+, by erasing the DF_1_PIE bit, set in the flags entry of the Dynamic Linking Section of the ELF header of the probable PIE executable : commit
    • PIE flag removal is implemented to be automatic and will be initiated only when a file is being copied into elf-cache/0 and it's a valid ELF valid(starts with \177ELF\002) with valid elf header.
    • dce-umip-nenmo : Identified that the socket polling events are being enqueued and dequeued from the rr task scheduler m_active queue, making it never return, but only return for specific cases like RngRun=2, but never returns back on gdb no matter what the conditions are.
    • Also, the LinuxSockImpl::Poll(...) which should probably be called whenever the polling event returns back with an event(POLLIN, POLLOUT, etc.), which would further call applications like PacketSink to increase the number of RX bytes, is never being called.
  • Week 8 (July 26 - Aug 2)
    • Discussed with mentors on DCE releses and the order of priority of each.
    • Started work on DCE-1.11 release with Tom Sir
    • Design Decisions and discussions with mentors in progress for docker based DCE setup
    • Tom Sir and I discussed the bug in dce-umip-nemo, and DCE's LTE script failures, and possible causes behind it.
    • Started work on the possible fix suggested by Tom Sir to DCE's LTE script failures #122
  • Week 9 (Aug 2 - Aug 9)
    • Debugged dce-iperf script failures when using the Linux stack. Ipv4Linux implementation in DCE, works differently as compared to ns-3's Ipv4L3Protocol. More technical details about why the script fails can be found here
    • Tom Sir suggested a patch for the dce-iperf example script to avoid static node index references while using GetAddress(). PR #126
    • Tom Sir and I debugged the LTE script issues with Mac48Address usage in DCE(constraint due to net-next-nuse's implementation). #122
    • Discussed DCE docker build design and implemented a protoype. Docker Repo
    • We tried to close all (known) issues for a possible DCE-1.11 release
    • Started work on getting a DCE-1.12 release for Ubuntu-20.04
  • Week 10 (Aug 9 - Aug 16)
    • Participated in code review in a new PR on fopencookie based FILE stream callback redirection #128
    • Figured out an execve based edge case where the implementation failed
    • Also, tried to provide technical specifications and reasons for the failure above. Details here
    • Opened PR for adding --apiscan feature to DCE #129
    • Followed up with possible fixes on suggestions from Tom Sir on my PR #129

Final Project Evaluation Report

The final project evaluation report is hosted on : https://ns-3-dce-linux-upgrade.github.io/

Please read through the report which covers details of all the work done during GSoC 2021 on this project.

Artifacts for Workshop on ns-3 publication

A Workshop on ns-3 paper based on the work in this project was published in June 2022. This section of the wiki page describes how to reproduce the simulation data presented in that paper.

Figures 4, 5, and 6 all provide performance results that are dependent on both how the software was compiled (optimizations) and on the underlying machine CPU. It will be difficult to reproduce precisely the same results on a different machine, but similar trends in the data should be reproducible.

It should also be noted that in Figures 4 and 5, the curves labeled 'Tazaki' are experimental data points that were copied from a previous paper by the author and not reproduced by us.

Figure 4 and 5 data were generated on a machine running Ubuntu 16.04 with an Intel Core i7-4770 3.40 GHz CPU. Figure 6 data was generated on a machine running Ubuntu 20.04 with an Intel Core i7-1065G7 1.30 GHz CPU.

Both ns-3-dce and ns-3.34 were build in optimized mode, and other software (net-next-nuse-4.4.0 and ELF loader) were built with the standard Bake configuration.

DCE software used is found on the performance branch of Tom Henderson's ns-3-dce repository. This branch as of commit a13b443 (Feb. 18, 2022) was used. This branch is consistent with the DCE 1.11 release, except that additional Bash scripts were added to generate each curve, and the program `example/dce-udp-perf.cc` was slightly modified as compared with the DCE 1.11 release.

For DCE 1.11 on Ubuntu 16.04, the ELF loader (elf-loader) must be compiled and used for some curves. ELF loader is not available for Ubuntu 20.04 at the time of this writing.

Figure 4

The curve labeled 'DCE-1.11 ELF' (which uses DCE kernel mode) can be reproduced using the 'figure3-elf.sh' script, which will generate an output file called 'output-fig3-elf'. This figure used to be numbered Figure 3 in an earlier version of the paper. The data points plotted are generated by dividing the values in column 4 (Packets) by the values in column 5 (seconds). The curve labeled 'Tazaki' was transcribed from the referenced publication.

Figure 5

Figure 5 provides four curves of generated data, plus a curve labeled 'Tazaki' which was transcribed from the referenced publication. A Bash script was used for each curve. The curve labeled 'ns-3' was generated using the 'run-rate-vs-time-ns3.sh' script, yielding a 'rate-vs-time.ns3.dat' file; the curve plotted is the plot of column 3 (rate) vs. column 5 (execution time). Similarly, the curve labeled 'DCE-1.11 ELF' was generated using the 'run-rate-vs-time-elf-kernel.sh' script, the curve labeled 'DCE-1.11 ELF user' was generated using the 'run-rate-vs-time-elf-user.sh' script, and the curve 'DCE-1.11' was generated using the 'run-rate-vs-time-kernel.sh' script.

Figure 6

Figure 6 was generated similarly to Figure 5 but on a different machine (Ubuntu 20.04 with different hardware) and with software labelled DCE 1.12 which is a precursor to the actual DCE 1.12 release. The scripts used to generate Figure 5 data can also be slightly modified to generate Figure 6 data.

Unlike Figures 4 and 5, which were based on software in Tom Henderson's 'performance' branch, Figure 6 is based on software staged in Parth Pratim Chatterjee's repositories, which can be built as follows.

  • DCE-1.12 (Native Build, Linux Kernel-4.4.0)

git clone https://gitlab.com/ParthPratim1/bake.git
cd bake
git checkout dce-1.12
./bake.py configure -e dce-linux-1.12
./bake.py download
./bake.py build

  • DCE-1.12 with net-next-nuse-5.10 (Native Build, Linux Kernel-5.10.0)

git clone https://gitlab.com/ParthPratim1/bake.git
cd bake
git checkout dce-1.12-linux-5.10
./bake.py configure -e dce-linux-1.12
./bake.py download
./bake.py build

git clone https://github.com/ParthPratim/dce-docker-beta.git
cd dce-docker-beta
sudo docker-compose up -d
sudo docker exec -it ns-3-dce /bin/bash
cd /home/bake
git init
git remote add origin https://gitlab.com/ParthPratim1/bake.git
git fetch origin
git checkout docker-new
./bake.py configure -e dce-linux-1.12
./bake.py download
./bake.py build