GSoC2018 DCE Upgrade

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

Return to GSoC 2018 Accepted Projects page.

Project overview

• Project name: Direct Code Execution upgrade

• Student: Sourabh Jain

• Mentors: Matthieu Coudron, Natale Patriciello

• Abstract: Direct Code Execution is a framework for ns-3. It facilitates to execute Linux kernel and Linux userspace networking applications inside ns-3 without changing their code. In order to provide such functionality, DCE integrates network simulator with custom Linux kernels like net-next-sim or net-next-nuse (LibOs). But net-next-nuse has not kept pace with new versions of Linux kernel. This has been the major hindrance for DCE to use the latest Linux kernel. An active alternate to LibOs is Linux Kernel Library (LKL) which provides similar features of LibOs and supports latest Linux kernel versions. In this project, DCE will extend to support the networking stack of LKL and embed real libc in LKL. The motivation behind the Linux Kernel Library (LKL) is to provide an interface to reuse Linux kernel subsystem with minimal efforts. Userspace application currently can reuse the virtual file system, networking stack, task management, scheduling and memory management subsystem of Linux using the APIs provided in LKL. LKL builds Linux kernel as shared object file which can be easily loaded into userspace and have Linux kernel feature directly available to userspace applications. In this project, we reuse the networking stack subsystem of Linux using LKL.

• Link to LKL Repo Here

• Link to ns-3-dce Repo Here

• About me: I am a master student at National Institute of Technology, Karnataka India. Currently, I am building a framework over ns-3-dce that automates the testing of TCP and AQM by using standard tools like netperf and iperf.

Technical Approach

In general, LKL offers userspace applications to use different kernel subsystems like the virtual file system, network stack, and scheduling. The plan is to adapt network stack functionality using LKL system call API. At the time of binary deployment over a node, DCE will load the LKL compiled binary. It then searches for sim_init function and tells to LKL the required set of function for a node. In return, LKL will give a structure of pointer to those functions so that DCE can use it later. In order to make DCE compatible with LKL, we need to adapt the host related function from LKL. There are approximately 35-40 host-related functions which are declared as structure lkl_host_operations in LKL. The objective is to redefine lkl_host_operation in the DCE environment. In addition to redefining those functions for DCE, we need to add glue to code to map LKL host functions to redefined DCE functions.

Steps to create DCE Upgrade Working Environment on Ubuntu

Note: Tools needed to download the source code of DCE and LKL are mercurial and git.

Install Direct Code Execution

• Required packages for DCE: This is a set of packages needed to install and run ns-3-dce.

 sudo apt install cmake cvs git bzr unrar p7zip-full autoconf build-essential bison flex g++ libc6 
 sudo apt install libc6-amd64 libdb-dev libexpat1-dev libpcap-dev libssl-dev python-dev python-pygraphviz 
 sudo apt install python-pygoocanvas  python-setuptools qt4-dev-tools

• Clone the Bake: Software utility used to download and build ns-3-dce.

 hg clone http://code.nsnam.org/bake bake

• Environment Variables: Following are the environment variables required to build the ns-3-dce

 export BAKE_HOME=`pwd`/bake
 export PATH=$PATH:$BAKE_HOME
 export PYTHONPATH=$PYTHONPATH:$BAKE_HOME

• Create a folder dce: This folder will contain ns-3-dce source code

 mkdir dce
 cd dce

• Configure DCE: This command will configure the ns-3-dce. Here -e indicates DCE version.

 bake.py configure -e dce-linux-dev

• Download and build ns-3-dce: Commands below will download and build the ns-3-dce and other dependent source code. Option -v is optional for debugging.

 bake.py download -v
 bake.py build -v

The above steps are taken from https://www.nsnam.org/docs/bake/tutorial/html/bake-tutorial.html

Install Linux Kernel library

• Clone the LKL source code

 git clone https://github.com/sourabhjains/linux

• Switch to dce-support branch

 cd linux
 git checkout dce-support  

• Build the LKL

 make -C tools/lkl

• Verify LKL shared library

 ls tools/lkl/lib/

Note: There should be a file called liblkl.so inside tools/lkl/lib

• More about LKL library

 readelf -h liblkl.so  : reads the header
 readeld -d liblkl.so  : read the dynamic section
 ldd liblkl.so         : print the shared object dependencies

Importing LKL library in DCE example

• Set Network stack with LKL library

 Use DceManagerHelper::SetNetworkStack and pass StringValue ("liblkl.so") as third arrgument.
 Set the DCE_PATH to liblkl.so accordingly.

Project Timeline

Week 1 (May 14 - May 20)

• Initialized the project repositories ns-3-dce and LKL and attached it with circleci.
• Added a DCE host for LKL here.
• Defined couple of host functions related to the thread, memory allocation and time.

Week 2 (May 21 - May 27)

• Understood the existing execution flow of net-next-nuse and the call forwarder mechanism to SimExported functions.
• Defined the DceImport and DceExport structures in LKL to handle the Linux system call and the callbacks to DCE.

In oder to avoid the confusion between imported or exported structure we have changed the names of DceImport and DceExport
structure with DceHandle and KernelHandle. The functions handle by ns-3-dce is defined in DceHandle and functions
handle by LKL is defined in KernelHandle.

struct DceHandle {
  int (*vprintf)(struct DceKernel *kernel, const char *str,
      va_list args);
  void *(*malloc)(struct DceKernel *kernel, unsigned long size);
  void (*free)(struct DceKernel *kernel, void *buffer);
  void *(*memcpy)(struct DceKernel *kernel, void *dst, const void *src,
       unsigned long size);
  void *(*memset)(struct DceKernel *kernel, void *dst, char value,
       unsigned long size);
  int (*atexit)(struct DceKernel *kernel, void (*function)(void));
  int (*access)(struct DceKernel *kernel, const char *pathname,
      int mode);
  char *(*getenv)(struct DceKernel *kernel, const char *name);
  int (*mkdir)(struct DceKernel *kernel, const char *pathname,
  mode_t mode);
  ...
}

struct KernelHandle {
  int (*sock_socket)(int domain, int type, int protocol,
      struct DceSocket **socket);
  int (*sock_close)(struct DceSocket *socket);
  ssize_t (*sock_recvmsg)(struct DceSocket *socket, struct msghdr *msg,
          int flags);
  ssize_t (*sock_sendmsg)(struct DceSocket *socket,
          const struct msghdr *msg, int flags);
  int (*sock_getsockname)(struct DceSocket *socket,
      struct sockaddr *name, int *namelen);
  int (*sock_getpeername)(struct DceSocket *socket,
      struct sockaddr *name, int *namelen);
  int (*sock_bind)(struct DceSocket *socket, const struct sockaddr *name,
      int namelen);
  int (*sock_connect)(struct DceSocket *socket,
      const struct sockaddr *name, int namelen,
      int flags);
  ...
}

• Defined the initialization function in LKL, this functions will initiate the Linux kernel and fills the global DceImport and DceExport structure.

Week 3 (May 28 - June 3)

• Enable the LKL to build the DCE host by adding a separate function in Makefile.

define dce_host
  $(call set_autoconf_var,DCE,y)
  $(call set_autoconf_var,VIRTIO_NET,y)
  LDFLAGS += -pie
  CFLAGS += -fPIC
  SOSUF := .so
  $(if $(filter $(1),elf64-x86-64-freebsd),$(call bsd_host))
  $(if $(filter yes,$(dpdk)),$(call virtio_net_dpdk))
  $(if $(filter yes,$(vde)),$(call virtio_net_vde))
  $(if $(strip $(call find_include,fuse.h)),$(call set_autoconf_var,FUSE,y))
  $(if $(strip $(call find_include,archive.h)),$(call set_autoconf_var,ARCHIVE,y))
  $(if $(strip $(call find_include,linux/if_tun.h)),$(call set_autoconf_var,VIRTIO_NET_MACVTAP,y))
  $(if $(filter $(1),elf64-x86-64-freebsd),$(call set_autoconf_var,NEEDS_LARGP,y))
endef

• Fixed the issues with DCE host to build LKL.

Week 4 (June 3 - June 10)

• Added new callbacks in DceImport structure related to pthread functions of DCE.
• Successfully build the LKL with DCE host and generated the LKL shared library.
• Attempted loading shared library of LKL in DCE with dce-iperf.cc example. It failed due to external dependencies like libc.so.

Week 5 (June 11 - June 17)

• Removed all dependency over standard libraries.
• DCE is able to load the LKL shared Library and successfully calls the init function.

Dce is able to load lkllib.so (statically linked library of Linux Kernel) and able to make a call to sim_init call.

void sim_init(struct KernelHandle *kernelHandle, const struct DceHandle *dceHandle, struct DceKernel *kernel)
{
  g_dceHandle = *dceHandle;
  g_kernel = kernel;
  #include "kernel_handle_assignment_generated.c"
  return
}

Tested a callback to Vprintf function defined in ns-3-dce from sim_init function.

• Tested a function from Import structure, LKL is able to make calls to import functions.
• The basic call forwarding from DCE to LKL and LKL to DCE is working.

Week 6 (June 18 - June 24)

• Updated the DCE with DceHandle/KernelHandle structures.
• Added a new kernel architecture Linux Kernel Library (LKL) in ns-3-dce/wscript. Now waf is able to configure the DCE with LKL.

This functionality is added in ns-3-dce/wscript so that DCE can be manually configurable with LKL architecture.
Note: In future we need to update the bakeconf.xml file of bake tool to automate this task.

architectures = ["lkl", "sim", "lib"]
# add include dir to KERNEL_DIR if architecture is lkl.
# This will help DCE to locate dce-init.h and lkl.h
        if (kernel_stack_dir == Options.options.kernel_stack+"/"+architectures[0]):
            kernel_stack_dir = os.path.join(kernel_stack_dir, "include")
        else:
            conf.check(header_name='sim.h',
            includes=os.path.join(kernel_stack_dir, 'include'))

• Added a basic client-server example in ns-3-dce/example to test LKL.
• Ctags is used to automatically generate the members of kernelHandle structure (used an existing script available here).

Currently, ctags has some issue with typeref tag. It does not consider struct keyword as return type of the function.

for example: foo.h

#include <stdio.h>
int first (int a, int b);
struct mystructure second (int a);
long long third (long a);

command: ctags --language-force=C -x --_xformat="%N:%{typeref}:%{signature}" --kinds-C=p foo.h

Output:
first:int:(int a,int b)
second:mystructure:(int a)
third:long long:(long a)

Here, in second function --_xformat misssed the struct keyword.

For now I manually added struct keyword in KernelHandle structure and raised an issue in ctags repository 
[https://github.com/universal-ctags/ctags/issues/1772 here].

Week 7 (June 23 - 29)

• Solved the error generated while configuring ns-3-dce with latest LKL API.
• Used -fpermissive CFLAG while configuring DCE to avoid type cast errors.

ns-3-dce generates invalid conversion type errors while building it with LKL. These errors are avoidable using fpermissive tags.

/home/lev/repo/linux/tools/lkl/include/lkl/linux/virtio_ring.h:146:16: error: invalid conversion from ‘void*’ to ‘lkl_vring_avail*’
[-fpermissive] vr->avail = p + num*sizeof(struct lkl_vring_desc);

• Updated SimSysIterator in LKL.

Week 8 (June 30 - July 7)

• Implemented Socket function handler in LKL
• Created an API for common functions define in DceHandle structure.

Week 9 (July 8 - July 14)

• Added sysctl interface in LKL and updated the LinuxSocketFdFactory: Set and Get methods in DCE, to update and access sysctl attributes.
• Refactored the unnecessary code related to sysctl.
• Fixed the LKL build issue with dce_socket (Now LKL consider dce_socket while building DCE host).
• Understood how DCE maintains the mapping between ns3::NetDevice and Linux devices.
• Understood the LKL APIs that will help to manipulate the net devices.

Week 10 (July 15 - July 22)

• Implemented the Network Devices related functions in LKL.
• Implemented mutex and semaphore related functions in LKL. Now I can provide DCE patches for such functions.
• A function is introduced in DceManager class called Panic to Handle kernel panic situation.

Week 11 (July 23 - July 29)

• Updated the sock_recvmsg and sock_sendmsg functions.
• Completed mutex, semaphore and panic-related patches in ns-3-dce.

The current action against kernel panic is logging the nodeId on which kernel panic occurred and exit the simulation.
DceManager::Panic ()
{
  ...
  NS_LOG_INFO ("Kernel Panic. nodeId: " << nodeId);
  dce_exit (-2);
}
This can be modified and do more than just logging the nodeID.

• Documented the DceHandle and KernelHandle functions in LKL.
• Mapped the lkl_print to ns-3-dce vfprintf.

Week 12 (July 30 - August 5)

• Completed the DCE net device component in LKL.
• Updated Makefile to consider the network related headers while building DCE host in LKL.
• Handled the socket ops related errors by adding a forward declaration for socket structure in DCE socket module.

Week 13 (August 6 - August 12)

• Added pthread function prototype in dce_handle_api.h to avoid warnings.
• Mapped the lkl host operations: ioremap and iomem_access to LKL version.
• facing some issues while building the LKL source code.

- The first issue is, LKL is not able to locate the structure of type file while compiling dce_socket.c. The file structure is
  needed during socket creation. We allocate space for file structure and assign it to the  file attribute of the newly created socket. 
  The problem is that compiler doesn't know the file structure so it isn't able to calculate the size of the file structure by using 
  sizeof operator.

- The second issue is related to an unknown attribute in structure msghdr.  Although the structure msghdr contains the member
  called msg_iter, the compiler still says 'struct msghdr has no member named msg_iter'.

GSoC Project Summary

The components like DCE host, NetDevice, and Socket in LKL have been completed. The overall framework is not yet ready to perform any simulation. We need to run dce-ping example iteratively to figure out the code crash points between LKL and ns-3-dce. This process might need some changes in existing modules to makes ns-3-dce compatible with LKL.

Statistics about the changes done on LKL and ns-3-dce repositories during GSoC 2018

Statistics of LKL repository

Statistics of ns-3-dce repository

Deliverables

Link to Phase 1 tasks

Enabling support for DCE Host in LKL: https://github.com/sourabhjains/linux/tree/dce-support

Link to Phase 2 tasks

NetDevice and Socket module in LKL

Link to Phase 3 tasks

LKL compatible code in ns-3-dce: https://github.com/sourabhjains/ns-3-dce/tree/lkl-integration

Link to GSoC final evaluation

https://ns-3-dce-gsoc-2018.github.io/

Future Work

The future work on this project will be available at the different branch of the same repository.

LKL: https://github.com/sourabhjains/linux/tree/dce

DCE: https://github.com/sourabhjains/ns-3-dce/tree/lkl