Difference between revisions of "Network Simulation Cradle Integration"

From Nsnam
Jump to: navigation, search
(Try to document some hints for porting new stacks to nsc.)
m (Add note that initcalls need to be declared $ignored unconditionally)
Line 330: Line 330:
(which tells the globaliser to NOT globalise the symbol at all) to
(which tells the globaliser to NOT globalise the symbol at all) to
This is safe to do for read-only symbols, for example initcalls)
This is safe to do for read-only symbols, for example initcalls).
In fact, _ALL_ initcalls must be marked $ignored. Otherwise, the linker does the wrong thing[tm]
and you'll find that the nsc library will segfault early (in do_initcalls) because the
init function was NULL.
Also see the [https://secure.wand.net.nz/mercurial/nsc/file/eed42331d91b/globaliser/README globaliser readme]
Also see the [https://secure.wand.net.nz/mercurial/nsc/file/eed42331d91b/globaliser/README globaliser readme]
in nsc.
in nsc.

Revision as of 20:47, 21 August 2008

Network Simulation Cradle Integration

Network Simulation Crade (nsc) is a project to allow real-world network stacks to be used in simulated networks. A porting effort to bring nsc to version 3 of the network simulator was started by Tom Henderson. This Project continues this work and also will update at least the Linux network stack to current git (2.6.26).


NSC provides its API in the form of a number of classes which are defined in sim/sim_interface.h in the nsc directory.

These are:


INetStack contains the 'low level' operations for the operating system network stack, e.g. in and output functions from and to the network stack (think of this as the 'network driver interface'. There are also functions to create new TCP or UDP sockets.


This is called by NSC when a packet should be sent out to the network. This simulator should use this callback to re-inject the packet into the simulator so the actual data can be delivered/routed to its destination, where it will eventually be handed into Receive() (and eventually back to the receivers NSC instance via INetStack->if_receive() ).


This is the structure defining a particular connection endpoint (file descriptor). It contains methods to operate on this endpoint, e.g. connect, disconnect, accept, listen, send_data/read_data, ...


This contains the wakeup callback, which is called by NSC whenever something of interest happens. Think of wakeup() as a replacement of the operating systems wakeup function: Whenever the operating system would wake up a process that has been waiting for an operation to complete (for example the TCP handshake during connect()), NSC invokes the wakeup() callback to allow the simulator to check for state changes in its connection endpoints.

NSC Todo and Wish list


Get protocols like SCTP and DCCP to work with nsc. The SCTP code in the 2.6.26 port works, but needs proper integration within ns-3-nsc. DCCP code is included in 2.6.26, but not compiled at the moment due to a problem with dccp connection shutdown (nsc gets stuck in a loop forever).


  • Integrate nsc into ns-3 without circumventing ns-3s tcp/ip infrastructure

Done as of June 1st, although there is room for improvement (e.g. removing redundant code, cleanups, etc).

  • allow nsc to use layer2 code directly without having to piggyback data inside ns3-tcp

Done, with a twist. Nsc uses layer 3 directly. For this to work, nsc-tcp-l4 does no longer include the nsc ipv4 header in the packet. Instead, the ns-3 ip header is used. In the long run, it would be better to use the complete tcp/ip packet, e.g. for ECN.

  • extend ns-3 API so users can create "Linux-Nodes", "FreeBSD-Nodes", etc.

Partially done. One can call e.g. 'internet.SetNscTcp("Linux");' before internet.Install(). This also allows to assign different nsc stack to some nodes.

  • Think about the ns-3-nsc API. It is desirable to allow a simulation to use nsc-setsockopts and tune the nsc stack via available sysctl tunables (e.g. for disabling SACK support on Linux at run time).

Mathieu Lacage suggested to use ns-3s Attribute facility for this. The current plan is to:

  • Add an nsc API that allows a user to iterate through all the sysctls provided by the current stack

done as of early July.

  • then, using this NSC API, dynamically build a tree of attribute objects in ns-3 that reflect the sysctl hierarchy.

A simple sysctl interface (that resembles the sysctl(8) utility, i.e. sysctl(const char *name, const char *value) has been added to nsc, its currently available in the Linux 2.6.18 and 2.6.26 stacks in the nsc repository.

  • add an errno mapping to allow the simulator to determine to detect the exact cause of errors (e.g. EAGAIN vs. ECONNRESET).


  • work on 2.6.26 again to make it build and work on x86-64.



  • rework ns-3-nsc, especially nsc-tcp-l4-protocol.cc.

The stack initialization/handling as well as the NSC-TCP related code is handled in nsc-tcp-l4-protocol.cc. This is wrong, and the two functionalities should be separated. As the SCTP is now available again in nscs Linux 2.6.26 port, this refactoring is being worked on as of 8th July because its a prerequisite to make sctp work out-of-the-box in ns-3-nsc.

  • Work on the ns3-nsc build system so nsc is downloaded (and compiled) automatically.

ns-3-nsc now understands ./waf configure --nsc, that is, without the --nsc switch a 'normal' ns-3 will be built. Code is retrieved/updated automatically if the --nsc argument is specified and the host system is x86-based.


This is here to document some parts of the actual ns-3/nsc plumbing.

The two main parts are:

  • nsc-tcp-l4-protocol, a subclass of Ipv4L4Protocol (and two nsc classes: ISendCallback and IInterruptCallback)
  • nsc-tcp-socket-impl, a subclass of TcpSocket

There is also a factory to create new NSC sockets (nsc-tcp-socket-factory-impl.cc).


This is the main class. Upon Initialization, it loads an nsc network stack to use (via dlopen()). Each instance of this class may use a different stack. The stack (=shared library) to use is set using the SetNscLibrary() method (at this time its called indirectly via the internet stack helper). The nsc stack is then set up accordingly (timers etc). The NscTcpL4Protocol::Receive() function hands the packet it receives (must be a complete tcp/ip packet) to the nsc stack for further processing. TO be able to send packets, this class implements the nsc send_callback method. This method is called by nsc whenever the nsc stack wishes to send a packet out to the network. Its arguments are a raw buffer, containing a complete Tcp/ip packet, and a length value. This method therefore has to convert the raw data to a Ptr<Packet> usable by ns-3. In order to avoid various ipv4 header issues, the nsc ip header is not included. Instead, the tcp header and the actual payload are put into the Ptr<Packet>, after this the Packet is passed down to layer 3 for sending the packet out (no further special treatment is needed in the send code path).

This class calls nsc-tcp-socket-impl both from the nsc wakeup() callback and from the Receive path (to ensure that possibly queued data is scheduled for sending).


This implements the nsc socket interface. Each instance has its own nscTcpSocket. Data that is Send() will be handed to the nsc stack via m_nscTcpSocket->send_data(). (and not to nsc-tcp-l4, this is the major difference compared to ns-3). The class also queues up data that is Send() before the underlying descriptor has entered an ESTABLISHED state. This class is called from the nsc-tcp-l4 class, when the nsc-tcp-l4 wakeup() callback is invoked by nsc. nsc-tcp-socket-impl then checks the current connection state (SYN_SENT, ESTABLISHED, LISTEN...) and schedules appropriate callbacks as needed, e.g. a LISTEN socket will schedule Accept to see if a new connection must be accepted, an ESTABLISHED socket schedules any pending data for writing, schedule a read callback, etc.

ns-3-nsc integration diagram

ns-3-nsc plumbing

Note that nsc-tcp-socket-impl does not interact with nsc-tcp directly: instead, data is redirected to nsc. nsc-tcp calls the nsc-tcp-sockets of a node when its wakeup callback is invoked by nsc.

Integration Notes

This is a loose transcript of an irc meeting about merging ns-3-nsc into ns-3-dev. Participants were Tom Henderson, Sam Jansen and Florian Westphal.


  • build 64bit + integration of nsc into ns-3 build system
  • ipv4 -- whether we can attempt to do the full TCP/IP stack and how that might impact the IPv4 refactoring and API changes we're otherwise considering
  • routing and multiple interfaces per node
  • socket errors + errno not being handled correctly
  • example scripts and documentation
  • sigcomm demo

64bit and build system integration

Sam said that he had updated the Linux 2.6.18 stack to work on x86-64 and that he would update other stacks once nsc had more of a build system. Florian mentioned that he has added an --nsc switch to enable nsc conditionally in ns-3. Action: Florian will add auto downloading of nsc and extra check to detect when nsc is being built on an unsupported platform.


Sam mentioned that SCTP is working again in the Linux 2.6.26 port, and that the ns-3 integration was more in terms of nsc being a protocol stack, rather than a TCP model. Florian said the main problem with ipv4 was that the ipv4 header was removed by ipv4-l3 processing.

routing and multiple interfaces

Sam stated that it was easy to add more interfaces to an nsc node and that the main problem is routing, so some route glue code needs to take an ns-3 route and program the corresponding nsc stack. Florian added that one would also need to map every ns-3 interface to an nsc interface. Sam and Florian agreed that this was a large project.

socket errno

Florian and Sam agreed that this is easy to solve. Action: Florian will fix this before merge.

example scripts

All agreed that nsc needs better tests/examples.

Action: Florian will write a new nsc example scenario.

sigcomm demo

Florian will help to add some nsc features to the demo script(s). Tom mentioned he'd like to build some documentation then for users to start to use nsc if they want. Sam said he and Florian would make sure to add more of that when integration got a bit further along.


nsc will be moved to the WAF build system. Florian will first work on a new nsc example scenario, then on the waf/nsc build system, Sam will help out as he can. After this Florian will fix the errno problem.

  • integrate into the waf system
  • fix 64 bit builds
  • add to nightly regression (and debug those issues)
  • finish up the errno integration
  • have a code review to plan to merge

NSC Stack Porting Hints

This is a list of suggested steps to take when porting a new stack (or a different version of an existing stack) into nsc.

For Linux, there is a script, linux-2.6.26/bootstrap.sh, which can be used to update the set of files.

The audit.sh script (in the main nsc directory) can be used to diff an existing nsc stack codebase versus an unmodified kernel version release.

Disable the Globaliser

First, one should disabled the globaliser entirely. The globaliser is only required to run more than one instance of the stack simultaneously. Its not needed to compile the code and link the shared library and only complicates things in the beginning.

Compile stack code

If you need to add a new stack from scratch, it is a good idea to first look at few other stacks and how they've been changed to work within NSC. The lwip stack is the most simple one, since its a separate stack not tied to an operating system. You'll need to read through your operating systems code to figure out what belongs to the network subsystem, the 'necessary infrastructure', what needs to be provided by cradle code, etc.

After the code has been brought up to date, (or the network stack has been isolated) one can progress to compiling the stack code. It is unlikely that this will work without some tweaks, as stack code called from NSC glue code may have been changed, i.e. functions may have been renamed (or their arguments have been changed), or some defines are missing, etc. This will require careful code reading to solve. If ypu're updating an existing stack it is often helpful to search the stack change log (for the Linux kernel, use e.g. 'git log -S<keyword>' to find particular changes of interest).

Provide missing functions

When a shared library has been produced, it is likely that it will contain a lot of unresolved symbols (see nm -u output), i.e. references to kernel helper/library code currently not found in the nsc glue code. For each of these, you need to find where the particular function/object is located in the kernel. Then you'll need to decide if you should include the complete .c file that defines the function/object, or provide a complete or simplified implementation in the nsc cradle code. Rule of thumb: Relatively 'standalone' library helpers, for example generic, kernel 'library code' can be included verbatim, but you should avoid including kernel infrastructure outside of the network core. If you are unsure, you can try to add the new file, and compare nm -u output before and after your change. If you see new unresolved symbols, then it might be more advisable to avoid including the file .

When it comes to adding code to the cradle code, it is recommended to first add a 'stub function' in nsc/unimplemented.c, that is

return_type function_name (arguments) {

 return return_type;


UNIMPLEMENTED() is a cradle code macro that will result in a fatal error/program termination if the function is called. You'll find that a lot of functions are not called at all in NSC and can therefore be kept around as stub functions.

Functions that are being used are later moved to nsc/implemented.c. Simple/standalone functions can be copied verbatim, other functions need to be reworked to interface them with libc (in the linux kernel, for example kmalloc and printk). Other functions are implemented so that they only perform the work required in the NSC/network subsystem context, see for instance new_inode()/sock_alloc_inode() in nsc/implemented.c in the linux 2.6.26 stack.

In the next step, you can try to use the library in a simulator. Keep in mind that because the globaliser was disabled, you should make sure that the stack is only used on a single node.

If things work correctly, the globaliser is the next step, so (re)enable the globaliser and rebuild.

Enable globaliser

Next, check the shared library for unresolved symbols again.

Its possible you'll see undefined symbols named global_XXXX. Solving this will usually boil down to either adding


which tells the globaliser that the symbol requires some 'magic' processing or,


(which tells the globaliser to NOT globalise the symbol at all) to the This is safe to do for read-only symbols, for example initcalls). In fact, _ALL_ initcalls must be marked $ignored. Otherwise, the linker does the wrong thing[tm] and you'll find that the nsc library will segfault early (in do_initcalls) because the init function was NULL.

Also see the globaliser readme in nsc.

Current Status

The ns-3-nsc integration repository contains the current status. This port currently uses the nsc mercurial development tip. The nsc glue code has been moved off to its own files in src/internet-node:

  • src/internet-node/nsc-tcp-socket-impl.cc contains ns-3s tcp-socket code, with nsc modifications.
  • src/internet-node/nsc-tcp-l4-protocol.cc contains the ns3 tcp-l4 code, with nsc modifications.
  • src/internet-node/nsc-tcp-socket-factory-impl.cc is like tcp-socket-factory-impl.cc , except that

it uses the NscTcpL4Protocol class.

AddInternetStack() has been extended to allow setting a TcpStack to use, this allows a simulation to switch to a particular stack supported by NSC.

NSC is now hosted at WAND (NSC repository). A private branch with more experimental changes is located here.