App Store Technical Requirements
The long-term goal is to move ns-3 to separate modules, for build and maintenance reasons. For build reasons, since ns-3 is becoming large, we would like to allow users to enable/disable subsets of the available model library. For maintenance reasons, it is important that we move to a development model where modules can evolve on different timescales and be maintained by different organizations.
An analogy is the GNOME desktop, which is composed of a number of individual libraries that evolve on their own timescales. A build framework called jhbuild exists for building and managing the dependencies between these disparate projects.
Once we have a modular build, and an ability to separately download and install third-party modules, we will need to distinguish the maintenance status or certification of modules. The ns-3 project will maintain a set of core ns-3 modules including those essential for all ns-3 simulations, and will maintain a master build file containing metadata to contributed modules; this will allow users to fetch and build what they need. Eventually, modules will have a maintenance state associated with them describing aspects such as who is the maintainer, whether it is actively maintained, whether it contains documentation or tests, whether it passed an ns-3 project code review, whether it is currently passing the tests, etc. The current status of all known ns-3 modules will be maintained in a database and be browsable on the project web site.
Figure caption: Mock-up of future model status page (models and colors selected are just for example purposes)
The basic idea of the ns-3 app store would be to store on a server a set of user-submitted metadata which describes various source code packages. Typical metadata would include:
- unique name
- version number
- last known good ns-3 version (tested against)
- download url
- untar/unzip command
- configure command
- build command
- system prerequisites (if user needs to apt-get install other libraries)
- list of ns-3 package dependencies (other ns-3 packages which this package depends upon)
Build and configuration
- Enable optimized, debug, and static builds
- Separate tests from models
- Doxygen support
- Test.py framework supports the modularization
- Python bindings support the modularization
- integrate lcov code coverage tests
- integrate with buildbots
API and work flow
Assume that our download script is called download.py and our build script is called build.py.
wget http://www.nsnam.org/releases/ns-allinone-3.x.tar.bz2 bunzip2 ns-allinone-3.x.tar.bz2 && tar xvf ns-allinone-3.x.tar.bz2 cd ns-allinone-3.x
In this directory, users will find the following directory layout:
build.py download.py constants.py? VERSION LICENSE README
Download essential modules:
This will leave a layout such as follows:
download.py build.py pygccxml pybindgen ns-3.10 gcc-xml
Open issue: do we want to keep with the practice of including version number in the directory name?
The ns-3.10 directory will contain an ns-3 configuration file, .ns3rc, which specifies the ns-3 modules that should be enabled. The default version of .ns3rc stored in the source code repository will enable all of the ns-3 modules that are considered to be the core of an ns-3 release. For starters, it will include every module that is in ns-3.9.
Here is an example .ns3rc file that enables only the simulator module:
#! /usr/bin/env python # A list of the modules that are enabled. modules_enabled = ['simulator']
Note that ns-3 will first try to use the .ns3rc file it finds in the current working directory, and failing to find that, will check for the .ns3rc file in the ~ directory.
Open issue: If the user chooses the --enable-modules option for waf, should the .ns3rc file be modified to match the modules that were specified?
For typical users, the next step is as follows:
The above will take the following steps:
- build all typical prerequisites such as pybindgen
- cd ns3-10
- ./waf configure && ./waf && ./waf install
The above will install headers into build/debug/ns3/ and libraries for each of the enabled modules specified by the .ns3rc file into build/debug/lib. Each enabled module will be placed in a separate library that has a name of the form libns3-simulator.so for the simulator module.
Open issue: What granularity of modules should we maintain? Continue with core, simulator, common, node? Merge them somehow?
Open issue: How to express and honor platform limitations, such as only trying to download/build NSC on platforms supporting it.
Once the build process is done, there will be a bunch of libraries in a common build directory, and all applicable headers. Specifically, we envision for module simulator, there may be a libns3-simulator.so, libns3-simulator-test.so, a python .so, and possibly others.
We have a few choices for supporting Python. First, note that this type of system provides an opportunity for the python build tools to be added as packages to the overall build system, so that users can more easily build their bindings. We can try to build lots of small python modules, or run a scan at the very end of the ns-3 build process to build a customized python ns3 module, such as:
which operates on the headers in the build/debug/ns3 directory and creates a python module .so library that matches the ns-3 configured components.
Open issue: What is the eventual python API? Should each module be imported separately such as import node as ns.node, or should we try to go for a single ns3 python module?
Or, we could continue to maintain bindings. Another consideration is that constantly generating bindings will slow down the builds.
Tests are run by running ./test.py, which knows how to find the available test libraries and run the tests.
Presently, test.py hardcodes the examples or samples; it needs to become smarter to learn what examples are around and need testing.
Doxygen can be run such as "build.py doxygen" on the build/debug/ns3 directory. I would guess that we don't try to modularize this but instead to run on the build/debug/ns3/ directory once all headers have been copied in.
To configure different options such as -g, -fprofile-arcs, e.g., use the CFLAGS environment variable.
To run programs:
./build.py shell build/debug/examples/csma/csma-bridge
Build all known ns-3 modules:
In the above case, suppose that the program could not download and fetch a module. Here, it can interactively prompt the user "Module foo not found; do you wish to continue [Y/n]?".
Build the core of ns-3, plus device models WiFi and CSMA:
./build ns3-core wifi csma
ns3-core is a meta-module containing simulator common node mobility etc. Open issue: what is the right granularity for this module?
Example third-party ns-3 module
Suppose a research group at Example Univ. publishes a new WiMin device module. It gets a new unique module name from ns-3 project, such as wimin. It also contributes metadata to the master ns-3 build script.
The "download.py" system should have enough metadata to fetch it, whether it is a tarball release or some other kind of release.
Open issue: How does user toggle the download behavior of a module? Is there a sticky "./download.py --disable-module=wimin" option? Or does user manually edit the module metadata file?
Open issue: Where does this module download to? Options include:
In the first case above, this will result in an ns-3-allinone directory that is flat and possibly including non-ns3 modules mixed with ns-3 modules (e.g. pygccxml, nam-1, aodv, olsr, simulator, click, ... will all be at the same directory level).
In the second case, it has advantage of keeping all ns-3-specific modules within an ns-3 subdirectory. This may support better the developer who has multiple ns-3 branches that are supported by "common" libraries such as nsc, pybindgen, etc. that do not have to change.
The third case is somewhat analogous to the present situation where we have ns-3-allinone/ns-3-dev/src/wimin.
at the top level should recurse and cause the wimin module to be built based on the provided (jhbuild) metadata.
At this top level, there need to be some global CFLAGS in effect that cause debug/optimized/static to be consistently performed.
Let's assume that the downloaded module looked like this:
ns-3-allinone/ns-3-dev/wimin/model /examples /doc /test /helper /bindings
After the build step, it looks like this:
ns-3-allinone/ns-3-dev/wimin/model /examples /doc /test /helper /lib/libns3-wimin.so /lib/libns3-wimin-test.so /bindings/ns3wimin.so /include /bin
The install step will next put it into these places:
installprefix/bin installprefix/include/wimin installprefix/bindings/python installprefix/lib
where installprefix defaults to either "ns-3-allinone/install/debug" or "ns-3-allinone/ns-3-dev/install/debug" but can be overridden if a different prefix is configured.
Plan and Status
We are aiming to first make existing ns-3 modular, and then work on the second issue of supporting easy integration of modules maintained by third parties.
Making the existing ns-3 into a modular system is mainly a job of reorganizing the source tree and modifying waf. Presently, we are looking at the following issues:
- A new .ns3rc file that maintains a list of modules to include/exclude from the build
- Define the module granularity and fix broken module dependencies
- Work on making the python bindings more modular
- Make tests more modular; separate tests from the code, and enable per-module test executables
- Better handle examples in a modular test framework
We hope to make the above changes shortly after ns-3.10 is released in January.
For the second issue, we have been looking at GNOME jhbuild to build packages at the top-level ns-3-allinone directory. jhbuild may need to be wrapped by a specialized ns-3 build.py or download.py wrapper. However, we encountered these limitations so far:
- jhbuild has no support for scons
- waf support is limited; for instance, controlling configuration options is limited
- jhbuild has no support for hierarchical builds: although you can have metamodules, and moduleset includes, there is (AFAIK) no option to say "build only this metamodule". The options to "start at module X", or "skip module Y" are not sufficient.