BakeIntegration

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

This page is for describing functional requirements of the bake tool for ns-3.26 and for future releases.

Background and Motivation

Presently, ns-3 has a large backlog of unmerged contributed code, and the build of ns-3 can take very long, including modules that are not of interest to most users. We therefore want to move the project towards a more modular arrangement of 'core' modules and 'contributed' modules, and a more user-friendly way of tailoring the build of ns-3 to a specific user's needs. We need a build orchestration tool to facilitate this.

bake.py is a tool developed at INRIA to orchestrate the Direct Code Execution (DCE) build. It was developed with some goals of generality (e.g. ns-3-specific constructs were avoided), but in practice is only being used these days to build DCE and some optional components of the ns-3-allinone build. bake.py is a build orchestration tool, in that it is not a build tool like 'make' or 'Waf', but instead calls to those build tools to orchestrate a build of several libraries in a correct order. bake.py was developed because the available build orchestration tools examined at the time of development were considered to be insufficient to meet the requirements.

For the ns-3.27 release, we want to take the step towards making a number of contributed ns-3 modules readily available. We do not have time in this release cycle to create the perfect tool or transition away from bake to another build orchestration tool for ns-3.27, so we will prioritize getting something reasonably usable in place for ns-3.27, with as many modules enabled as we have time to do, and study later whether we want to make a bigger change such as moving to an externally supported build orchestration tool such as nixpkgs, or work on binary package management.

Executive summary

1. . ns-3 will add a 'contrib/' directory in parallel with 'src/'; contrib/ will be the landing spot for contributed modules
   • Patch pending in https://www.nsnam.org/bugzilla/show_bug.cgi?id=2630
2. . contributed modules will be organized according to the standard module layout and hosted somewhere on the web (tar archives, github, bitbucket, etc.)
3. . a small bake XML configuration file will tell bake how to download the new module into contrib/, and will be available to express any other library dependencies of the module. These XML files will be stored in 'bake/contrib' directory.
   • Patch pending in https://www.nsnam.org/bugzilla/show_bug.cgi?id=2631
4. . users will tailor their contributed module downloads using the bake tool; users will continue to tailor their ns-3 build using Waf
5. . modules will become more stand-alone, with ability to run Sphinx documentation and Doxygen on them independently.
   • create-module.py program can migrate from src/ to contrib/ directory and extended to generate initial bake module XML
6. . we will start with a small set of such modules for ns-3.27 initial release and add them throughout the year. Tentative modules for ns-3.27:
   • Magister's Satellite Network Simulator 3 (SNS3)
   • UAN integration with World Ocean Simulation System (WOSS)
   • SiFT routing
   • Routes Mobility Model
   • (others TBD)

Development repos:

1) ns-3-modular: This contains the ns-3 contrib patch https://bitbucket.org/tomhend/ns-3-dev-modular

2) bake-modular: This contains the bake contrib patch https://bitbucket.org/tomhend/bake-modular

3) sift-ns3: This is a sample contrib module, for testing purposes https://bitbucket.org/tomhend/sift-ns3

How bake is used now

Bake is used to construct a virtual environment for DCE. Users can request the enabling of one or more modules, and bake will check for needed system dependencies, will perform the download of libraries that are not system dependencies, can locally patch downloaded libraries if needed, can build and install libraries into a local bin and lib directory, and can build ns-3.

The below shows how this is basically done today.

 1) ./bake.py check
 2) export BAKE_HOME=`pwd`
 3) export PATH=$PATH:$BAKE_HOME:$BAKE_HOME/build/bin
 4) export PYTHONPATH=$PYTHONPATH:$BAKE_HOME:$BAKE_HOME/build/lib
 5) ./bake.py configure -e dce-linux-1.8 -e dce-quagga-1.8
 6) ./bake.py show
 7) ./bake.py download
 8) ./bake.py build

Bake will, in turn:

1) check that necessary bake prerequisites are filled

2-4) set environment variables so that bake, the python modules, and the built libraries can be found by the user's shell

5) create a bakefile.xml after the configure step, caching the user's configuration details, and selecting the DCE Linux and quagga modules and their module dependencies

6) show the enabled modules and their dependencies

7) download the module sources into a 'source' directory

8) build, using the module's native build tool, within each module's source directory, and perform a 'make install' into a local 'build' directory containing a 'bin', 'lib', and 'include' directory for the built components

Users can recurse into the source/ns-3-dce directory to start using this directly; that is, bake.py can be thought of as the step to get a custom development environment in place, then it can get out of the way and users can just use Waf at the ns-3 level.

The user's shell keeps track of where bake and its installed components live via environment variables LD_LIBRARY_PATH, PATH, and PYTHONPATH.

Proposed changes for ns-3.27

The basic goal is to add a 'contrib' directory for contributed modules, and to publish metadata about available contributed modules. Some of these modules will be pure ns-3 modules; some will have additional library dependencies that bake can coordinate. Users should be able to easily discover the list of available modules and pick the ones of interest. Contributors who work with the release manager can get their module descriptions published, but will host their code at the location of their choice (e.g. github, bitbucket, or their departmental web server).

As a concrete example, let's assume that routes mobility model and epidemic routing are two modules of interest to a future ns-3.27 user. ns-3.27 will be distributed with all existing modules as before, but not with these contributed modules. Upon downloading the base ns-3.27, the directory structure will be the same as before:

 $ ls ns-3.27/src/
 antenna           dsdv           netanim                test
 aodv              dsr            network                topology-read
 ...

but there will be a sparse contrib/ directory with a build script and a create-module program:

 $ ls ns-3.27/contrib/
 create-module.py   wscript

Let's assume that the user uses bake to download 'routes-mobility-model' and 'epidemic-routing' modules as contributed modules. After doing so, the following will show up under a 'contrib' directory:

 $ ls ns-3.26/contrib/
 create-module.py  epidemic-routing   routes-mobility-model  wscript

All Waf commands will treat the 'src/' and 'contrib/' directories equally; that is, Waf will operate on the union of 'src/' and 'contrib/' when performing operations like building documentation, enabling or disabling modules, etc. As a consequence, providers of contributed modules need to ensure that their modules behave as if they were located in the src/ directory.

Module discovery

The next topic is to describe how users will discover that contributed modules are available to be added to their build.

First, the project will maintain documentation on the web site about the status and availability of contributed modules, on a per-release basis, so users who go to the main documentation page for ns-3.27 would see a pointer to an HTML page with a listing of contributed modules compatible with that release (perhaps other details such as testing status, etc.).

Users will also be able to discover available modules through bake.py. To enable this, bake module description will be extended with an 'ns-contrib' module type.

 ./bake.py list --contrib
 contrib:  epidemic-routing
 contrib:  routes-mobility-model

In practice, the above will be more fully populated with many contrib modules and version numbering of them.

Example

To work through a concrete example, suppose the user obtains the base ns-3 with pybindgen and netanim (current defaults for allinone), the user will do:

 ./bake.py configure -e ns-allinone-3.27
 ./bake.py show

which outputs:

 ....
 module: ns-allinone-3.27 (enabled)
   depends on:
    netanim-3.107 (optional:True)
    pybindgen-0.17.0.post49+ng0e4e3bc (optional:True)
    pyviz-prerequisites (optional:True)
    ns-3.27 (optional:False)
 ...

We can use a new command 'list' to show available modules, and sub-commands specialized to the ns-3 contributed modules.

 ./bake.py list

 /* lists all modules in the bakefile */

  ./bake.py list --contrib

 /* list all modules of type 'ns-contrib'; e.g.
 contrib:  routes-mobility-model
 */

In the future (not for ns-3.27), we may append some version information about the compatibility of the modules, with minimum and maximum ns-3 versions. Let's assume that epidemic-routing went unmaintained and stopped working with ns-3 versions greater than 3.25, but routes-mobility-model is always maintained to stay compatible with the latest release. The user doing a

 ./bake.py list contrib

would see something like this:

 module name            version    minimum ns-3 version       maximum ns-3 version
 ...                    ...        ...                        ...
 routes-mobility-model  1.0        3.25                       unspecified
 epidemic-routing       0.9.1      3.23                       3.26
 ...                    ...        ...                        ...

To further specialize this list, the user could then list all contrib specific to ns-3.25

 ./bake.py list contrib 3.25

and the above list would be filtered to include only those available for ns-3.25 (not implemented yet).

We should also support some filtering to get specific information, such as making the following to be equivalent:

 ./bake.py list | grep epidemic
 ./bake.py list epidemic
 ./bake.py list epi

To get more information about any module, we can support a new 'info' command, with support for extensions:

 ./bake.py info epidemic
 ./bake.py info epi

would print out something based on the metadata found in the XML:

 module: epidemic-routing
 description:  ...
 homepage:  ...
 version:  ... 
 minimum-ns-3-version:  ...
 maximum-ns-3-version: ...
 ... 

and if these information fields are missing in the XML, they will simply not print out.

The next steps would be to actually configure the contributed modules to 'enabled' status, so that they are downloaded to the src/contrib directory when ./bake.py download is called.

We aim to support an API like this, using the '-e' option to configure to support the adding of contributed modules. In general, we aim to not require (but still allow) users to specify the contributed module version, as follows:

 ./bake.py configure -c
 ./bake.py show  /* should result in an empty config */

 ./bake.py configure -a routes-mobility-model
 > Error: Module "routes-mobility-model" has unmet dependency: ns-3.27 

 ./bake.py configure -e ns-3.27   /* Now I have a base of ns-3.27 module, and I can add to it */

 ./bake.py configure -e pybindgen  /* add python bindings support */

 ./bake.py configure -e routes-mobility-model    /* add routes-mobility-model to build */

 ./bake.py configure -e epidemic-prime
 Error:  epidemic-prime not found for ns-3.27

 ./bake.py configure -d routes-mobility-model /* remove routes-mobility-model from configuration */

 ./bake.py show /* should show now that ns-3.27 and pybindgen are configured */

Enabling of modules can also be done on a single statement:

 ./bake.py configure -e ns-3.25 -e pybindgen -e routes-mobility-model -e epidemic-routing

Open issues

What modules go into the ns-allinone package in the future, if not the universe of compatible libraries and extensions? I would lean towards requiring that contrib modules be fetched online and that we do not bundle the universe but define a minimum set, like we do today.

How this relates to binary package distribution/preparation is TBD.

Whether we try to replace bake with something like nixpkgs or spack TBD.