Difference between revisions of "BakeIntegration"

From Nsnam
Jump to: navigation, search
m (add link to spack)
 
(24 intermediate revisions by 2 users not shown)
Line 1: Line 1:
 
{{TOC}}
 
{{TOC}}
  
This page is for describing functional requirements of the [http://www-sop.inria.fr/members/Daniel.Camara/bake/ bake tool] for [http://www.nsnam.org/wiki/index.php/Ns-3.16 ns-3.16] and for future releases.
+
This page is for describing functional requirements of the [http://www-sop.inria.fr/members/Daniel.Camara/bake/ bake tool] for [http://www.nsnam.org/wiki/index.php/Ns-3.26 ns-3.26] and for future releases.
  
= Broad functional goals =
+
= Background and Motivation =
  
1. Enable a more distributed package environment for ns-3 third-party developers.
+
Presently, ns-3 has a large backlog of unmerged contributed code, and the build of ns-3 can take very long, including modules that are not of interest to most users. We therefore want to move the project towards a more modular arrangement of 'core' modules and 'contributed' modules, and a more user-friendly way of tailoring the build of ns-3 to a specific user's needs.  We need a build orchestration tool to facilitate this.
  
1.1 Split out some of existing ns-3-dev into a smaller "core" and additional "extensions"This should reduce build time for users.
+
bake.py is a tool developed at INRIA to orchestrate the Direct Code Execution (DCE) buildIt was developed with some goals of generality (e.g. ns-3-specific constructs were avoided), but in practice is only being used these days to build DCE and some optional components of the ns-3-allinone buildbake.py is a build orchestration tool, in that it is not a build tool like 'make' or 'Waf', but instead calls to those build tools to orchestrate a build of several libraries in a correct order.  bake.py was developed because the available build orchestration tools examined at the time of development were considered to be insufficient to meet the requirements.  
  
1.2  Provide straightforward way for a third-party developer to maintain their own ns-3 extension and make it available to ns-3 users.
+
For the ns-3.27 release, we want to take the step towards making a number of contributed ns-3 modules readily available.  We do not have time  in this release cycle to create the perfect tool or transition away from bake to another build orchestration tool for ns-3.27, so we will prioritize getting something reasonably usable in place for ns-3.27, with as many modules enabled as we have time to do, and study later whether we want to make a bigger change such as moving to an externally supported build orchestration tool such as nixpkgs, or work on binary package management.
  
2. Provide a tool for developers to manage multiple builds.
+
= Executive summary =
  
3. Start to provide access to INRIA's [http://www-sop.inria.fr/members/Frederic.Urbani/ns3dceccnx/ Direct Code Execution (DCE)] environment
+
#. ns-3 will add a 'contrib/' directory in parallel with 'src/'; contrib/ will be the landing spot for contributed modules
 +
#* Patch pending in https://www.nsnam.org/bugzilla/show_bug.cgi?id=2630
 +
#. contributed modules will be organized according to the standard module layout and hosted somewhere on the web (tar archives, github, bitbucket, etc.)
 +
#. a small bake XML configuration file will tell bake how to download the new module into contrib/, and will be available to express any other library dependencies of the module.  These XML files will be stored in 'bake/contrib' directory.
 +
#* Patch pending in https://www.nsnam.org/bugzilla/show_bug.cgi?id=2631
 +
#. users will tailor their contributed module downloads using the bake tool; users will continue to tailor their ns-3 build using Waf
 +
#. modules will become more stand-alone, with ability to run Sphinx documentation and Doxygen on them independently.
 +
#* create-module.py program can migrate from src/ to contrib/ directory and extended to generate initial bake module XML
 +
#. we will start with a small set of such modules for ns-3.27 initial release and add them throughout the year.  Tentative modules for ns-3.27:
 +
#* Magister's Satellite Network Simulator 3 (SNS3)
 +
#* UAN integration with World Ocean Simulation System (WOSS)
 +
#* SiFT routing
 +
#* Routes Mobility Model
 +
#* (others TBD)
  
= ns-3.16 goals =
+
Development repos:
  
Below are proposed goals for ns-3.16.
+
1) ns-3-modular:  This contains the ns-3 contrib patch
 +
https://bitbucket.org/tomhend/ns-3-dev-modular
  
'''1. bake.py provides existing download.py and build.py with no change of functionality'''
+
2) bake-modular: This contains the bake contrib patch
 +
https://bitbucket.org/tomhend/bake-modular
  
'''1.1'''  For ns-3-dev (after merge):
+
3) sift-ns3: This is a sample contrib module, for testing purposes
 +
https://bitbucket.org/tomhend/sift-ns3
  
  hg clone http://code.nsnam.org/ns-3-allinone
+
= How bake is used now =
  cd ns-3-allinone
+
  ./bake.py configure -e ns-3-dev
+
  ./bake.py download
+
  ./bake.py build
+
  
alternative: the "install" command will "download" then "build"
+
Bake is used to construct a virtual environment for DCE. Users can request the enabling of one or more modules, and bake will check for needed system dependencies, will perform the download of libraries that are not system dependencies, can locally patch downloaded libraries if needed, can build and install libraries into a local bin and lib directory, and can build ns-3.
  
  hg clone http://code.nsnam.org/ns-3-allinone
+
The below shows how this is basically done today.
  cd ns-3-allinone
+
  ./bake.py configure -e ns-3-dev
+
  ./bake.py install
+
  
'''1.2'''  For ns-3.16:
+
  1) ./bake.py check
 +
  2) export BAKE_HOME=`pwd`
 +
  3) export PATH=$PATH:$BAKE_HOME:$BAKE_HOME/build/bin
 +
  4) export PYTHONPATH=$PYTHONPATH:$BAKE_HOME:$BAKE_HOME/build/lib
 +
  5) ./bake.py configure -e dce-linux-1.8 -e dce-quagga-1.8
 +
  6) ./bake.py show
 +
  7) ./bake.py download
 +
  8) ./bake.py build
  
  tar xjf ns-allinone-3.16.tar.bz2
+
Bake will, in turn:
  cd ns-allinone-3.16
+
  ./bake.py configure -e ns-3.16
+
  ./bake.py build
+
  
'''1.3'''  existing download.py and build.py
+
1) check that necessary bake prerequisites are filled
  
We keep existing download.py and build.py files, but they print out a backward-compatibility warning and call the equivalent bake commands; e.g.
+
2-4) set environment variables so that bake, the python modules, and the built libraries can be found by the user's shell
  
  cd ns-allinone-3.16
+
5) create a bakefile.xml after the configure step, caching the user's configuration details, and selecting the DCE Linux and quagga modules and their module dependencies
  ./build.py
+
  "Warning:  build.py is deprecated, please use './bake.py configure -e ns-3.16 && ./bake.py build' instead.
+
  
'''2.  different levels of verbosity show different amount of build information'''
+
6) show the enabled modules and their dependencies
  
  ./bake.py install
+
7) download the module sources into a 'source' directory
  
yields something sparse, such as
+
8) build, using the module's native build tool, within each module's source directory, and perform a 'make install' into a local 'build' directory containing a 'bin', 'lib', and 'include' directory for the built components
  
  >> Downloading ns-3-dev
+
Users can recurse into the source/ns-3-dce directory to start using this directly; that is, bake.py can be thought of as the step to get a custom development environment in place, then it can get out of the way and users can just use Waf at the ns-3 level.
  >> Download ns-3-dev - OK
+
  >> Building ns-3-dev
+
  >> Built ns-3-dev - OK
+
   
+
while this:
+
  
  ./bake.py install -vv
+
The user's shell keeps track of where bake and its installed components live via environment variables
 +
LD_LIBRARY_PATH, PATH, and PYTHONPATH.
  
yields the full build output (full waf configure and build information).
+
= Proposed changes for ns-3.27 =
  
'''3Managing multiple ns-3 versions and branches'''
+
The basic goal is to add a 'contrib' directory for contributed modules, and to publish metadata about available contributed modulesSome of these modules will be pure ns-3 modules; some will have additional library dependencies that bake can coordinate.  Users should be able to easily discover the list of available modules and pick the ones of interest.  Contributors who work with the release manager can get their module descriptions published, but will host their code at the location of their choice (e.g. github, bitbucket, or their departmental web server).
  
Maintainers commonly work with multiple buildsThis can be most easily accomplished using subdirectories.
+
As a concrete example, let's assume that [[RoutesMobilityModel | routes mobility model]] and [https://codereview.appspot.com/13831049/ epidemic routing] are two modules of interest to a future ns-3.27 userns-3.27 will be distributed with all existing modules as before, but not with these contributed modules. Upon downloading the base ns-3.27, the directory structure will be the same as before:
  
   hg clone http://code.nsnam.org/daniel/bake
+
   $ ls ns-3.27/src/
 +
  antenna          dsdv          netanim                test
 +
  aodv              dsr            network                topology-read
 +
  ...
  
Put bake main directory on your path and on python path
+
but there will be a sparse contrib/ directory with a build script and a create-module program:
  
   export PATH=$PATH:`pwd`/bake
+
   $ ls ns-3.27/contrib/
   export PYTHONPATH=$PYTHONPATH:`pwd`/bake
+
   create-module.py  wscript
  
'''Note:''' It is an open question whether bake/ is separate from ns-3-allinone/ repository, or part of ns-3-allinone/ repositoryContinuing with this example:
+
Let's assume that the user uses bake to download 'routes-mobility-model' and 'epidemic-routing' modules as contributed modulesAfter doing so, the following will show up under a 'contrib' directory:
  
   mkdir ns-3-dev
+
   $ ls ns-3.26/contrib/
   cd ns-3-dev
+
   create-module.py  epidemic-routing  routes-mobility-model  wscript
  bake.py configure -e ns-3-dev
+
  bake.py download
+
  bake.py build
+
  
and a little later:
+
All Waf commands will treat the 'src/' and 'contrib/' directories equally; that is, Waf will operate on the union of 'src/' and 'contrib/' when performing operations like building documentation, enabling or disabling modules, etc.  As a consequence, providers of contributed modules need to ensure that their modules behave as if they were located in the src/ directory.
  
  cd ..
+
== Module discovery ==
  mkdir ns-3.16
+
  cd ns-3.16
+
  bake.py configure -e ns-3.16
+
  bake.py download
+
  bake.py build
+
  
The resulting directory three then contains bake itself, and each repo or branch of ns-3.  Within each of these ns-3 subdirectories, there is a source/ and a build/ directory corresponding to that configuration.
+
The next topic is to describe how users will discover that contributed modules are available to be added to their build.
  
'''4.  Managing debug, optimized, and static builds'''
+
First, the project will maintain documentation on the web site about the status and availability of contributed modules, on a per-release basis, so users who go to the main documentation page for ns-3.27 would see a pointer to an HTML page with a listing of contributed modules compatible with that release (perhaps other details such as testing status, etc.).
  
See the previous item.  Users can maintain separate ns-3-dbg, ns-3-opt, ns-3-static, etc. directories. It is just a matter of calling the configuration with a -p dbg, for example. This will make use of the debug predefined configuration and will add -g and create a distinct binary directory for the ns-3 binaries and libraries.  
+
Users will also be able to discover available modules through bake.py. To enable this, bake module description will be extended with an 'ns-contrib' module type.
  
For example for ns-3.16:
+
  ./bake.py list --contrib
  bake.py configure -e ns-3.16 -p dbg
+
  contrib:  epidemic-routing
   bake.py build
+
   contrib:  routes-mobility-model
  
The new binary directory build_dbg was created on the same level of the regular build.
+
In practice, the above will be more fully populated with many contrib modules and version numbering of them.
  
'''5.  Passing configure options to bake, especially ns-3 ones such as --disable-python'''
+
== Example ==
  
Each module (nsc, netanim, etc.) has its own set of configuration options.  The way that bake presently handles this is by editing the bakeConf.xml file directly (or by setting a predefined module; see below).  It may be possible to pass configuration options down to ns-3 such as done with build.py presently.  However, this complicates bake, so it is proposed for now that any options be configured via environment variables, if possible, or by creating personal predefined configurations or, in the last case by editing attributes in bakeConf.xml.  For example:
+
To work through a concrete example, suppose the user obtains the base ns-3 with pybindgen and netanim (current defaults for allinone), the user will do:
 
+
  <attribute name="configure_arguments" value="configure --prefix=$INSTALLDIR --enable-examples
+
  --enable-tests --with-nsclick=$SRCDIR/click --with-pybindgen=$SRCDIR/pybindgen --with-
+
  openflow=$SRCDIR/openflow-ns3 --with-nsc=$SRCDIR/nsc"/>
+
  
'''6. cleaning the directory'''
+
  ./bake.py configure -e ns-allinone-3.27
 +
  ./bake.py show
  
* 'bake.py clean' will remove the build/ directory and any build artifacts; * './bake.py distclean' will perform a clean and also remove source/
+
which outputs:
  
'''7. Configuration manipulation'''
+
  ....
 +
  module: ns-allinone-3.27 (enabled)
 +
    depends on:
 +
    netanim-3.107 (optional:True)
 +
    pybindgen-0.17.0.post49+ng0e4e3bc (optional:True)
 +
    pyviz-prerequisites (optional:True)
 +
    ns-3.27 (optional:False)
 +
  ...
  
The act of configuring bake will leave a "bakefile.xml" file storing the state of the configuration.  This can be further manipulated by enabling/disabling options.
+
We can use a new command 'list' to show available modules, and sub-commands specialized to the ns-3 contributed modules.
  
For example, consider the user who has configured ns-3-dev but wants to deselect/disable netanim. This sequence of commands can accomplish that:
+
  ./bake.py list
  
   bake.py configure -e ns-3-dev
+
   /* lists all modules in the bakefile */
  bake.py fix-config -d net_anim
+
  
Some time later, the following command:
+
  ./bake.py list --contrib
 +
 +
  /* list all modules of type 'ns-contrib'; e.g.
 +
  contrib: routes-mobility-model
 +
  */
  
  bake.py configure -e ns-3-dev
+
In the future (not for ns-3.27), we may append some version information about the compatibility of the modules, with minimum and maximum ns-3 versions.  Let's assume that epidemic-routing went unmaintained and stopped working with ns-3 versions greater than 3.25, but routes-mobility-model is always maintained to stay compatible with the latest release.  The user doing a
  
will reconfigure and overwrite entirely the bakefile.xml.
+
  ./bake.py list contrib
  
It is also possible for users to store their own custom configs and modules in a .bakerc file that can be colocated with the current directory or the user's home directory.  For example, consider the user who wants a minimal ns-3 with only the wifi module enabled.  He or she could populate the following in a .bakerc file:
+
would see something like this:
  
   <predefined name="ns3-min">
+
   module name           version    minimum ns-3 version      maximum ns-3 version
    <enable name="ns-3-dev"/>
+
  ...                    ...        ...                        ...
    <disable name="nsc"/>
+
  routes-mobility-model  1.0        3.25                      unspecified
    <disable name="net_anim"/>
+
  epidemic-routing      0.9.1      3.23                      3.26
    <disable name="pybindgen"/>
+
  ...                    ...        ...                        ...
    <disable name="click"/>
+
    <disable name="openflow-ns3"/>
+
    <append module="ns-3-dev" name="configure_arguments" value=" --enable-modules=core,network,wifi"/>
+
  </predefined>
+
'''8. Show behavior'''
+
  
bake.py shows available packages, whether they are enabled, and their dependencies; e.g.:
+
To further specialize this list, the user could then list all contrib specific to ns-3.25
  
./bake.py show --enabled
+
  ./bake.py list contrib 3.25
  module: click (enabled)
+
 
    No dependencies!
+
and the above list would be filtered to include only those available for ns-3.25 (not implemented yet).
  module: openflow-ns3 (enabled)
+
    No dependencies!
+
  module: gccxml-ns3 (enabled)
+
    No dependencies!
+
  module: nsc (enabled)
+
    No dependencies!
+
  module: qt4 (enabled)
+
    No dependencies!
+
  module: pygccxml (enabled)
+
    depends on:
+
      gccxml-ns3 (optional:True)
+
  module: net_anim (enabled)
+
    depends on:
+
      qt4 (optional:True)
+
  module: pybindgen (enabled)
+
    depends on:
+
      pygccxml (optional:True)
+
  module: ns-3-dev (enabled)
+
    depends on:
+
      net_anim (optional:True)
+
      nsc (optional:True)
+
      pybindgen (optional:True)
+
      click (optional:True)
+
      openflow-ns3 (optional:True)
+
  
In the example above, the gccxml-ns3 dependency is shown for pygccxml.  It is out of scope for bake.py to manage package dependencies for some of these (e.g. showing "scons" as a dependency for nsc).  However, if the user does not have a required tool, during the course of the build, bake reports this to the user. If the developer explicitly created a system dependency, for example, it is possible to add a message so that the user can try to find/install the missing package.
+
We should also support some filtering to get specific information, such as making the following to be equivalent:
  
The user can check the missing bake dependencies by calling:
+
  ./bake.py list | grep epidemic
 +
  ./bake.py list epidemic
 +
  ./bake.py list epi
  
  bake.py check
+
To get more information about any module, we can support a new 'info' command, with support for extensions:
  > Python - Ok
+
  > Mercurial - Ok
+
  > CVS - Ok
+
  > Bazaar - Ok
+
  > Tar tool - Ok
+
  > Unzip tool - Ok
+
  > Unrar tool - Ok
+
  > GIT - is missing
+
  > Make - Ok
+
  > cMake - Ok
+
  > path tool - Ok
+
  > Autotools - Ok
+
  
Perhaps an enhancement to the above would be to show the "build" and "install" status of each.
+
  ./bake.py info epidemic
 +
  ./bake.py info epi
  
'''9.  Other behavior'''
+
would print out something based on the metadata found in the XML:
  
bake.py handles control-C signal to abort the build.
+
  module: epidemic-routing
 +
  description:  ...
 +
  homepage:  ...
 +
  version:  ...
 +
  minimum-ns-3-version:  ...
 +
  maximum-ns-3-version: ...
 +
  ...  
  
Bake trusts the standard tools to manage dirty builds and avoid building things unnecessarily. If the user wants, it is possible to call bake  uninstall to remove the built code and start a clean build.  
+
and if these information fields are missing in the XML, they will simply not print out.
  
'''10. bake.py handling of ns-3 enabled_modules'''
+
The next steps would be to actually configure the contributed modules to 'enabled' status, so that they are downloaded to the src/contrib directory when ./bake.py download is called.
  
For ns-3.16, there is no special handling of this configure options; existing ns-3 mechanisms continue to work (e.g. .ns3rc)It is still an open question of what more to do (if any) for this issue.
+
We aim to support an API like this, using the '-e' option to configure to support the adding of contributed modulesIn general, we aim to not require (but still allow) users to specify the contributed module version, as follows:
  
'''11. ns-3's scratch directory'''
+
  ./bake.py configure -c
 +
  ./bake.py show  /* should result in an empty config */
  
We propose to keep this as is for now. Over time, it may be that we need to provide something similar to scratch/ at the bake level, although it is possible that some of the magic of waf will be replaced by the more explicit use of environment variables along with autotools/Makefile skeletons as well.
+
  ./bake.py configure -a routes-mobility-model
 +
  > Error: Module "routes-mobility-model" has unmet dependency: ns-3.27
  
waf may need to learn of these new (default) install locations.
+
  ./bake.py configure -e ns-3.27  /* Now I have a base of ns-3.27 module, and I can add to it */
  
'''12DCE'''
+
  ./bake.py configure -e pybindgen /* add python bindings support */
  
bake.py provides access to ns-3 DCE with at least one well-documented example (iperf) of how to use real application code within an ns-3 program.  We focus on user-mode applications for ns-3.16 and kernel-mode for the next release.  DCE has its own documentation system; we will look into whether its patch for ns-3 can be merged to ns-3-dev or whether it still needs to sit outside of ns-3-dev.
+
  ./bake.py configure -e routes-mobility-model    /* add routes-mobility-model to build */
  
'''13external module'''
+
  ./bake.py configure -e epidemic-prime
 +
  Error: epidemic-prime not found for ns-3.27
  
'''(defer to ns-3.17)''' bake.py allows at least one "external" ns-3-based module to be downloaded and built with ns-3.  For example, assume there is an external routing protocol module (ns-3-bgp); we may need to make one of these up for the sake of providing an example.
+
  ./bake.py configure -d routes-mobility-model /* remove routes-mobility-model from configuration */
  
For ns-3-bgp to build, it needs to link to ns-3 libraries.  So, there needs to be some install step in bake to install to a well-known place in the allinone directory, or in the system.
+
  ./bake.py show /* should show now that ns-3.27 and pybindgen are configured */
  
For a user program in ns-3.16/scratch to use ns-3-bgp, it needs to be able to find the library and headers.  So, the install step on ns-3-bgp should leave the installation in a well-known place by ns-3's waf.
+
Enabling of modules can also be done on a single statement:
  
So, the goal here would be:
+
  ./bake.py configure -e ns-3.25 -e pybindgen -e routes-mobility-model -e epidemic-routing
  
  ./bake.py configure -e ns-3-bgp
+
= Open issues =
  ./bake.py download
+
  ./bake.py build      # builds ns-3, then builds ns-3-bgp
+
  
Then, user program in ns-3.16/scratch/ can include the module without specifying the full path to ns-3-bgp; i.e., waf may need to become smarter to find these headers and libraries.
+
What modules go into the ns-allinone package in the future, if not the universe of compatible libraries and extensions?    I would lean towards requiring that contrib modules be fetched online and that we do not bundle the universe but define a minimum set, like we do today.
  
= Beyond ns-3.16 =
+
How this relates to binary package distribution/preparation is TBD.
  
To be determined.
+
Whether we try to replace bake with something like [https://nixos.org/nixpkgs/manual/ nixpkgs] or [https://spack.readthedocs.io/en/latest/ spack] TBD.

Latest revision as of 19:09, 23 February 2017

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

This page is for describing functional requirements of the bake tool for ns-3.26 and for future releases.

Background and Motivation

Presently, ns-3 has a large backlog of unmerged contributed code, and the build of ns-3 can take very long, including modules that are not of interest to most users. We therefore want to move the project towards a more modular arrangement of 'core' modules and 'contributed' modules, and a more user-friendly way of tailoring the build of ns-3 to a specific user's needs. We need a build orchestration tool to facilitate this.

bake.py is a tool developed at INRIA to orchestrate the Direct Code Execution (DCE) build. It was developed with some goals of generality (e.g. ns-3-specific constructs were avoided), but in practice is only being used these days to build DCE and some optional components of the ns-3-allinone build. bake.py is a build orchestration tool, in that it is not a build tool like 'make' or 'Waf', but instead calls to those build tools to orchestrate a build of several libraries in a correct order. bake.py was developed because the available build orchestration tools examined at the time of development were considered to be insufficient to meet the requirements.

For the ns-3.27 release, we want to take the step towards making a number of contributed ns-3 modules readily available. We do not have time in this release cycle to create the perfect tool or transition away from bake to another build orchestration tool for ns-3.27, so we will prioritize getting something reasonably usable in place for ns-3.27, with as many modules enabled as we have time to do, and study later whether we want to make a bigger change such as moving to an externally supported build orchestration tool such as nixpkgs, or work on binary package management.

Executive summary

  1. . ns-3 will add a 'contrib/' directory in parallel with 'src/'; contrib/ will be the landing spot for contributed modules
  2. . contributed modules will be organized according to the standard module layout and hosted somewhere on the web (tar archives, github, bitbucket, etc.)
  3. . a small bake XML configuration file will tell bake how to download the new module into contrib/, and will be available to express any other library dependencies of the module. These XML files will be stored in 'bake/contrib' directory.
  4. . users will tailor their contributed module downloads using the bake tool; users will continue to tailor their ns-3 build using Waf
  5. . modules will become more stand-alone, with ability to run Sphinx documentation and Doxygen on them independently.
    • create-module.py program can migrate from src/ to contrib/ directory and extended to generate initial bake module XML
  6. . we will start with a small set of such modules for ns-3.27 initial release and add them throughout the year. Tentative modules for ns-3.27:
    • Magister's Satellite Network Simulator 3 (SNS3)
    • UAN integration with World Ocean Simulation System (WOSS)
    • SiFT routing
    • Routes Mobility Model
    • (others TBD)

Development repos:

1) ns-3-modular: This contains the ns-3 contrib patch https://bitbucket.org/tomhend/ns-3-dev-modular

2) bake-modular: This contains the bake contrib patch https://bitbucket.org/tomhend/bake-modular

3) sift-ns3: This is a sample contrib module, for testing purposes https://bitbucket.org/tomhend/sift-ns3

How bake is used now

Bake is used to construct a virtual environment for DCE. Users can request the enabling of one or more modules, and bake will check for needed system dependencies, will perform the download of libraries that are not system dependencies, can locally patch downloaded libraries if needed, can build and install libraries into a local bin and lib directory, and can build ns-3.

The below shows how this is basically done today.

 1) ./bake.py check
 2) export BAKE_HOME=`pwd`
 3) export PATH=$PATH:$BAKE_HOME:$BAKE_HOME/build/bin
 4) export PYTHONPATH=$PYTHONPATH:$BAKE_HOME:$BAKE_HOME/build/lib
 5) ./bake.py configure -e dce-linux-1.8 -e dce-quagga-1.8
 6) ./bake.py show
 7) ./bake.py download
 8) ./bake.py build

Bake will, in turn:

1) check that necessary bake prerequisites are filled

2-4) set environment variables so that bake, the python modules, and the built libraries can be found by the user's shell

5) create a bakefile.xml after the configure step, caching the user's configuration details, and selecting the DCE Linux and quagga modules and their module dependencies

6) show the enabled modules and their dependencies

7) download the module sources into a 'source' directory

8) build, using the module's native build tool, within each module's source directory, and perform a 'make install' into a local 'build' directory containing a 'bin', 'lib', and 'include' directory for the built components

Users can recurse into the source/ns-3-dce directory to start using this directly; that is, bake.py can be thought of as the step to get a custom development environment in place, then it can get out of the way and users can just use Waf at the ns-3 level.

The user's shell keeps track of where bake and its installed components live via environment variables LD_LIBRARY_PATH, PATH, and PYTHONPATH.

Proposed changes for ns-3.27

The basic goal is to add a 'contrib' directory for contributed modules, and to publish metadata about available contributed modules. Some of these modules will be pure ns-3 modules; some will have additional library dependencies that bake can coordinate. Users should be able to easily discover the list of available modules and pick the ones of interest. Contributors who work with the release manager can get their module descriptions published, but will host their code at the location of their choice (e.g. github, bitbucket, or their departmental web server).

As a concrete example, let's assume that routes mobility model and epidemic routing are two modules of interest to a future ns-3.27 user. ns-3.27 will be distributed with all existing modules as before, but not with these contributed modules. Upon downloading the base ns-3.27, the directory structure will be the same as before:

 $ ls ns-3.27/src/
 antenna           dsdv           netanim                test
 aodv              dsr            network                topology-read
 ...

but there will be a sparse contrib/ directory with a build script and a create-module program:

 $ ls ns-3.27/contrib/
 create-module.py   wscript

Let's assume that the user uses bake to download 'routes-mobility-model' and 'epidemic-routing' modules as contributed modules. After doing so, the following will show up under a 'contrib' directory:

 $ ls ns-3.26/contrib/
 create-module.py  epidemic-routing   routes-mobility-model  wscript

All Waf commands will treat the 'src/' and 'contrib/' directories equally; that is, Waf will operate on the union of 'src/' and 'contrib/' when performing operations like building documentation, enabling or disabling modules, etc. As a consequence, providers of contributed modules need to ensure that their modules behave as if they were located in the src/ directory.

Module discovery

The next topic is to describe how users will discover that contributed modules are available to be added to their build.

First, the project will maintain documentation on the web site about the status and availability of contributed modules, on a per-release basis, so users who go to the main documentation page for ns-3.27 would see a pointer to an HTML page with a listing of contributed modules compatible with that release (perhaps other details such as testing status, etc.).

Users will also be able to discover available modules through bake.py. To enable this, bake module description will be extended with an 'ns-contrib' module type.

 ./bake.py list --contrib
 contrib:  epidemic-routing
 contrib:  routes-mobility-model

In practice, the above will be more fully populated with many contrib modules and version numbering of them.

Example

To work through a concrete example, suppose the user obtains the base ns-3 with pybindgen and netanim (current defaults for allinone), the user will do:

 ./bake.py configure -e ns-allinone-3.27
 ./bake.py show

which outputs:

 ....
 module: ns-allinone-3.27 (enabled)
   depends on:
    netanim-3.107 (optional:True)
    pybindgen-0.17.0.post49+ng0e4e3bc (optional:True)
    pyviz-prerequisites (optional:True)
    ns-3.27 (optional:False)
 ...

We can use a new command 'list' to show available modules, and sub-commands specialized to the ns-3 contributed modules.

 ./bake.py list
 /* lists all modules in the bakefile */
  ./bake.py list --contrib

 /* list all modules of type 'ns-contrib'; e.g.
 contrib:  routes-mobility-model
 */

In the future (not for ns-3.27), we may append some version information about the compatibility of the modules, with minimum and maximum ns-3 versions. Let's assume that epidemic-routing went unmaintained and stopped working with ns-3 versions greater than 3.25, but routes-mobility-model is always maintained to stay compatible with the latest release. The user doing a

 ./bake.py list contrib

would see something like this:

 module name            version    minimum ns-3 version       maximum ns-3 version
 ...                    ...        ...                        ...
 routes-mobility-model  1.0        3.25                       unspecified
 epidemic-routing       0.9.1      3.23                       3.26
 ...                    ...        ...                        ...

To further specialize this list, the user could then list all contrib specific to ns-3.25

 ./bake.py list contrib 3.25

and the above list would be filtered to include only those available for ns-3.25 (not implemented yet).

We should also support some filtering to get specific information, such as making the following to be equivalent:

 ./bake.py list | grep epidemic
 ./bake.py list epidemic
 ./bake.py list epi

To get more information about any module, we can support a new 'info' command, with support for extensions:

 ./bake.py info epidemic
 ./bake.py info epi

would print out something based on the metadata found in the XML:

 module: epidemic-routing
 description:  ...
 homepage:  ...
 version:  ... 
 minimum-ns-3-version:  ...
 maximum-ns-3-version: ...
 ... 

and if these information fields are missing in the XML, they will simply not print out.

The next steps would be to actually configure the contributed modules to 'enabled' status, so that they are downloaded to the src/contrib directory when ./bake.py download is called.

We aim to support an API like this, using the '-e' option to configure to support the adding of contributed modules. In general, we aim to not require (but still allow) users to specify the contributed module version, as follows:

 ./bake.py configure -c
 ./bake.py show  /* should result in an empty config */
 ./bake.py configure -a routes-mobility-model
 > Error: Module "routes-mobility-model" has unmet dependency: ns-3.27 
 ./bake.py configure -e ns-3.27   /* Now I have a base of ns-3.27 module, and I can add to it */
 ./bake.py configure -e pybindgen  /* add python bindings support */
 ./bake.py configure -e routes-mobility-model    /* add routes-mobility-model to build */
 ./bake.py configure -e epidemic-prime
 Error:  epidemic-prime not found for ns-3.27
 ./bake.py configure -d routes-mobility-model /* remove routes-mobility-model from configuration */
 ./bake.py show /* should show now that ns-3.27 and pybindgen are configured */

Enabling of modules can also be done on a single statement:

 ./bake.py configure -e ns-3.25 -e pybindgen -e routes-mobility-model -e epidemic-routing

Open issues

What modules go into the ns-allinone package in the future, if not the universe of compatible libraries and extensions? I would lean towards requiring that contrib modules be fetched online and that we do not bundle the universe but define a minimum set, like we do today.

How this relates to binary package distribution/preparation is TBD.

Whether we try to replace bake with something like nixpkgs or spack TBD.