Difference between revisions of "App Store Technical Requirements"

From Nsnam
Jump to: navigation, search
(Module dependencies)
(update app store requirements)
 
(15 intermediate revisions by 3 users not shown)
Line 1: Line 1:
 
{{TOC}}
 
{{TOC}}
  
= Goals =
+
= Background =
  
The long-term goal is to move ns-3 to separate modules, for build and  
+
The goal is to support modularization of the ns-3 code base and federation of development activitiesTo date, the main path to distributing extensions to ns-3 has been to request a code review for merge to the mainline tree.  The future goal is to shrink the ns-3 core, and to better facilitate modules that are independently developed and maintained.  As a concrete example, the [https://www.r-project.org/ R statistical framework] does not integrate every extension to its main release; instead, it allows users to [https://cran.cnr.berkeley.edu/web/packages/available_packages_by_date.html fetch the packages of interest] and add to their installationCommunities-of-interest have even arisen to allow researchers to organize and curate sets of R extensions (e.g. [https://www.bioconductor.org/ bioinformatic packages]).  We envision the same for ns-3.
maintenance reasonsFor build reasons, since ns-3 is becoming large,  
+
we would like to allow users to enable/disable subsets of the available
+
model libraryFor maintenance reasons, it is important that we move to  
+
a development model where modules can evolve on different timescales and  
+
be maintained by different organizations.
+
  
An analogy is the GNOME desktop, which is composed of a number of
+
In the ns-3.27 release, the bake build orchestration tool was extended to allow third party contributed code modules to be added to ns-3, and to be downloaded, built, and installed with the main ns-3 releaseThis allows modules to be hosted elsewhere and integrated with the ns-3-allinone framework. Consider this to be the back-end support for modularization.  We have not gone further since that time.
individual libraries that evolve on their own timescalesA build
+
framework called [http://live.gnome.org/Jhbuild jhbuild] exists for
+
building and managing the dependencies between these disparate projects.
+
  
Once we have a modular build, and an ability to separately download and
+
For example, the sift routing code contribution has been added as a modulePlease review how users are expected to add sift to their ns-3.27 release [[BakeIntegration | on this wiki page]].
install third-party modules, we will need to distinguish the maintenance
+
status or certification of modulesThe ns-3 project will maintain a
+
set of core ns-3 modules including those essential for all ns-3
+
simulations, and will maintain a master build file containing metadata
+
to contributed modules; this will allow users to fetch and build what
+
they need.  Eventually, modules will have a maintenance state associated
+
with them describing aspects such as who is the maintainer, whether it
+
is actively maintained, whether it contains documentation or tests,
+
whether it passed an ns-3 project code review, whether it is currently
+
passing the tests, etc. The current status of all known ns-3 modules
+
will be maintained in a database and be browsable on the project web site.
+
  
[[image:Maintenance-status-example.PNG]]
+
There are two main components:
  
'''Figure caption''':  ''Mock-up of future model status page (models and colors selected are just for example purposes)''
+
1) '''front end''' The ns-3 app store (https://apps.nsnam.org) is largely operational and will be put into actual use for ns-3.29.  The app store can be used to directly download contributed modules, and to post information about the module.
  
The basic idea of the ns-3 app store would be to store on a server a set of user-submitted metadata which describes various source code packages. Typical metadata would include:
+
2) '''back end''' The back end will be the [https://www.nsnam.org/docs/bake/tutorial/html/bake-over.html bake build orchestration tool].
* unique name
+
* version number
+
* last known good ns-3 version (tested against)
+
* description
+
* download url
+
* untar/unzip command
+
* configure command
+
* build command
+
* system prerequisites (if user needs to apt-get install other libraries)
+
* list of ns-3 package dependencies (other ns-3 packages which this package depends upon)
+
  
= Requirements =
+
= Workflows =
  
== Build and configuration ==
+
We envision three types of users:
  
* Enable optimized, debug, and static builds
+
#. '''End user:'''  The normal user will browse to the app store to learn about available apps, learn about resources (user forums, documentation) about each app, and possibly leave feedback on a review board. The normal user will obtain the necessary information to fetch and install each app from this page.
* Separate tests from models
+
#. '''Developer:''' Developers will not be able to create new apps on the app store, but will be able to edit the app page that he or she owns. To make an app available on the store, it has to go through an ns-3 review process.
* Doxygen support
+
#. '''Maintainer:''' ns-3 maintainers will create a new page for an accepted app, and create a developer account for the author. Then the developer will be able to edit the page to suit their app.
* Test.py framework supports the modularization
+
* Python bindings support the modularization
+
* integrate lcov code coverage tests
+
* integrate with buildbots
+
 
+
== API and work flow ==
+
 
+
Assume that our download script is called download.py and our
+
build script is called build.py.
+
 
+
  wget http://www.nsnam.org/releases/ns-allinone-3.x.tar.bz2
+
  bunzip2 ns-allinone-3.x.tar.bz2 && tar xvf ns-allinone-3.x.tar.bz2
+
  cd ns-allinone-3.x
+
 
+
In this directory, users will find the following directory layout:
+
 
+
  build.py download.py constants.py? VERSION LICENSE README
+
 
+
Download essential modules:
+
 
+
  ./download.py
+
 
+
This will leave a layout such as follows:
+
 
+
  download.py build.py pygccxml pybindgen ns-3.10 gcc-xml
+
 
+
 
+
For typical users, the next step is as follows:
+
 
+
  ./build.py
+
 
+
The above will take the following steps:
+
* build all typical prerequisites such as pybindgen
+
* cd ns3-10
+
* ./waf configure && ./waf && ./waf install
+
 
+
The above will install headers into build/debug/ns3/ and libraries for each of the enabled modules into build/debug/lib. Each enabled module will be placed in a separate library that has a name of the form libns3-simulator.so for the simulator module.
+
 
+
'''Open issue:''' How to express and honor platform limitations, such as
+
only trying to download/build NSC on platforms supporting it.
+
 
+
Once the build process is done, there will be a bunch of libraries in a
+
common build directory, and all applicable headers. Specifically,
+
we envision for module simulator, there may be a libns3-simulator.so, libns3-simulator-test.so,
+
a python .so, and possibly others. 
+
 
+
=== Python ===
+
 
+
We have a few choices for supporting Python. First, note that this type
+
of system provides an opportunity for the python build tools to be added
+
as packages to the overall build system, so that users can more easily
+
build their bindings.  We can try to build lots of small python modules,
+
or run a scan at the very end of the ns-3 build process to build a customized
+
python ns3 module, such as:
+
 
+
* python-scan.py
+
* ...
+
* pybindgen
+
 
+
which operates on the headers in the build/debug/ns3 directory and creates
+
a python module .so library that matches the ns-3 configured components.
+
 
    
 
    
'''Open issue:''' What is the eventual python API?  Should each module be
+
= Requirements =
imported separately such as import node as ns.node, or should we try to
+
The following requirements were used to develop the app store (https://apps.nsnam.org).
go for a single ns3 python module?  Or, we could continue to maintain bindings.  Another consideration is that constantly generating bindings will slow down the builds.
+
 
+
=== Tests ===
+
 
+
Tests are run by running ./test.py, which knows how to find the available
+
test libraries and run the tests.
+
 
+
Presently, test.py hardcodes the examples or samples; it needs to become
+
smarter to learn what examples are around and need testing.
+
 
+
=== Doxygen ===
+
 
+
Doxygen can be run such as "build.py doxygen" on the build/debug/ns3
+
directory.  I would guess that we don't try to modularize this but instead
+
to run on the build/debug/ns3/ directory once all headers have been copied in.
+
 
+
=== Build flags ===
+
 
+
To configure different options such as  -g, -fprofile-arcs, e.g., use the
+
CFLAGS environment variable.
+
 
+
=== Running programs ===
+
 
+
To run programs:
+
 
+
  ./build.py shell
+
  build/debug/examples/csma/csma-bridge
+
or
+
  python build/debug/examples/csma/csma-bridge.py
+
 
+
=== Other examples ===
+
 
+
Build all known ns-3 modules:
+
 
+
  ./build.py all
+
 
+
In the above case, suppose that the program could not download and fetch
+
a module.  Here, it can interactively prompt the user "Module foo not
+
found; do you wish to continue [Y/n]?".
+
 
+
Build the core of ns-3, plus device models WiFi and CSMA:
+
 
+
  ./build ns3-core wifi csma
+
 
+
ns3-core is a meta-module containing simulator common node mobility etc. 
+
'''Open issue:''' what is the right granularity for this module?
+
 
+
=== Example third-party ns-3 module ===
+
 
+
Suppose a research group at Example Univ. publishes a new WiMin device
+
module.  It gets a new unique module name from ns-3 project, such as
+
wimin.  It also contributes metadata to the master ns-3 build script.
+
 
+
==== Downloading ====
+
 
+
The "download.py" system should have enough metadata to fetch it, whether
+
it is a tarball release or some other kind of release. 
+
 
+
'''Open issue:''' How does user toggle the download behavior of a module?
+
Is there a sticky "./download.py --disable-module=wimin" option?  Or does
+
user manually edit the module metadata file?
+
 
+
'''Open issue:''' Where does this module download to?  Options include:
+
# ns-3-allinone/wimin
+
# ns-3-allinone/ns-3-dev/wimin
+
# ns-3-allinone/ns-3-dev/modules/wimin
+
 
+
In the first case above, this will result in an ns-3-allinone directory that
+
is flat and possibly including non-ns3 modules mixed with ns-3 modules
+
(e.g. pygccxml, nam-1, aodv, olsr, simulator, click, ... will all be at
+
the same directory level).
+
 
+
In the second case, it has advantage of keeping all ns-3-specific modules
+
within an ns-3 subdirectory.  This may support better the developer who
+
has multiple ns-3 branches that are supported by "common" libraries such
+
as nsc, pybindgen, etc. that do not have to change.
+
 
+
The third case is somewhat analogous to the present situation where we
+
have ns-3-allinone/ns-3-dev/src/wimin.
+
 
+
==== Building ====
+
 
+
  ./build.py
+
 
+
at the top level should recurse and cause the wimin module to be built
+
based on the provided (jhbuild) metadata.
+
 
+
At this top level, there need to be some global CFLAGS in effect that
+
cause debug/optimized/static to be consistently performed.
+
 
+
Let's assume that the downloaded module looked like this:
+
 
+
  ns-3-allinone/ns-3-dev/wimin/model
+
                              /examples
+
                              /doc
+
                              /test
+
                              /helper
+
                              /bindings
+
 
+
After the build step, it looks like this:
+
 
+
  ns-3-allinone/ns-3-dev/wimin/model
+
                              /examples
+
                              /doc
+
                              /test
+
                              /helper
+
                              /lib/libns3-wimin.so
+
                              /lib/libns3-wimin-test.so
+
                              /bindings/ns3wimin.so
+
                              /include
+
                              /bin
+
 
+
==== Installing ====
+
 
+
The install step will next put it into these places:
+
 
+
  installprefix/bin
+
  installprefix/include/wimin
+
  installprefix/bindings/python
+
  installprefix/lib
+
 
+
where installprefix defaults to either "ns-3-allinone/install/debug"
+
or "ns-3-allinone/ns-3-dev/install/debug" but can be overridden if
+
a different prefix is configured.
+
 
+
= Plan and Status =
+
 
+
== Plan ==
+
  
We are aiming to first make the existing ns-3 modular, and then work
+
#. The front page should be clean, consisting of categories, newest releases, and popular downloads.
on the second issue of supporting easy integration of modules
+
#. Each "app" will be an ns-3 module, with a description page including:  Description, screenshots (optional),  download tab, resources tab, etc.
maintained by third parties.
+
#. Clear documentation must be posted on how to use the app store, and how to submit a new app to the app store
 +
#. The "Download" tab will fetch a source archive, and the user will download to a location of their choice.  The user is responsible for moving or copying the package to their ns-3 development directory, and then the ns-3 build tools (invoked at command line by users in the normal way) will do the rest.
 +
#. XML for the bake configuration will also be maintained for each module, allowing module contributors to post new releases of their module.
  
This plan will be carried out in two stages:
+
= Enhancements =
* Stage 1: (for review/merge now) make existing ns-3 modular
+
** users can explicitly disable unneeded modules from their build
+
** tests are decoupled from the main libraries; changes to test programs do not cause all examples to be relinked
+
* Stage 2: (post ns-3.11) enable the "app store" concept
+
** allow jhbuild or similar scripts to orchestrate a larger build, including managing third-party dependencies (e.g. click or openflow)
+
** each module gets an independent version number, and metadata coordinates version dependencies between modules
+
** allow users to use their own build system if they want
+
  
== Stage 1 Status ==
+
Once the basic capability is deployed, enhancements are possible.  Some of these extensions may involve modifying the app store installation, but some will involve modifying ns-3 build scripts, documentation, and other scripts (e.g. Jenkins scripts).
  
Stage 1 is proposed for ns-3.11 release.  The primary goal is to be able to enable/disable modules and allow users to tailor the build to include the components that they need. 
+
== Integration with bake ==
  
Making the existing ns-3 into a modular system is mainly a job of reorganizing the source tree and modifying waf and the python bindings generation code.  
+
Bake uses XML to describe modules.  It would be nice to have a standardized XML schema that describes an ns-3 module, and make it usable from both the app store and from bake.  For instance, there could be an "authors" field in the XML description that can be displayed by bake at the command line and can also be parsed and shown as the Authors in the app store description.
  
There is a prototype repository at http://code.nsnam.org/watrous/ns-3-dev-pending that basically contains what is proposed for ns-3.11.  The items that have been completed to date are:
+
== User feedback and comments ==
# Resolve the main circular dependencies in the core modules of ns-3, and reorganize the following modules:  simulator, core, common, node, internet-stack into a new set of modules:  core, network, internet, propagation, spectrum, along the lines of what was discussed on the list
+
# A new file .ns3rc that allows users to specify which items are in the build or out of the build.
+
# separate libraries for each module:  each module builds one "model" library and one "test" library.  Only the test-runner links the test code.
+
# The example programs that test.py runs (if examples are enabled in the build) are specified in each examples directory in a new file "examples-to-run.py", rather than in the main test.py program. 
+
  
ns-3-dev-pending is not finished but is far enough along for people to get the idea of what it will eventually look like for ns-3.11.  The items that are unfinished but are proposed to be finished for this release are:
+
We probably want to explore and tailor the capability for users to provide feedback and hints about appsWe do not want the capability to be abused by spammers (e.g. a captcha?).
# Generate python bindings, at least the first step belowGustavo previously proposed to do this in two stages: 
+
## require all modules to be built for python while modular python binding generation is worked on.  The python user would import a single monolithic python module, as is presently done.
+
## make python bindings truly modular; bindings would be maintained in a bindings/ directory with each module, and the python modules will be decomposed along module boundaries (e.g. "import ns3-network")
+
# not all modules are cut over to the new module directory structure, but we believe that the main dependency issues are resolved or easily resolvable, and the other modules can be cut over in short order
+
# src/helper will go away; helpers will be maintained on a per-module basis.
+
# Not all test code has been properly redistributed in the new directory layout.  The plan is that "build verification tests" (BVTs) will live with the model code, and other tests (e.g. UNIT) will live in a test/ directory.
+
# some examples or samples may migrate from the top level examples/ or samples/ directory to the module examples/ directory for this release.  However, since some examples contain many more dependencies than the individual modules that are linked, we suggest to keep a top-level directory for such example programs.
+
# Presently the system requires that each module have a test library.  This requires at least that each wscript have a test block in its wscript file so that libns3-modulename-test.so is created.
+
# ./waf --doxygen needs to be re-enabled; print-introspected-doxygen utility is not presently built as part of all modules.
+
# Fix project documentation (tutorial, wiki, manual, etc.) to align with the new layout
+
  
=== New module layout ===
+
== Documentation and module creation scripts ==
  
A prototypical module looks like this:
+
* Doxygen and Sphinx scripts should be modified to allow building on a per-module basis; ns-3 web site should still integrate all of these module documentations coherently
 +
* The create-module.py program should be updated
 +
* The ns-3 tutorial must be updated
  
  src/modulename/
+
== Regression testing ==
                doc/
+
                examples/
+
                          examples-to-run.py
+
                helper/
+
                model/
+
                test/
+
                utils/
+
                wscript
+
  
Not all directories will be present in each module.
+
We need to write some new [https://ns-buildmaster.ee.washington.edu:8010/ Jenkins scripts] to test the known modules for regressions.
  
=== Module dependencies ===
+
== Versioning ==
  
[[File:Module-relationships.png]]
+
Managing the version numbers of different apps will be necessary.  We want users to be able to declare that apps are compatible with specific ns-3 versions.
  
[[File:plot_module_dependencies.sh]]
+
== Patch management ==
  
=== Modular libraries ===
+
Apps may fall out of date (become unmaintained) but it may be the case that some simple patches could be applied on top of them to make them workable again with the latest ns-3.  The framework should support the ability to declare that a module installation consists of downloading an archive and then patching it with a file fetched from elsewhere.  A module may also require patching of ns-3 core itself; the framework should support this.
  
Enabling a module will cause two libraries to be built:  libns3-modulename.so and libns3-modulename-test.so.  For example, try these commands:
+
== Dependency management ==
  
  ./waf  configure --disable-python --enable-modules=core
+
If some modules depend on other modules or system dependencies, that needs to be expressed and handled by the tools. This is mainly a bake build issue, but it probably requires also some documentation in the app store that the module depends, for example, on a system library such as Boost.
  ./waf
+
  cd build/debug/
+
  ls
+
  
and the following libraries should be present:
+
== Module grouping ==
  
  bindings  libns3-core.so      ns3      scratch utils
+
We will probably want support for groups of modulesUsers should be able to specify, for example, "Ad hoc routing" and fetch modules for several protocols, which may be hosted at different sites.
  examples  libns3-core-test.so  samples  src
+
  
Running test.py will cause only those tests that depend on module core to be run:
+
= contrib module requirements for ns-3.29 =
+
  20 of 24 tests passed (20 passed, 4 skipped, 0 failed, 0 crashed, 0 valgrind errors)
+
  
Repeat for the "network" module instead of the "core" module, and the following will be built, since network depends on core:
+
== Goals ==
  
  bindings  libns3-core.so      libns3-network.so      ns3      scratch  utils
+
1. The source code layout of a module is the same as a module found in the <code>src</code> directory of ns-3:
  examples  libns3-core-test.so  libns3-network-test.so  samples  src
+
  
=== the .ns3rc file ===
+
  bindings doc examples helper model test wscript
  
Using a command-line option (--enable-modules=modulename) to ./waf at configure stage can be used to control which modules are builtThere is an alternate way, which is to write an .ns3rc file:
+
2.  Additionally, the module must contain two version files.  <code>VERSION</code> is the version number of the module, and should be a conventional sequence-based identifier (such as '1.0.0') allowing users to easily interpret older and newer versions<code>NS3-VERSION</code> is a recommended or known-good version of ns-3 with which the module is compatible, and should just be a single line.
  
   cat .ns3rc
+
   release <release tag number>
  
  #! /usr/bin/env python
+
or
  
   # A list of the modules that will be enabled when ns-3 is run.
+
   changeset <mercurial changeset hash>
  # Modules that depend on the listed modules will be enabled also.
+
  #
+
  # All modules can be enabled by choosing 'all_modules'.
+
  modules_enabled = ['core', 'network', 'internet', 'mpi', 'mobility', 'bridge', 'propagation', 'spectrum']
+
  
 +
Examples:
  
The precedence rules are as follows:
+
  release ns-3.28
  
# the --enable-modules configure string overrides any .ns3rc file
+
  changeset 13418:82d130348a36
# the .ns3rc file in the top level ns-3 directory is next consulted, if present
+
# the system searches for ~/.ns3rc if the above two are unspecified
+
# /etc/ns3rc is checked
+
  
If none of the above limits the modules to be built, all modules that waf knows about will be built.
+
The build system will use the NS3-VERSION to warn of possible incompatibilities.
  
Presently, the .ns3rc file in the repository http://code.nsnam.org/watrous/ns-3-dev-pending lives in the top-level directoryAs such, it will be prone to accidental checkins from maintainers that enable/disable the default configurationTherefore, it is proposed that the maintained version live in the utils/ directory, and users will need to manually copy to their preferred place (top level directory, their home directory) to enable persistent modular build configuration.
+
3.  The source code must be provided from a hosting provider such as GitHub or Bitbucket, or as an source archive downloadable from the webThe module should be provided such that it can be cloned directly (or untarred) into the contrib directory without cloning also the upstream ns-3-dev repository.
  
=== examples-to-run.py ===
+
For example:
  
In ns-3.10, test.py hardcodes a number of examples to run in the test suite.  In ns-3.11, we propose to add a new file "examples-to-run.py" in each module test/ directory, to tell the test framework which of the models should be run and under which test conditions.  The syntax is the same as in current test.py; e.g. the src/spectrum/test/examples-to-run.py is given below:
+
  $ cd ns-3-allinone/ns-3-dev/contrib
 +
  $ git clone https://github.com/sampleuser/sample-module
 +
  $ ls sample-module
 +
  bindings doc examples helper model NS3-VERSION test VERSION
  
  #! /usr/bin/env python
+
4.  The module should be named in lowercase with hyphen separation of words, similar to ns-3 <code>src</code> directory conventions.
  ## -*- Mode: python; py-indent-offset: 4; indent-tabs-mode: nil; coding: utf-8; -*-
+
  # A list of C++ examples to run in order to ensure that they remain
+
  # buildable and runnable over timeEach tuple in the list contains
+
  #
+
  #    (example_name, do_run, do_valgrind_run).
+
  #
+
  # See test.py for more information.
+
  cpp_examples = [
+
    ("adhoc-aloha-ideal-phy", "True", "True"),
+
    ("adhoc-aloha-ideal-phy-with-microwave-oven", "True", "True"),
+
  ]
+
  # A list of Python examples to run in order to ensure that they remain
+
  # runnable over time.  Each tuple in the list contains
+
  #
+
  #    (example_name, do_run).
+
  #
+
  # See test.py for more information.
+
  python_examples = []
+
  
=== Open issue review ===
+
5.  All module C++ source code should be put into a separate C++ namespace within ns3 namespace, corresponding to the module name, similar to how ns-3's aodv and olsr are structured.
  
'''Open issue:''' Should src/ directory be renamed to modules/, or kept as is?
+
5.  ns-3's <code>create-module.py</code> should be moved to the contrib/ directory, and documentation updated to describe this change.
  
* Suggest to keep as src/ at least for now, to minimize changes
+
== Non-goals ==
  
'''Open issue:''' Should modules be flat in the modules/ directory, or preserve
+
The below are non-goals for ns-3.29 release, although they may be worked on for future releases.
hierarchy such as modules/routing/dsdv?
+
  
* Suggest to flatten, per Nicola's (and others) suggestion
+
1.  provide build scripts so that each module can be built, tested, and installed as a completely standalone library without invoking ns-3's Waf, although this will not be precluded if a user wishes to provide some build scripts to do so.
  
'''Open issue:''' Should tests be built by default, or should they be disabled?
+
2provide build scripts to build the module Doxygen separately from the rest of ns-3 (perhaps this could be supported in ns-3.29 if someone can work on it).  
In other programs, you typically do a "make test" explicitlyHere, we
+
could add a "./waf test" explicit command, or we could make it built by
+
default but disabled if users "./waf configure --disable-tests".
+
  
* The present situation is that test libraries will be built by default.
+
3.  support modules that require patching of the mainline ns-3 release to function.  We could experiment with including a <code>patches</code> directory that can store patches for the mainline that must be applied by bake.
  
'''Open issue:''' Bug 848:  Linux FHS requires version number appended to the
+
= Packaging for ns-3.29 release =
library. Should we start doing this?
+
  
* Currently, we are not proposing to fix this for ns-3.11.
+
The <code>ns-allinone-3.29</code> release tarball will contain all modules (the universe) supported for ns-3.29 and found in the app store.  However, the default build of ns-3.29 will exclude the contributed modules.
  
'''Open issue:''' Do we want to keep including version number in the ns-3 directory name?
+
We may decide to later release subsets of the universe of modules (such as the so-called 'spins' of Linux desktop distros), but for simplicity's sake for ns-3.29, we will just include all.
  
* No change proposed for ns-3.11
+
We may decide to migrate some relatively unused existing ns-3 modules from <code>src</code> to <code>contrib</code> at release time, to shrink the default build.
  
'''Open issue:''' Install/uninstall to typical file system locations, e.g.
+
Proposed API:
/usr/lib/libns3-simulator.so.3.10
+
/usr/lib/libns3-simulator.so.3 -> /usr/lib/libns3-simulator.so.3.10
+
  
* No change proposed for ns-3.11.
+
  # by default, will exclude contrib modules:
 +
  $ ./waf configure --enable-examples --enable-tests 
 +
  # selectively include sample-module in the usual way:
 +
  $ ./waf configure --enable-examples --enable-tests --enable-modules=core,network,lte,sample-module
 +
  # include all src and contrib modules meeting dependency requirements:
 +
  $ ./waf configure --enable-examples --enable-tests --enable-modules=all
  
'''Open issue:''' Documentation organization.  Proposing that each module
+
= Development workflow for contrib modules =
contain
+
its .rst file(s) in the module/<modulename>/doc/ directory.  Modules that
+
want to chain up to the main documentation tree can do so by adding
+
themselves
+
to the index.rst.  In the future, when users develop modules in the app
+
store
+
framework, users can choose to chain up to the main manual, or they can
+
provide
+
their own makefiles and even choose their own markup (e.g. latex) if
+
they want, and separate html/pdf can be generated.
+
  
* No change proposed for ns-3.11
+
The usual way for people to work on a new feature has been to fork ns-3-dev and either edit existing modules or create new modules in <code>src</code>.  This results in a parallel repository to ns-3-dev, completely stand-alone from ns-3-dev.
  
== Stage 2 Status ==
+
In the new model, authors will create standalone module repositories and will need to place them within the contrib/ folder of ns-3 to work on them.
  
For the second stage, we have been looking at GNOME jhbuild to build packages at
+
Will [https://gist.github.com/gitaarik/8735255 Git submodules] be the preferred recommended way for people to set up a contrib development environment? Someone should investigate this or propose alternatives.
the top-level ns-3-allinone directory.  jhbuild may need to be
+
wrapped by a specialized ns-3 build.py or download.py wrapper. However, we
+
encountered these limitations so far:
+
# jhbuild has no support for scons
+
# waf support is limited; for instance, controlling configuration options is limited
+
# jhbuild has no support for hierarchical builds: although you can have metamodules, and moduleset includes, there is (AFAIK) no option to say "build only this metamodule".  The options to "start at module X", or "skip module Y" are not sufficient.
+

Latest revision as of 15:21, 19 April 2018

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Background

The goal is to support modularization of the ns-3 code base and federation of development activities. To date, the main path to distributing extensions to ns-3 has been to request a code review for merge to the mainline tree. The future goal is to shrink the ns-3 core, and to better facilitate modules that are independently developed and maintained. As a concrete example, the R statistical framework does not integrate every extension to its main release; instead, it allows users to fetch the packages of interest and add to their installation. Communities-of-interest have even arisen to allow researchers to organize and curate sets of R extensions (e.g. bioinformatic packages). We envision the same for ns-3.

In the ns-3.27 release, the bake build orchestration tool was extended to allow third party contributed code modules to be added to ns-3, and to be downloaded, built, and installed with the main ns-3 release. This allows modules to be hosted elsewhere and integrated with the ns-3-allinone framework. Consider this to be the back-end support for modularization. We have not gone further since that time.

For example, the sift routing code contribution has been added as a module. Please review how users are expected to add sift to their ns-3.27 release on this wiki page.

There are two main components:

1) front end The ns-3 app store (https://apps.nsnam.org) is largely operational and will be put into actual use for ns-3.29. The app store can be used to directly download contributed modules, and to post information about the module.

2) back end The back end will be the bake build orchestration tool.

Workflows

We envision three types of users:

  1. . End user: The normal user will browse to the app store to learn about available apps, learn about resources (user forums, documentation) about each app, and possibly leave feedback on a review board. The normal user will obtain the necessary information to fetch and install each app from this page.
  2. . Developer: Developers will not be able to create new apps on the app store, but will be able to edit the app page that he or she owns. To make an app available on the store, it has to go through an ns-3 review process.
  3. . Maintainer: ns-3 maintainers will create a new page for an accepted app, and create a developer account for the author. Then the developer will be able to edit the page to suit their app.

Requirements

The following requirements were used to develop the app store (https://apps.nsnam.org).

  1. . The front page should be clean, consisting of categories, newest releases, and popular downloads.
  2. . Each "app" will be an ns-3 module, with a description page including: Description, screenshots (optional), download tab, resources tab, etc.
  3. . Clear documentation must be posted on how to use the app store, and how to submit a new app to the app store
  4. . The "Download" tab will fetch a source archive, and the user will download to a location of their choice. The user is responsible for moving or copying the package to their ns-3 development directory, and then the ns-3 build tools (invoked at command line by users in the normal way) will do the rest.
  5. . XML for the bake configuration will also be maintained for each module, allowing module contributors to post new releases of their module.

Enhancements

Once the basic capability is deployed, enhancements are possible. Some of these extensions may involve modifying the app store installation, but some will involve modifying ns-3 build scripts, documentation, and other scripts (e.g. Jenkins scripts).

Integration with bake

Bake uses XML to describe modules. It would be nice to have a standardized XML schema that describes an ns-3 module, and make it usable from both the app store and from bake. For instance, there could be an "authors" field in the XML description that can be displayed by bake at the command line and can also be parsed and shown as the Authors in the app store description.

User feedback and comments

We probably want to explore and tailor the capability for users to provide feedback and hints about apps. We do not want the capability to be abused by spammers (e.g. a captcha?).

Documentation and module creation scripts

  • Doxygen and Sphinx scripts should be modified to allow building on a per-module basis; ns-3 web site should still integrate all of these module documentations coherently
  • The create-module.py program should be updated
  • The ns-3 tutorial must be updated

Regression testing

We need to write some new Jenkins scripts to test the known modules for regressions.

Versioning

Managing the version numbers of different apps will be necessary. We want users to be able to declare that apps are compatible with specific ns-3 versions.

Patch management

Apps may fall out of date (become unmaintained) but it may be the case that some simple patches could be applied on top of them to make them workable again with the latest ns-3. The framework should support the ability to declare that a module installation consists of downloading an archive and then patching it with a file fetched from elsewhere. A module may also require patching of ns-3 core itself; the framework should support this.

Dependency management

If some modules depend on other modules or system dependencies, that needs to be expressed and handled by the tools. This is mainly a bake build issue, but it probably requires also some documentation in the app store that the module depends, for example, on a system library such as Boost.

Module grouping

We will probably want support for groups of modules. Users should be able to specify, for example, "Ad hoc routing" and fetch modules for several protocols, which may be hosted at different sites.

contrib module requirements for ns-3.29

Goals

1. The source code layout of a module is the same as a module found in the src directory of ns-3:

 bindings doc examples helper model test wscript

2. Additionally, the module must contain two version files. VERSION is the version number of the module, and should be a conventional sequence-based identifier (such as '1.0.0') allowing users to easily interpret older and newer versions. NS3-VERSION is a recommended or known-good version of ns-3 with which the module is compatible, and should just be a single line.

 release <release tag number>

or

 changeset <mercurial changeset hash>

Examples:

 release ns-3.28
 changeset 13418:82d130348a36

The build system will use the NS3-VERSION to warn of possible incompatibilities.

3. The source code must be provided from a hosting provider such as GitHub or Bitbucket, or as an source archive downloadable from the web. The module should be provided such that it can be cloned directly (or untarred) into the contrib directory without cloning also the upstream ns-3-dev repository.

For example:

 $ cd ns-3-allinone/ns-3-dev/contrib
 $ git clone https://github.com/sampleuser/sample-module
 $ ls sample-module
 bindings doc examples helper model NS3-VERSION test VERSION

4. The module should be named in lowercase with hyphen separation of words, similar to ns-3 src directory conventions.

5. All module C++ source code should be put into a separate C++ namespace within ns3 namespace, corresponding to the module name, similar to how ns-3's aodv and olsr are structured.

5. ns-3's create-module.py should be moved to the contrib/ directory, and documentation updated to describe this change.

Non-goals

The below are non-goals for ns-3.29 release, although they may be worked on for future releases.

1. provide build scripts so that each module can be built, tested, and installed as a completely standalone library without invoking ns-3's Waf, although this will not be precluded if a user wishes to provide some build scripts to do so.

2. provide build scripts to build the module Doxygen separately from the rest of ns-3 (perhaps this could be supported in ns-3.29 if someone can work on it).

3. support modules that require patching of the mainline ns-3 release to function. We could experiment with including a patches directory that can store patches for the mainline that must be applied by bake.

Packaging for ns-3.29 release

The ns-allinone-3.29 release tarball will contain all modules (the universe) supported for ns-3.29 and found in the app store. However, the default build of ns-3.29 will exclude the contributed modules.

We may decide to later release subsets of the universe of modules (such as the so-called 'spins' of Linux desktop distros), but for simplicity's sake for ns-3.29, we will just include all.

We may decide to migrate some relatively unused existing ns-3 modules from src to contrib at release time, to shrink the default build.

Proposed API:

 # by default, will exclude contrib modules:
 $ ./waf configure --enable-examples --enable-tests  
 # selectively include sample-module in the usual way:
 $ ./waf configure --enable-examples --enable-tests --enable-modules=core,network,lte,sample-module
 # include all src and contrib modules meeting dependency requirements:
 $ ./waf configure --enable-examples --enable-tests --enable-modules=all

Development workflow for contrib modules

The usual way for people to work on a new feature has been to fork ns-3-dev and either edit existing modules or create new modules in src. This results in a parallel repository to ns-3-dev, completely stand-alone from ns-3-dev.

In the new model, authors will create standalone module repositories and will need to place them within the contrib/ folder of ns-3 to work on them.

Will Git submodules be the preferred recommended way for people to set up a contrib development environment? Someone should investigate this or propose alternatives.