App Store Technical Requirements

From Nsnam
Revision as of 21:53, 5 January 2011 by Watrous (Talk | contribs) (API and work flow)

Jump to: navigation, search

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Goals

The long-term goal is to move ns-3 to separate modules, for build and maintenance reasons. For build reasons, since ns-3 is becoming large, we would like to allow users to enable/disable subsets of the available model library. For maintenance reasons, it is important that we move to a development model where modules can evolve on different timescales and be maintained by different organizations.

An analogy is the GNOME desktop, which is composed of a number of individual libraries that evolve on their own timescales. A build framework called jhbuild exists for building and managing the dependencies between these disparate projects.

Once we have a modular build, and an ability to separately download and install third-party modules, we will need to distinguish the maintenance status or certification of modules. The ns-3 project will maintain a set of core ns-3 modules including those essential for all ns-3 simulations, and will maintain a master build file containing metadata to contributed modules; this will allow users to fetch and build what they need. Eventually, modules will have a maintenance state associated with them describing aspects such as who is the maintainer, whether it is actively maintained, whether it contains documentation or tests, whether it passed an ns-3 project code review, whether it is currently passing the tests, etc. The current status of all known ns-3 modules will be maintained in a database and be browsable on the project web site.

Maintenance-status-example.PNG

Figure caption: Mock-up of future model status page (models and colors selected are just for example purposes)

The basic idea of the ns-3 app store would be to store on a server a set of user-submitted metadata which describes various source code packages. Typical metadata would include:

  • unique name
  • version number
  • last known good ns-3 version (tested against)
  • description
  • download url
  • untar/unzip command
  • configure command
  • build command
  • system prerequisites (if user needs to apt-get install other libraries)
  • list of ns-3 package dependencies (other ns-3 packages which this package depends upon)

Requirements

Build and configuration

  • Enable optimized, debug, and static builds
  • Separate tests from models
  • Doxygen support
  • Test.py framework supports the modularization
  • Python bindings support the modularization
  • integrate lcov code coverage tests
  • integrate with buildbots

API and work flow

Assume that our download script is called download.py and our build script is called build.py.

  wget http://www.nsnam.org/releases/ns-allinone-3.x.tar.bz2
  bunzip2 ns-allinone-3.x.tar.bz2 && tar xvf ns-allinone-3.x.tar.bz2
  cd ns-allinone-3.x

In this directory, users will find the following directory layout:

  build.py download.py constants.py? VERSION LICENSE README

Download essential modules:

  ./download.py

This will leave a layout such as follows:

  download.py build.py pygccxml pybindgen ns-3.10 gcc-xml


For typical users, the next step is as follows:

  ./build.py

The above will take the following steps:

  • build all typical prerequisites such as pybindgen
  • cd ns3-10
  • ./waf configure && ./waf && ./waf install

The above will install headers into build/debug/ns3/ and libraries for each of the enabled modules into build/debug/lib. Each enabled module will be placed in a separate library that has a name of the form libns3-simulator.so for the simulator module.

Open issue: How to express and honor platform limitations, such as only trying to download/build NSC on platforms supporting it.

Once the build process is done, there will be a bunch of libraries in a common build directory, and all applicable headers. Specifically, we envision for module simulator, there may be a libns3-simulator.so, libns3-simulator-test.so, a python .so, and possibly others.

Python

We have a few choices for supporting Python. First, note that this type of system provides an opportunity for the python build tools to be added as packages to the overall build system, so that users can more easily build their bindings. We can try to build lots of small python modules, or run a scan at the very end of the ns-3 build process to build a customized python ns3 module, such as:

  • python-scan.py
  • ...
  • pybindgen

which operates on the headers in the build/debug/ns3 directory and creates a python module .so library that matches the ns-3 configured components.

Open issue: What is the eventual python API? Should each module be imported separately such as import node as ns.node, or should we try to go for a single ns3 python module? Or, we could continue to maintain bindings. Another consideration is that constantly generating bindings will slow down the builds.

Tests

Tests are run by running ./test.py, which knows how to find the available test libraries and run the tests.

Presently, test.py hardcodes the examples or samples; it needs to become smarter to learn what examples are around and need testing.

Doxygen

Doxygen can be run such as "build.py doxygen" on the build/debug/ns3 directory. I would guess that we don't try to modularize this but instead to run on the build/debug/ns3/ directory once all headers have been copied in.

Build flags

To configure different options such as -g, -fprofile-arcs, e.g., use the CFLAGS environment variable.

Running programs

To run programs:

  ./build.py shell
  build/debug/examples/csma/csma-bridge

or

  python build/debug/examples/csma/csma-bridge.py

Other examples

Build all known ns-3 modules:

  ./build.py all

In the above case, suppose that the program could not download and fetch a module. Here, it can interactively prompt the user "Module foo not found; do you wish to continue [Y/n]?".

Build the core of ns-3, plus device models WiFi and CSMA:

  ./build ns3-core wifi csma

ns3-core is a meta-module containing simulator common node mobility etc. Open issue: what is the right granularity for this module?

Example third-party ns-3 module

Suppose a research group at Example Univ. publishes a new WiMin device module. It gets a new unique module name from ns-3 project, such as wimin. It also contributes metadata to the master ns-3 build script.

Downloading

The "download.py" system should have enough metadata to fetch it, whether it is a tarball release or some other kind of release.

Open issue: How does user toggle the download behavior of a module? Is there a sticky "./download.py --disable-module=wimin" option? Or does user manually edit the module metadata file?

Open issue: Where does this module download to? Options include:

  1. ns-3-allinone/wimin
  2. ns-3-allinone/ns-3-dev/wimin
  3. ns-3-allinone/ns-3-dev/modules/wimin

In the first case above, this will result in an ns-3-allinone directory that is flat and possibly including non-ns3 modules mixed with ns-3 modules (e.g. pygccxml, nam-1, aodv, olsr, simulator, click, ... will all be at the same directory level).

In the second case, it has advantage of keeping all ns-3-specific modules within an ns-3 subdirectory. This may support better the developer who has multiple ns-3 branches that are supported by "common" libraries such as nsc, pybindgen, etc. that do not have to change.

The third case is somewhat analogous to the present situation where we have ns-3-allinone/ns-3-dev/src/wimin.

Building

 ./build.py

at the top level should recurse and cause the wimin module to be built based on the provided (jhbuild) metadata.

At this top level, there need to be some global CFLAGS in effect that cause debug/optimized/static to be consistently performed.

Let's assume that the downloaded module looked like this:

 ns-3-allinone/ns-3-dev/wimin/model
                             /examples
                             /doc
                             /test
                             /helper
                             /bindings

After the build step, it looks like this:

 ns-3-allinone/ns-3-dev/wimin/model
                             /examples
                             /doc
                             /test
                             /helper
                             /lib/libns3-wimin.so
                             /lib/libns3-wimin-test.so
                             /bindings/ns3wimin.so
                             /include
                             /bin

Installing

The install step will next put it into these places:

 installprefix/bin
 installprefix/include/wimin
 installprefix/bindings/python
 installprefix/lib

where installprefix defaults to either "ns-3-allinone/install/debug" or "ns-3-allinone/ns-3-dev/install/debug" but can be overridden if a different prefix is configured.

Plan and Status

We are aiming to first make existing ns-3 modular, and then work on the second issue of supporting easy integration of modules maintained by third parties.

Making the existing ns-3 into a modular system is mainly a job of reorganizing the source tree and modifying waf. Presently, we are looking at the following issues:

  1. A new .ns3rc file that maintains a list of modules to include/exclude from the build
  2. Define the module granularity and fix broken module dependencies
  3. Work on making the python bindings more modular
  4. Make tests more modular; separate tests from the code, and enable per-module test executables
  5. Better handle examples in a modular test framework

We hope to make the above changes shortly after ns-3.10 is released in January.

For the second issue, we have been looking at GNOME jhbuild to build packages at the top-level ns-3-allinone directory. jhbuild may need to be wrapped by a specialized ns-3 build.py or download.py wrapper. However, we encountered these limitations so far:

  1. jhbuild has no support for scons
  2. waf support is limited; for instance, controlling configuration options is limited
  3. jhbuild has no support for hierarchical builds: although you can have metamodules, and moduleset includes, there is (AFAIK) no option to say "build only this metamodule". The options to "start at module X", or "skip module Y" are not sufficient.