Difference between revisions of "App Store Technical Requirements"

From Nsnam
Jump to: navigation, search
(Plan)
(modularity plan update)
Line 59: Line 59:
 
= Plan =
 
= Plan =
  
Start by experimenting with how to make a self-contained model (e.g. ns-3-dev/src/devices/uan) into a separate library. 
+
A current prototype is being worked in watrous/ns-3-modular/
  
Study how existing package management systems work (Debian, jhbuild, others?)
+
== Goals ==
  
More details to be posted later.
+
* user can explicitly enable/disable ns-3 optional packages
 +
** in a more generic form than having to modify waf, such as was done for NSC
 +
** avoid the existing “--enable-“ prefix for now; use something like “pkgadd/pkgrm”
 +
* avoid doing everything in waf
 +
** we want to allow future module developers to pick their module build system (make, waf, scons)
 +
** waf may be too painful to orchestrate everything; a dedicated python program may be better in the long run
 +
 
 +
Non-goals (for this stage of work):
 +
* able to download code from third party
 +
* able to update code from third party
 +
* package metadata
 +
* some way to standardize on package configuration step (e.g. configure.xml)
 +
* address the issue of third-party libraries (e.g. it++, openflow)
 +
* change C++ API and make it available in Python
 +
 
 +
== Steps ==
 +
 
 +
1.  Create ns-3-modular repository sandbox for this work
 +
 
 +
2.  Remove all trace of uan from system:
 +
* bindings
 +
* src/devices/uan
 +
* waf wscripts
 +
 
 +
3.  Add uan directory back into ns-3-modular/packages subdirectory. 
 +
 
 +
* Include a bindings/ directory containing ns3_module_uan.py. 
 +
* Include also a ns3_module_uan.cc until pybindgen is upgraded to handle ns-3 packages (note that this implies that we cannot generate new bindings for the moment)
 +
 
 +
4.  Define the following top-level (ns-3-modular) bash shell scripts for proof-of-concept.  These shell scripts are in lieu of modifying waf at this stage of prototyping.
 +
 
 +
  configure.sh
 +
  build.sh
 +
  install.sh
 +
  run.sh and pyrun.sh
 +
  distclean.sh
 +
 
 +
5.  configure.sh <args>
 +
 
 +
If --pkgadd-uan is passed as args, it calls “./waf configure” and then “configure.sh” in the packages/uan directory, and it writes “uan” to a “.packages” file in the top level directory.  It also calls “configure.sh” in the packages/uan directory.
 +
 
 +
Here, it should be noted that build/c4che/debug.py contains lots of environment variable settings that might be importable into a Makefile using make “shell” command.
 +
 
 +
6.  build.sh
 +
 
 +
Should first call ./waf build and then, if “uan” is present in .packages file, call build.sh in packages/uan. 
 +
 
 +
7.  install.sh
 +
 
 +
If “uan” is present in .packages file, copy over uan libraries and executables to the right place in ns-3-modular/build/ directory.
 +
 
 +
8.  run.sh, pyrun.sh
 +
 
 +
This is functionally equivalent to ./waf --run and ./waf --pyrun.  For starters, just set LD_LIBRARY_PATH and PYTHONPATH appropriately and call the named executable outside of waf.
 +
 
 +
An issue is what to do about python library.  Presently, python programs can just do an “import ns3”-- do they need to do an “import uan” as well? 
 +
 
 +
9.  distclean.sh
 +
 
 +
Performs the equivalent of a make distclean in the packages/uan directory, and does a ./waf distclean and removes the .packages file.
 +
 
 +
The outcome of this work is that one should be able to take the following steps:
 +
 
 +
  ./configure.sh --pkgadd-uan
 +
  ./build.sh
 +
  ./install.sh
 +
  ./run.sh uan-rc-example
 +
  ./pyrun.sh uan-example.py # you will need to create a trivial uan-example.py to test this
 +
  ./distclean.sh
 +
  ./configure.sh
 +
  ./build.sh
 +
  ./install.sh
 +
  ./run.sh uan-rc-example (error, uan-rc-example not found)
 +
  ./pyrun.sh uan-example.py (error, uan module not found)
 +
  ./distclean.sh
 +
 
 +
== Open issues ==
 +
 
 +
./waf --doxygen is not expected to work in this phase of prototyping
 +
 
 +
./waf --python-scan is not expected to pick up any uan API changes, in this phase of prototyping
 +
 
 +
./test.py is not expected to pick up any uan tests in this phase of prototyping
 +
 
 +
./configure.sh <args> should take <args> ($@) and strip out “--pkgadd-uan” and pass the rest to ./waf configure.
 +
 
 +
./configure.sh --pkgrm-uan should remove uan from the build (i.e. strip from the .packages directory)
 +
 
 +
Need to consider implications of wrapping all uan API within a uan namespace.
 +
* Have to consider backward compatibility and Python issues of this.
 +
 
 +
The above issues can start to be tackled as a second step.

Revision as of 18:20, 26 October 2010

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Requirements

Devise a new source code organization, build system, and package management (metadata and scripts) system that provides the following.

The top level goals are as follows:

  • Modular build, with capability to enable/disable models from the build
  • Package management system, to allow users to manage model packages that are provided by ns-3 as well as those that may be provided by third parties
  • Uniform way to handle and manage (non-system) external libraries used with ns-3

Beyond modularity, there is a need to distinguish the maintenance status of models:

  • ns-3-dev continues to contain the core models, test suites, and documentation that are actively maintained by the project and built by all users
  • ns-3-models contains additional models that can be added to the build with user-friendly build instructions. There will be two classifications:
    • maintained: These contain models that are actively maintained by the project. All models in this directory are checked for each release, and bugs in tracker are responded to
    • unmaintained: These contain models that conform to the package metadata system, but the project does not do any testing or maintenance of them. They may fall out of date over time. They may be revived in the future and added to the maintained status.
  • models can also be stored at external web sites; such models will typically have "unmaintained" status. Note that "unmaintained" means unmaintained by the project; someone may be actively maintaining them behind the scenes.

The basic idea of the ns-3 app store would be to store on a server a set of user-submitted metadata which describes various source code packages. Typical metadata would include:

  • unique name
  • version number
  • last known good ns-3 version (tested against)
  • description
  • download url
  • untar/unzip command
  • configure command
  • build command
  • system prerequisites (if user needs to apt-get install other libraries)
  • list of ns-3 package dependencies (other ns-3 packages which this package depends upon)

There are a few metadata systems to be considered, such as Debian spec files and jhbuild.

API and work flow

There are separate tasks that people will want to do on a per-module basis, and perhaps on a top-level repository with specific components enabled/disabled:

  • download code
  • update code
  • configure; enable/disable them from the build
  • enable optimized/debug
  • build, compile, and link model code
  • install: copy exported .so binaries, .h to an "install" location (maybe specified during configure)
  • build examples: compile and link examples
  • build tests: compile and link etsts
  • run tests
  • generate documentation

One idea is to have a workflow like this:

 hg clone http://code.nsnam.org/build
 cd build
 ./build.py module-list.xml download
 ./build.py module-list.xml configure
 ./build.py module-list.xml build
 ./build.py module-list.xml install

module-list.xml would describe the list of modules to build, for each module the list of commands for each task.

Plan

A current prototype is being worked in watrous/ns-3-modular/

Goals

  • user can explicitly enable/disable ns-3 optional packages
    • in a more generic form than having to modify waf, such as was done for NSC
    • avoid the existing “--enable-“ prefix for now; use something like “pkgadd/pkgrm”
  • avoid doing everything in waf
    • we want to allow future module developers to pick their module build system (make, waf, scons)
    • waf may be too painful to orchestrate everything; a dedicated python program may be better in the long run

Non-goals (for this stage of work):

  • able to download code from third party
  • able to update code from third party
  • package metadata
  • some way to standardize on package configuration step (e.g. configure.xml)
  • address the issue of third-party libraries (e.g. it++, openflow)
  • change C++ API and make it available in Python

Steps

1. Create ns-3-modular repository sandbox for this work

2. Remove all trace of uan from system:

  • bindings
  • src/devices/uan
  • waf wscripts

3. Add uan directory back into ns-3-modular/packages subdirectory.

  • Include a bindings/ directory containing ns3_module_uan.py.
  • Include also a ns3_module_uan.cc until pybindgen is upgraded to handle ns-3 packages (note that this implies that we cannot generate new bindings for the moment)

4. Define the following top-level (ns-3-modular) bash shell scripts for proof-of-concept. These shell scripts are in lieu of modifying waf at this stage of prototyping.

 configure.sh
 build.sh
 install.sh
 run.sh and pyrun.sh
 distclean.sh

5. configure.sh <args>

If --pkgadd-uan is passed as args, it calls “./waf configure” and then “configure.sh” in the packages/uan directory, and it writes “uan” to a “.packages” file in the top level directory. It also calls “configure.sh” in the packages/uan directory.

Here, it should be noted that build/c4che/debug.py contains lots of environment variable settings that might be importable into a Makefile using make “shell” command.

6. build.sh

Should first call ./waf build and then, if “uan” is present in .packages file, call build.sh in packages/uan.

7. install.sh

If “uan” is present in .packages file, copy over uan libraries and executables to the right place in ns-3-modular/build/ directory.

8. run.sh, pyrun.sh

This is functionally equivalent to ./waf --run and ./waf --pyrun. For starters, just set LD_LIBRARY_PATH and PYTHONPATH appropriately and call the named executable outside of waf.

An issue is what to do about python library. Presently, python programs can just do an “import ns3”-- do they need to do an “import uan” as well?

9. distclean.sh

Performs the equivalent of a make distclean in the packages/uan directory, and does a ./waf distclean and removes the .packages file.

The outcome of this work is that one should be able to take the following steps:

 ./configure.sh --pkgadd-uan
 ./build.sh 
 ./install.sh
 ./run.sh uan-rc-example
 ./pyrun.sh uan-example.py # you will need to create a trivial uan-example.py to test this
 ./distclean.sh
 ./configure.sh
 ./build.sh
 ./install.sh
 ./run.sh uan-rc-example (error, uan-rc-example not found)
 ./pyrun.sh uan-example.py (error, uan module not found)
 ./distclean.sh

Open issues

./waf --doxygen is not expected to work in this phase of prototyping

./waf --python-scan is not expected to pick up any uan API changes, in this phase of prototyping

./test.py is not expected to pick up any uan tests in this phase of prototyping

./configure.sh <args> should take <args> ($@) and strip out “--pkgadd-uan” and pass the rest to ./waf configure.

./configure.sh --pkgrm-uan should remove uan from the build (i.e. strip from the .packages directory)

Need to consider implications of wrapping all uan API within a uan namespace.

  • Have to consider backward compatibility and Python issues of this.

The above issues can start to be tackled as a second step.