Difference between revisions of "BakeIntegration"

From Nsnam
Jump to: navigation, search
(bake integration)
(more bake proposal)
Line 93: Line 93:
  
 
The resulting bake directory then contains bake itself, and subdirectories for each repo or branch of ns-3.  Within each of these subdirectories, there is a source/ and a build/ directory corresponding to that configuration.   
 
The resulting bake directory then contains bake itself, and subdirectories for each repo or branch of ns-3.  Within each of these subdirectories, there is a source/ and a build/ directory corresponding to that configuration.   
 
  
 
'''4.  Managing debug, optimized, and static builds'''
 
'''4.  Managing debug, optimized, and static builds'''
  
See the previous item.
+
See the previous item.  Users can maintain separate ns-3-dbg, ns-3-opt, ns-3-static, etc. directories.
  
 
'''5.  Passing configure options to bake, especially ns-3 ones such as --disable-python'''
 
'''5.  Passing configure options to bake, especially ns-3 ones such as --disable-python'''
Line 107: Line 106:
 
   openflow=$SRCDIR/openflow-ns3 --with-nsc=$SRCDIR/nsc"/>
 
   openflow=$SRCDIR/openflow-ns3 --with-nsc=$SRCDIR/nsc"/>
  
 +
'''6.  cleaning the directory'''
  
 +
''Note:'' this proposal not yet reviewed by Daniel.
  
6. cleaning the directory
+
* './bake.py clean' will remove the build/ directory and any build artifacts and configuration information; * './bake.py distclean' will perform a clean and also remove source/
  
''Note:'' proposal not yet reviewed by Daniel.
+
'''7. Configuration manipulation'''
  
'./bake.py clean' will remove the build/ directory and any build artifacts and configuration information; './bake.py distclean' will perform a clean and also remove source/
+
The act of configuring bake will leave a "bakefile.xml" file storing the state of the configuration. This can be further manipulated by enabling/disabling options.
  
7Configuration manipulation
+
For example, consider the user who has configured ns-3-dev but wants to deselect/disable netanimThis sequence of commands can accomplish that:
  
-d and -e behavior
+
  ./bake.py configure -e ns-3-dev
 +
  ./bake.py fix-config -f bakefile.xml -d netanim
  
8.  Show behavior
+
Some time later, the following command:
 +
 
 +
  ./bake.py configure -e ns-3-dev
 +
 
 +
will reconfigure and overwrite entirely the bakefile.xml.
 +
 
 +
It is also possible for users to store their own custom configs and modules in a .bakerc file that can be colocated with the current directory or the user's home directory.  For example, consider the user who wants a minimal ns-3 with only the wifi module enabled.  He or she could populate the following in a .bakerc file:
 +
 
 +
  <predefined name="ns3-min">
 +
    <enable name="ns-3-dev"/>
 +
    <disable name="nsc"/>
 +
    <disable name="net_anim"/>
 +
    <disable name="pybindgen"/>
 +
    <disable name="click"/>
 +
    <disable name="openflow-ns3"/>
 +
    <append module="ns-3-dev" name="configure_arguments" value=" --enable-modules=core,network,wifi"/>
 +
 
 +
'''8.  Show behavior'''
  
 
bake.py shows available packages, whether they are enabled, and their dependencies; e.g.:
 
bake.py shows available packages, whether they are enabled, and their dependencies; e.g.:
Line 150: Line 169:
 
       click (optional:True)
 
       click (optional:True)
 
       openflow-ns3 (optional:True)
 
       openflow-ns3 (optional:True)
 
  
 
In the example above, the gccxml-ns3 dependency is shown for pygccxml.  It is out of scope for bake.py to manage package dependencies for some of these (e.g. showing "scons" as a dependency for nsc).  However, it would be nice if, during the course of the build, if a package configure bails out, that "bake.py build" reports this to a user in a nice way so the user can try to find/install the missing package.
 
In the example above, the gccxml-ns3 dependency is shown for pygccxml.  It is out of scope for bake.py to manage package dependencies for some of these (e.g. showing "scons" as a dependency for nsc).  However, it would be nice if, during the course of the build, if a package configure bails out, that "bake.py build" reports this to a user in a nice way so the user can try to find/install the missing package.
Line 169: Line 187:
 
   > path tool - Ok
 
   > path tool - Ok
 
   > Autotools - Ok
 
   > Autotools - Ok
 
  
 
Perhaps an enhancement to the above would be to show the "build" and "install" status of each.
 
Perhaps an enhancement to the above would be to show the "build" and "install" status of each.
  
Users should be able to disable (in a sticky way) a package they are not interested in; e.g.:
+
'''9. Other behavior'''
 +
 
 +
bake.py handles control-C signal to abort the build.
  
  ./bake.py configure -d netanim
+
It would be nice if bake.py does not rebuild things unnecessarily.  This is a policy question to answer (about salvaging possibly dirty builds).
  
would result in netanim being "(disabled)".   Either of the following would restore netanim:
+
'''10. bake.py handling of ns-3 enabled_modules'''
  
  ./bake.py reconfigure (or some other way to get back to defaults)
+
For ns-3.16, there is no special handling of this configure options; existing ns-3 mechanisms continue to work (e.g. .ns3rc)It is still an open question of what more to do (if any) for this issue.
  ./bake.py configure -e netanim
+
  
9Other behavior
+
'''11ns-3's scratch directory'''
  
- bake.py does not rebuild things unnecessarily
+
We propose to keep this as is for now. Over time, it may be that we need to provide something similar to scratch/ at the bake level, although it is possible that some of the magic of waf will be replaced by the more explicit use of environment variables along with autotools/Makefile skeletons as well.
- bake.py handles control-C to abort the build
+
  
10. bake.py handling of ns-3 enabled_modules
+
waf may need to learn of these new (default) install locations.
  
11.  DCE
+
'''12.  DCE'''
  
bake.py provides access to ns-3 DCE with at least one well-documented example (iperf?  quagga?) of how to use real application code within an ns-3 program.  Perhaps we focus on user-mode applications for ns-3.16 and kernel-mode for the next release.
+
bake.py provides access to ns-3 DCE with at least one well-documented example (iperf) of how to use real application code within an ns-3 program.  We focus on user-mode applications for ns-3.16 and kernel-mode for the next release.  DCE has its own documentation system; we will look into whether its patch for ns-3 can be merged to ns-3-dev or whether it still needs to sit outside of ns-3-dev.
  
12.  external module
+
'''13.  external module'''
  
(defer) bake.py allows at least one "external" ns-3-based module to be downloaded and built with ns-3.  For example, assume there is an external routing protocol module (ns-3-bgp); we may need to make one of these up for the sake of providing an example.
+
'''(defer to ns-3.17)''' bake.py allows at least one "external" ns-3-based module to be downloaded and built with ns-3.  For example, assume there is an external routing protocol module (ns-3-bgp); we may need to make one of these up for the sake of providing an example.
  
 
For ns-3-bgp to build, it needs to link to ns-3 libraries.  So, there needs to be some install step in bake to install to a well-known place in the allinone directory, or in the system.
 
For ns-3-bgp to build, it needs to link to ns-3 libraries.  So, there needs to be some install step in bake to install to a well-known place in the allinone directory, or in the system.
Line 206: Line 223:
 
   ./bake.py download
 
   ./bake.py download
 
   ./bake.py build      # builds ns-3, then builds ns-3-bgp
 
   ./bake.py build      # builds ns-3, then builds ns-3-bgp
  ./bake.py install    # installs both (does install have to happen before ns-3-bgp build?)
 
  
 
Then, user program in ns-3.16/scratch/ can include the module without specifying the full path to ns-3-bgp; i.e., waf may need to become smarter to find these headers and libraries.
 
Then, user program in ns-3.16/scratch/ can include the module without specifying the full path to ns-3-bgp; i.e., waf may need to become smarter to find these headers and libraries.

Revision as of 06:54, 13 November 2012

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

This page is for describing functional requirements of the bake tool for ns-3.16 and for future releases.

Broad functional goals

1. Enable a more distributed package environment for ns-3 third-party developers.

1.1 Split out some of existing ns-3-dev into a smaller "core" and additional "extensions". This should reduce build time for users.

1.2 Provide straightforward way for a third-party developer to maintain their own ns-3 extension and make it available to ns-3 users.

2. Provide a tool for developers to manage multiple builds.

3. Start to provide access to INRIA's Direct Code Execution (DCE) environment

ns-3.16 goals

Below are proposed goals for ns-3.16.

1. bake.py provides existing download.py and build.py with no change of functionality

1.1 For ns-3-dev (after merge):

 hg clone http://code.nsnam.org/ns-3-allinone
 cd ns-3-allinone
 ./bake.py configure -e ns-3-dev
 ./bake.py download
 ./bake.py build

alternative: the "install" command will "download" then "build"

 hg clone http://code.nsnam.org/ns-3-allinone
 cd ns-3-allinone
 ./bake.py configure -e ns-3-dev
 ./bake.py install

1.2 For ns-3.16:

 tar xjf ns-allinone-3.16.tar.bz2
 cd ns-allinone-3.16
 ./bake.py configure -e ns-3.16
 ./bake.py build

1.3 existing download.py and build.py

We keep existing download.py and build.py files, but they print out a backward-compatibility warning and call the equivalent bake commands; e.g.

 cd ns-allinone-3.16
 ./build.py
  "Warning:  build.py is deprecated, please use './bake.py configure -e ns-3.16 && ./bake.py build' instead.

2. different levels of verbosity show different amount of build information

 ./bake.py install 

yields something sparse, such as

 >> Downloading ns-3-dev
 >> Download ns-3-dev - OK 
 >> Building ns-3-dev
 >> Built ns-3-dev - OK
   

while this:

 ./bake.py install -vv

yields the full build output (full waf configure and build information).

3. Managing multiple ns-3 versions and branches

Maintainers commonly work with multiple builds. This can be most easily accomplished using subdirectories.

 hg clone http://code.nsnam.org/bake

Note: It is an open question whether bake/ is separate from ns-3-allinone/ repository, or part of ns-3-allinone/ repository. Continuing with this example:

 cd bake
 mkdir ns-3-dev
 cd ns-3-dev
 ../bake.py configure -c ../bakeConf.xml -e ns-3-dev
 ../bake.py download
 ../bake.py build

and a little later:

 cd bake
 mkdir ns-3.16
 cd ns-3.16
 ../bake.py configure -c ../bakeConf.xml -e ns-3.16
 ../bake.py download
 ../bake.py build

The resulting bake directory then contains bake itself, and subdirectories for each repo or branch of ns-3. Within each of these subdirectories, there is a source/ and a build/ directory corresponding to that configuration.

4. Managing debug, optimized, and static builds

See the previous item. Users can maintain separate ns-3-dbg, ns-3-opt, ns-3-static, etc. directories.

5. Passing configure options to bake, especially ns-3 ones such as --disable-python

Each module (nsc, netanim, etc.) has its own set of configuration options. The way that bake presently handles this is by editing the bakeConf.xml file directly (or by setting a predefined module; see below). It may be possible to pass configuration options down to ns-3 such as done with build.py presently. However, this complicates bake, so it is proposed for now that any options be configured via environment variables, if possible, or else by editing attributes in bakeConf.xml. For example:

 <attribute name="configure_arguments" value="configure --prefix=$INSTALLDIR --enable-examples 
 --enable-tests --with-nsclick=$SRCDIR/click --with-pybindgen=$SRCDIR/pybindgen --with-
 openflow=$SRCDIR/openflow-ns3 --with-nsc=$SRCDIR/nsc"/>

6. cleaning the directory

Note: this proposal not yet reviewed by Daniel.

  • './bake.py clean' will remove the build/ directory and any build artifacts and configuration information; * './bake.py distclean' will perform a clean and also remove source/

7. Configuration manipulation

The act of configuring bake will leave a "bakefile.xml" file storing the state of the configuration. This can be further manipulated by enabling/disabling options.

For example, consider the user who has configured ns-3-dev but wants to deselect/disable netanim. This sequence of commands can accomplish that:

 ./bake.py configure -e ns-3-dev
 ./bake.py fix-config -f bakefile.xml -d netanim

Some time later, the following command:

 ./bake.py configure -e ns-3-dev

will reconfigure and overwrite entirely the bakefile.xml.

It is also possible for users to store their own custom configs and modules in a .bakerc file that can be colocated with the current directory or the user's home directory. For example, consider the user who wants a minimal ns-3 with only the wifi module enabled. He or she could populate the following in a .bakerc file:

 <predefined name="ns3-min">
   <enable name="ns-3-dev"/>
   <disable name="nsc"/>
   <disable name="net_anim"/>
   <disable name="pybindgen"/>
   <disable name="click"/>
   <disable name="openflow-ns3"/>
   <append module="ns-3-dev" name="configure_arguments" value=" --enable-modules=core,network,wifi"/> 

8. Show behavior

bake.py shows available packages, whether they are enabled, and their dependencies; e.g.:

./bake.py show --enabled
 module: click (enabled)
   No dependencies!
 module: openflow-ns3 (enabled)
   No dependencies!
 module: gccxml-ns3 (enabled)
   No dependencies!
 module: nsc (enabled)
   No dependencies!
 module: qt4 (enabled)
   No dependencies!
 module: pygccxml (enabled)
   depends on:
      gccxml-ns3 (optional:True)
 module: net_anim (enabled)
   depends on:
      qt4 (optional:True)
 module: pybindgen (enabled)
   depends on:
      pygccxml (optional:True)
 module: ns-3-dev (enabled)
   depends on:
      net_anim (optional:True)
      nsc (optional:True)
      pybindgen (optional:True)
      click (optional:True)
      openflow-ns3 (optional:True)

In the example above, the gccxml-ns3 dependency is shown for pygccxml. It is out of scope for bake.py to manage package dependencies for some of these (e.g. showing "scons" as a dependency for nsc). However, it would be nice if, during the course of the build, if a package configure bails out, that "bake.py build" reports this to a user in a nice way so the user can try to find/install the missing package.

The user can check the missing dependencies by calling:

./bake.py check
 > Python - Ok
 > Mercurial - Ok
 > CVS - Ok
 > Bazaar - Ok
 > Tar tool - Ok
 > Unzip tool - Ok
 > Unrar tool - Ok
 > GIT - is missing
 > Make - Ok
 > cMake - Ok
 > path tool - Ok
 > Autotools - Ok

Perhaps an enhancement to the above would be to show the "build" and "install" status of each.

9. Other behavior

bake.py handles control-C signal to abort the build.

It would be nice if bake.py does not rebuild things unnecessarily. This is a policy question to answer (about salvaging possibly dirty builds).

10. bake.py handling of ns-3 enabled_modules

For ns-3.16, there is no special handling of this configure options; existing ns-3 mechanisms continue to work (e.g. .ns3rc). It is still an open question of what more to do (if any) for this issue.

11. ns-3's scratch directory

We propose to keep this as is for now. Over time, it may be that we need to provide something similar to scratch/ at the bake level, although it is possible that some of the magic of waf will be replaced by the more explicit use of environment variables along with autotools/Makefile skeletons as well.

waf may need to learn of these new (default) install locations.

12. DCE

bake.py provides access to ns-3 DCE with at least one well-documented example (iperf) of how to use real application code within an ns-3 program. We focus on user-mode applications for ns-3.16 and kernel-mode for the next release. DCE has its own documentation system; we will look into whether its patch for ns-3 can be merged to ns-3-dev or whether it still needs to sit outside of ns-3-dev.

13. external module

(defer to ns-3.17) bake.py allows at least one "external" ns-3-based module to be downloaded and built with ns-3. For example, assume there is an external routing protocol module (ns-3-bgp); we may need to make one of these up for the sake of providing an example.

For ns-3-bgp to build, it needs to link to ns-3 libraries. So, there needs to be some install step in bake to install to a well-known place in the allinone directory, or in the system.

For a user program in ns-3.16/scratch to use ns-3-bgp, it needs to be able to find the library and headers. So, the install step on ns-3-bgp should leave the installation in a well-known place by ns-3's waf.

So, the goal here would be:

 ./bake.py configure -e ns-3-bgp
 ./bake.py download
 ./bake.py build       # builds ns-3, then builds ns-3-bgp

Then, user program in ns-3.16/scratch/ can include the module without specifying the full path to ns-3-bgp; i.e., waf may need to become smarter to find these headers and libraries.

Beyond ns-3.16

To be determined.