Difference between revisions of "Developer FAQ"

From Nsnam
Jump to: navigation, search
(Checking in code)
(General guidelines)
Line 209: Line 209:
=== General guidelines ===
=== General guidelines ===
Please always update your code to the tip of ns-3-dev before checking in.  Never force a commit (which creates another head on the repository).  If someone else committed something before you got a chance to perform your commit, you will need to pull those changes, resolve any possible conflicts, and commit.   
Please always update your code to the tip of ns-3-dev before checking in.  Never force a push (which creates another head on the repository).  If someone else pushed something before you got a chance to perform your push, you will need to pull those changes, resolve any possible conflicts, and then push. Also don't push with "--new-branch" ([http://mailman.isi.edu/pipermail/ns-developers/2013-May/011169.html the use of ''named branches'' is not allowed]).
Before checking in code, it is good practice to pull from ns-3-dev and deal with any merge issues in advance:
Before checking in code, it is good practice to pull from ns-3-dev and deal with any merge issues in advance:
Line 230: Line 230:
   hg pull
   hg pull
   hg update
   hg update
   hg commit -m"branch merge"
   hg commit -m "branch merge"
   ./test.py  (make sure you didn't break anything)
   ./test.py  (make sure you didn't break anything)
   hg push ssh://code@code.nsnam.org/repos/ns-3-dev
   hg push ssh://code@code.nsnam.org/repos/ns-3-dev
Line 236: Line 236:
If your commit closes a bug in the bugzilla database, please label the commit with the bug number, note the resultant hexidecimal changelog ID, and mark the bug as RESOLVED FIXED and list the changelog ID in the tracker entry.  e.g.:
If your commit closes a bug in the bugzilla database, please label the commit with the bug number, note the resultant hexidecimal changelog ID, and mark the bug as RESOLVED FIXED and list the changelog ID in the tracker entry.  e.g.:
   hg commit -m"bug 1365:  unused variable declaration of a private data member"
   hg commit -m "bug 1365:  unused variable declaration of a private data member"
=== When to commit ===
=== When to commit ===

Revision as of 17:35, 31 May 2013

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Mercurial repository layout for developers

  1. In your home dir on code.nsnam.org, you will find a new directory named "repositories/username". e.g.: /home/tomh/repositories/tomh. If you create a repository in this directory, it will appear automatically on http://code.nsnam.org
    • Note: To enable this for new users, edit /var/www/cgi-hg/hgweb.config
  2. You can obviously ssh to your personal account and manage these repositories
  3. You can push to these repositories with the command: hg push ssh://tomh@code.nsnam.org/repositories/tomh/ns-3-com
  4. You can pull with the usual commands: hg clone http://code.nsnam.org/tomh/ns-3-com
  5. If you want to allow another user to push into your repository, all you have to do is change the unix permissions of your repository to allow this user write permissions. This means that if you want to give everyone write permissions, you can "chmod -R g +rw /home/tomh/repositories/tomh/ns-3-com/". If you want to allow only a smaller subset of users to push, we will need to create unix group which matches this subset
  6. The push command for the main tree is still: hg push ssh://code@code.nsnam.org/repos/ns-3-dev
  7. New developers, please read this: When creating a new repository, do not "hg clone" it into your directory on code.nsnam.org (which will generate a big ns-commits mail message); instead, just copy (cp -r) or rsync it to your local "/home/username/repositories/username" directory.

Mercurial tips

  1. How to undo a commit: Let's suppose you are working on a private repository and you check something in, but some other files were inadvertently checked in, and you want to revert and start over. There are two ways to do this:
    1. hg revert: This can be used to revert the repository to a previous revision number. For example, to revert to changeset number #1000, type hg revert -r 1000 --all. This does not remove your checkin from the repository history. For example, if your mistaken checkin was number 1001, and you revert back to 1000 and then commit, you will be at changeset number 1002 now even though the code matches what was in there at changeset 1000.
    2. hg rollback: This can be used to completely wipe clean the last transaction only (commit, import, push, pull). Use with care-- cannot be undone.
  2. How to rename a file: hg rename old-file-name new-file-name This is preferable to adding the new file name and removing the old file name, because it preserves revision history. Don't forget to commit once you are done.
  3. How to merge a branch: If you have forked a branch repository, have worked on it, and are ready to merge it back to ns-3-dev, here are the steps to take (Also, read this chapter to better understand how the mercurial source tree is structured when merging is occurring):
    1. cd into your branch
    2. hg pull http://code.nsnam.org/ns-3-dev
    3. hg merge
    4. resolve all of the merge issues, if any, and confirm that it builds and validates
    5. hg ci -m"merge your-branch-name with tip"
    6. hg push ssh://code@code.nsnam.org/repos/ns-3-dev
  4. How to create patches for circulation: hg export tip Suppose you don't have write access to the main repositories, but have some patches you'd like to circulate. This command outputs out all the diffs for the latest changeset you committed into your local repository. Simply redirect this output to a file, and you can circulate your patch for consideration. This is for when you have committed changesets. If you would like to export uncommitted changes as a patch, use: hg diff This gets the diffs of all the uncommitted changes versus what is checked into the repository. Redirect to a file for circulation among developers, or for inclusion with a bug report, etc.
  5. Save a push URL so that you don't need to enter it each time: If you are tired of having to specify the same ssh URL each time when you go to push, you can specify a default push location in the branch's local hgrc.
    1. cd into your branch
    2. vi .hg/hgrc
    3. add the following line to the [paths] section
    4. default-push = ssh://username@code.nsnam.org/repositories/username/ns-3-reponame
    5. hg push should now "just work" without the URL
  6. How to fix two-headed repositories: Note, please never check in code by forcing it (with hg push -f option). This will cause the repository to have multiple heads. If the repository ends up with two heads, this can fix it:
    1. hg merge
    2. hg commit -m"merge two heads"
    3. hg push ...

The WAF build system

See also the Waf User FAQ.

Obtaining WAF

A snapshot of WAF is distributed with ns-3 releases and mercurial branches. This snapshopt has been tested to work correctly with ns-3, although the trunk version from the main WAF repository usually works equally well.

Documentation resources

There is a WAF book. Some useful tips can be found in the WAF Wiki. Finally, there is a plethora of examples distributed in WAF itself, in the 'demos' directory.

How to add new ns-3 modules

Ns-3 is organized as a set of modules. Each module is built as a set of object files, has a name, may depend on other modules, and installs a specific set of public header files.

Starting with version 3.11, ns-3 went to a modular directory structure. The following chapter in the ns-3 Manual shows how to add a module in the new directory structure:

Adding a New Module to ns-3 (version 3.11 or later)

If you are using a version of ns-3 earlier than 3.11, then follow the instructions in this FAQ.

To add a new ns-3 module to the WAF build system, begin by creating a directory under the src/ subtree, with the source files inside. We will use p2p module as example here. Each module needs to define a wscript file. For instance let us see what src/devices/point-to-point/wscript contains:

## -*- Mode: python; py-indent-offset: 4; indent-tabs-mode: nil; coding: utf-8; -*-

def build(bld):
    module = bld.create_ns3_module('point-to-point', ['node'])
    module.source = [
    headers = bld.create_obj('ns3header')
    headers.module = 'point-to-point'
    headers.source = [

A wscript file is basically a special python module. The main entry point to this module is the build(bld) python function.

In the code above, the module variable represents a ns-3 module; internally it is a WAF 'objects' build object that will be linked to be come part of the ns3 library. It is created by calling a special method bld.create_ns3_module, whose first parameter is the name of the module, and the second parameter is a list of other modules that this module depends on. Additionally, module.sources has to be set to the list of source files (excluding header files) that constitute the module. Warning: beware that the name of the module must match the name of the directory where it is built. In this case, the module is in 'src/devices/point-to-point', so the module name must be 'point-to-point'.

There is usually also a headers object. It is used to declare public header files. These files are copied to the build directory, and can be included from any module or program with #include "ns3/header-name.h".

A final step, after the wscript file is created, is to register it. Open the file src/wscript and add the new module to the all_modules list variable:

all_modules = (
    'devices/point-to-point', # <---- example here

Adding programs

Use the special method bld.create_ns3_program(name, [...dependencies...]). Example:

    obj = bld.create_ns3_program('main-simple',
                                 ['node', 'internet-node', 'applications'])
    obj.source = 'main-simple.cc'

Generating new python bindings

See the ns-3 python wiki page.

Release Process

Release Process


Your .hgrc file

Each mercurial checkin is made by a user. If there is no .hgrc configuration file in the user's home directory, mercurial will default to using the accountname@hostname. This leads to commit lines like the following:

 changeset:   7:e53ac3c458e9
 user:        tomh@powerbook.local

To avoid this, and have it print something nicer, like

 changeset:   7:e53ac3c458e9
 user:        Tom Henderson <tomh@tomh.org>

you need to add an .hgrc file to your home directory, such as follows:

 username = Tom Henderson <tomh@tomh.org>

or, for each checkin, you will need to specify the user string manually, such as:

 hg commit -u"Tom Henderson <tomh@tomh.org>" -m"commit message"

If you want to commit a change on behalf of another user, such as attributing a bug fix to the original author, you can use the above -u command to override what is in your .hgrc file

Coding style

Of course, you need to follow the ns-3 coding style: http://www.nsnam.org/developers/contributing-code/coding-style/

If you use emacs

All ns-3 files include this:

/* -*- Mode:C++; c-file-style:"gnu"; indent-tabs-mode:nil; -*- */

The "gnu" indentation style mostly conforms to the ns3 coding guidelines, but has a few quirks. One of these is that code after "namespace ns3 {" is indented, while the ns3 coding style says it should not. One workaround is typing the following command for every open buffer:

C-c C-o innamespace <ret> 0 <ret>

Fixing this issue permanently would require using a different modeline, as discussed in this thread.

If you use VIM

If you use VIM you should add the following lines to your ~/.vimrc:

set ts=2
set sw=2
set sta
set et
set ai
set si
set cin

And the following lines to realize white space errors (trailing white spaces):

let c_no_bracket_error=1
let c_no_curly_error=1
let c_comment_strings=1
let c_gnu=1

Submitting patches for consideration

The best way to submit a patch for consideration is to request a patch review:

  1. download http://codereview.appspot.com/static/upload.py
  2. record the changes you want to request a review for in a mercurial repository: "hg commit ..."
  3. within the mercurial repository, run upload.py to submit a review with http://codereview.appspot.com/. Make sure you specify ns-3-reviews@googlegroups.com as a CC.
  4. paste your review request on http://www.nsnam.org/wiki/index.php/Reviews
  5. announce your review request on ns-developers

When you send a tree without a detailed summary of your changes, it would help if you could send a list of the changesets you want to merge. To generate it, first merge with ns-3-dev and then, from your modified directory, run "hg outgoing -p http://code.nsnam.org/ns-3-dev"

Also, avoid spurious coding style and whitespace changes when preparing such a patch, as it distracts from the readability of your proposed technical changes.

If you already pulled changes from ns-3-dev into your private repository after you started doing your private modifications, there is an issue to be considered. upload.py does not let you specify a range of revisions, nor a set of changesets. So if you just run upload.py in your private repository, also the changesets pulled from ns-3-dev will be published on codereview, which is of course not desirable.

A possible workaround is to pull your changes into a temporary repository which is an up-to-date clone of ns-3-dev. The following code should do the trick. This code assumes that your private repository is in path DEV_BRANCH_WITH_NEW_FEATURE, and that it is in sync with ns-3-dev.

hg clone http://code.nsnam.org/ns-3-dev ns-3-tmp
cd ns-3-tmp
export REVNO=`hg tip -q | sed 's/:.*$//'`
hg merge
hg commit -m "merged new feature"
upload.py --rev=$REVNO --cc=ns-3-reviews@googlegroups.com 

Checking in code

General guidelines

Please always update your code to the tip of ns-3-dev before checking in. Never force a push (which creates another head on the repository). If someone else pushed something before you got a chance to perform your push, you will need to pull those changes, resolve any possible conflicts, and then push. Also don't push with "--new-branch" (the use of named branches is not allowed).

Before checking in code, it is good practice to pull from ns-3-dev and deal with any merge issues in advance:

 cd ns-3-dev
 hg pull -u

Once you think you have code to commit (or after you have done a bunch of small commits), it is good practice to run the test suite:


Depending on the intrusiveness and nature of your patch, you may want to go further with your testing, including testing debug, optimized, and static versions, and running test.py with the -g option for valgrind.

When you are ready to push upstream, the command to check in code is:

 hg push ssh://code@code.nsnam.org/repos/ns-3-dev

If it fails because you forgot to do an update before this step, you will need to:

 hg pull
 hg update
 hg commit -m "branch merge"
 ./test.py   (make sure you didn't break anything)
 hg push ssh://code@code.nsnam.org/repos/ns-3-dev

If your commit closes a bug in the bugzilla database, please label the commit with the bug number, note the resultant hexidecimal changelog ID, and mark the bug as RESOLVED FIXED and list the changelog ID in the tracker entry. e.g.:

 hg commit -m "bug 1365:  unused variable declaration of a private data member"

When to commit

When is it OK to commit code (from a bugzilla patch, or from a code review)?

  • If you are the maintainer of an existing module, you can commit changes when you want, as long as you coordinate with the release manager (if it is close to a release or code freeze date).
  • If you are not the maintainer, please obtain +1 from the maintainer if the change is not trivial. You may have to remind him or her about it; if you do not get a response, email the release manager.
  • If you would like to add a new module, or to commit to a module that does not have a maintainer, it usually should have some review by core maintainers. Email the release manager for help.
  • When in doubt, email the release manager.

Checking in patches from other users

If you as a maintainer check in a patch authored by someone else, it is good practice to credit them in the commit message using the --user or -u option to hg; e.g.:

 hg commit -m"...commit message..." -u"A User <a.user@example.com>"

You can sometimes find the appropriate user string by searching the hg log for their previous commits; e.g.

 hg log | grep <username>

Reference: http://www.gnu.org/prep/maintain/html_node/Recording-Contributors.html

Checking that you don't introduce regressions or leaks

just before you do a checkin you should run the ns-3 tests. Make sure that you have valgrind installed, and change into the top level directory and type

 ./test.py --grind

If you do this, you will see a bunch of passing tests and then, if something goes wrong:

 FAIL: TestSuite test-tcp-large-transfer

Logging your changes in the release notes

The project maintains two files for tracking changes to the codebase. As you check in changes to the code that fix bugs, add features, or change the existing behavior of the simulator, you should also update these files.

  1. CHANGES.html: Used to log changes to the build system, new API, changes to existing API, and changed behavior.
  2. RELEASE_NOTES: Summarize new user-visible features and bugs fixed.

These files may both be updated, or maybe only one is updated for a given changeset. For instance, if you change the API on a class, you might add a simple bullet "Changed API for class X to improve ability to do Y" to RELEASE_NOTES, but add more detail about exactly which signatures changed in the CHANGES.html file. If the change is trivial (e.g. adding missing Doxygen), there is no need to edit these files.

Please update these files when you make changes to ns-3-dev rather than relying on the release manager later documenting them.

Fixing a bug from the Bugzilla tracker

Q. What I should do when a bug will be fixed (send a mail to the mailing list, write some comment to bugzilla, other ) ?

A. The typical thing to do is:

  • check in the bug fix to ns-3-dev with a commit message that indicates the bug number, such as "bug 903: TapBridge doesn't close cleanly"
  • remember what the changeset number is; e.g.
 changeset:   1763:4624d5aba98f
                 it is this hexadecimal value
  • mark the bug as "Resolved, Fixed" in Bugzilla, with a comment such as "fixed in changset 4624d5aba98f". Bugzilla will notify the ns-bugs mailing list of the fix, and the ns-commits list will be notified of the checkin.

Using gcov and lcov code coverage tools

Here is a brief howto for using the lcov front-end to gcc's code coverage tool gcov

A custom version of lcov and its associated tools (geninfo, genhtml) is stored in the directory utils/lcov/, and this is the version that waf uses to run lcov. As of ns-3.13, the version of lcov in ns-3 is lcov-1.9, with geninfo patched to deal with this branching bug.

You probably also need to have a version of lcovrc file installed somewhere (either at /etc/lcovrc or ~/.lcovrc). The easiest way to do this is to install lcov from your package manager; e.g.

 sudo apt-get install lcov

or install it from source. But just be aware that waf will use the version that is maintained in the utils/lcov/ directory.

Then, for instance, to see the coverage of the ns-3 test suite, type:

 ./waf configure --enable-gcov --enable-examples --enable-tests
 ./waf --lcov-report

You will find a file "index.html" in the directory build/debug-gcov/lcov-report/ that you can look at with your browser.

Note that if you want to test coverage of another program, you will want to zero the counters for lcov. You can't presently do this from within waf but you can run this command from your top-level ns-3 directory:

 utils/lcov/lcov --directory build --zerocounters

and it should report:

 Deleting all .da files in build and subdirectories

The preferred way to create a private repository

Let's say that developer "alice" wants to create a new repository "ns-3-dev-new-feature" that will exist on the code server as alice/ns-3-dev-new-feature. Suppose she wants to fork from the tip of ns-3-dev.

 cd /home/alice/repositories/alice
 cp -r /home/code/repos/ns-3-dev ns-3-dev-new-feature
 cd ns-3-dev-new-feature/.hg

At this point, edit the "hgrc" file to provide contact/description information:

 default = http://code.nsnam.org/alice/ns-3-dev-new-feature
 description = alice's new feature
 contact = <alice@example.com>

Next, check that your repository shows up properly on the web page.

At this point, you are done with the code server, and you can perform "hg clone http://code.nsnam.org/alice/ns-3-dev-new-feature" on your local machine.

Note: A common minor problem is if you do an "hg clone" into the new directory instead of a "cp -r", there will be a huge ns-commits mail message generated. This is why "cp -r" is preferred way to do this.

Testing code on nsnam.org hosts

Some times a compilation error can only reproduced in certain architectures. nsnam.org has some machines that can be accessed remotely for testing purposes, if you have an appropriate account. Contact Tom Henderson if you need an account.

Full Suite

Note that this takes several hours to run. It puts a specified branch through its paces on several machines/architectures. You can specify a branch to test and a notification email address where you will receive a report about how the branch performed in the tests (SUCCESS or FAILURE).

$ ssh ns-regression.ee.washington.edu
$ sudo -u nsnam bash
$ cd /usr/local/bin/
$ ./ns-3-run-tests.sh -h
usage: ./ns-3-run-tests.sh options

This script runs the ns-3 branch through the ns-regression testbed.
 -h Help:  show this message
 -n ns-3 branch  Default:  ns-3-dev
 -m mailto address.  Where to send the mail.  If unspecified, program output will just scroll onto stdout.
$ ./ns-3-run-tests.sh -n mathieu/ns-3-simu -m someone@wherever.com

Ubuntu x86_64

ssh ns-regression.ee.washington.edu

Mac OS X PowerPC

ssh ns-regression.ee.washington.edu
sudo -u nsnam bash
<enter your password>
ssh darwin-ppc

Adding a buildslave to our BuildBot

The ns-3 project uses BuildBot to test the ns-3-dev build daily. The buildmaster resides on a server at the University of Washington. Currently, we use several buildslaves running Fedora Core 10 with different versions of gcc as well as a Mac OS X PPC machine. For a quick snapshot of the current buildslaves in use, please see the buildbot waterfall.

If you would like to see another buildslave, for example mingw or cygwin, and you have a machine to donate daily cycles, please complete the following steps:

  1. Install buildbot using a package manager or installing from source: download
  2. When creating the buildslave, you will need to provide a slavename and password. Note that you will have to send us this name and password. You also need the hostname and port of our buildmaster: ns-regression.ee.washington.edu:9989
  3. Follow the buildbot manual to setup a buildslave: creating a buildslave
  4. Send John Abraham <john.abraham@gatech.edu> an email with your buildslave name and password, and we will update the buildmaster configuration to accept connections from your buildslave.

To change the buildmaster master script, edit master.cfg and then 'make reconfig' in the /home/buildmaster/master directory.

To restart buildbot on ns-regression:

 su buildmaster
 cd /home/buildmaster
 /usr/bin/buildbot start master