Python bindings

From Nsnam
Revision as of 20:00, 21 April 2011 by Jabraham3 (Talk | contribs) (The semi-automatic way)

Jump to: navigation, search

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Overview

The goal of Python bindings for NS-3 are two fold:

  1. Allow the programmer to write complete simulation scripts in Python;
  2. Prototype new models (e.g. routing protocols).

For the time being, the primary focus of the bindings is the first goal, but the second goal will eventually be supported as well. Python bindings for NS-3 are being developed using a new tool called PyBindGen.

The process by which Python bindings are handled is the following:

  1. Periodically a developer uses a GCC-XML based API scanning script, which saves the scanned API definition as bindings/python/ns3_module_*.py files. These files are kept under version control in the main NS-3 repository;
  2. Other developers clone the repository and use the already scanned API definitions;
  3. When configuring NS-3, pybindgen will be automatically downloaded if not already installed. Released NS-3 tarballs will ship a copy of pybindgen.

Build instructions

Just make sure Python development headers are installed (e.g., in debian/ubuntu: sudo apt-get install python-dev), then waf configure and waf build ns-3 as usual. If you are building a ns-3 repository version (rather than official release), you will also need the program bazaar in order to automatically donwload pybindgen (e.g., in debian/ubuntu: sudo apt-get install bzr).

If something goes wrong with compiling Python bindings and you just want to ignore them and move on with C++, you can disable Python with:

./waf --disable-python

Testing

"./waf check" now also runs some Python unit tests, at the end.

To debug the python bindings themselves, Gustavo reports this: "Unfortunately, Python does some memory allocation tricks that valgrind reports as errors. When I want to check Python bindings memory errors, I used a debug python build (configure python with ./configure --with-pydebug --without-pymalloc), and then valgrind works. But with a standard Python binary valgrind will always detect phony errors and we don't want that."

Running programs

waf contains some options that automatically update the python path to find the ns3 module. To run example programs, there are two ways to use waf to take care of this. One is to run a waf shell; e.g.:

./waf --shell
python examples/mixed-wireless.py

and the other is to use the --pyrun option to waf:

./waf --pyrun examples/mixed-wireless.py

To run a python script under the C debugger:

./waf --shell
gdb --args python examples/mixed-wireless.py

Instructions for wrapping new or changed APIs

The manual way

You will need to read some PyBindGen documentation.

API of existing module changed or added

So you have been changing existing NS-3 APIs and Python bindings no longer compile? Do not despair, you can manually edit the bindings API definitions to make them reflect the new changed API.

The API definitions are organized as follows. For each ns-3 module <name>, the file bindings/python/ns3_module_<name>.py describes its API. Each of those files have 3 toplevel functions:

  1. def register_types(module): this function takes care of registering new types (e.g. C++ classes, enums) that are defined in tha module;
  2. def register_methods(module): this function calls, for each class <name>, another function register_methods_Ns3<name>(module). These latter functions add method definitions for each class;
  3. def register_functions(module): this function registers ns-3 functions that belong to that module.

New module

If you are adding a new module, Python bindings will continue to compile but will not cover the new module. To cover a new module, you have to create a bindings/python/ns3_module_<name>.py file, similar to the what is described in the previous section, and register it in the variable LOCAL_MODULES in bindings/python/ns3modulegen.py

The semi-automatic way

It is possible to use GCC-XML and PyGccXml to scan API definitions automatically. The API scanner is not perfect, but it gets most of the job done.

First you need to install the necessary tools:

  1. Get CMake (can either build from source or download binaries for windows, linux i386);
  2. Get GCC-XML (must download from CVS). Instructions are to be found here. Note that, as with all CVS versions, stability may vary. This CVS snapshot is known to work: cvs up -D "2009-09-21".
  3. Download and install pygccxml, version 1.0.0; (To install: python setup.py install);If you use Ubuntu's apt-get then the command is "apt-get install python-pygccxml)
  4. Run ./waf configure, to let ns-3 detect the python scanning tools. You should see something along these lines; if not, something is wrong:
[...]
Checking for Python module pygccxml                                      : ok  
Checking for pygccxml version                                            : ok 1.0.0 
Checking for program gccxml                                              : ok /usr/local/bin/gccxml 
Checking for gccxml version                                              : ok 0.9.0 
[...]
Python Bindings               : enabled
Python API Scanning Support   : enabled

Now you are ready to scan; run ./waf --python-scan. If all goes well, the files in bindings/python/* should have been updated with new API definitions. Now you can build ns-3 as usual, and new Python bindings are compiled.

Pitfalls

"invalid use of incomplete type"

If you get a "invalid use of incomplete type" compilation error message, like this:

[722/796] cxx: build/debug/bindings/python/ns3_module_internet_stack.cc -> build/debug/bindings/python/ns3_module_internet_stack_3.o
debug/ns3/ptr.h: In destructor ‘ns3::Ptr<T>::~Ptr() [with T = ns3::Ipv4Interface]’:
debug/ns3/arp-cache.h:50:   instantiated from here
debug/ns3/ptr.h:439: error: invalid use of incomplete type ‘struct ns3::Ipv4Interface’
debug/ns3/arp-cache.h:40: error: forward declaration of ‘struct ns3::Ipv4Interface’

It means that PyBindGen is generating code that makes use of the implicit copy constructor, and the copy constructor does not work with incomplete types. The solution is to make the copy constructor private, then rescan the API.

class ArpCache : public Object
{
private:
  ArpCache (ArpCache const &);
  ArpCache& operator= (ArpCache const &);
[...] 
}
KeyError

If you get a KeyError during bindings code generation, like:

   File "/home/gjc/projects/ns/ns-3-allinone/ns-3-distributed/bindings/python/apidefs/gcc-LP64/ns3_module_mpi.py", line 13, in register_types
     module.add_class('DistributedSimulatorImpl', parent=root_module['ns3::SimulatorImpl'])
 KeyError: 'ns3::SimulatorImpl'

It is possible that the order of registering module types is incorrect. You should correct the inter-module dependencies in the module's wscript file:

 -  sim = bld.create_ns3_module('mpi', ['core'])
 +  sim = bld.create_ns3_module('mpi', ['core', 'simulator'])

Then re-scan the python bingins (waf --python-rescan).

RuntimeError: pygccxml error: unable to find fundamental type with name '__int128_t'.

Recent optimizations in the Time class make use of a __int128_t type, which pygccxml is not written to recognize. A simple workaround for this problem is to configure with --high-precision-as-double, scan, then switch back to normal options:

./waf configure --high-precision-as-double
./waf --python-scan
./waf configure [normal options]

Alternatively, update pygccxml to svn revision 1844.

ImportError: undefined symbol

If you declare but don't implement a public method, and rescan the bindings, compilation may succeed but you may get a cryptic runtime error such as:

 ./waf --pyrun examples/tutorial/first.py
 ImportError: /home/user/ns-3-allinone/ns-3-dev/build/debug/bindings/python/ns3/_ns3.so: undefined   symbol: _ZN3ns34dsdv12RoutingTable23InvalidateRoutesWithDstERKSt3mapINS_11Ipv4AddressEjSt4lessIS3_ESaISt4pairIKS3_jEEE
 Command ['/usr/bin/python', 'examples/tutorial/first.py'] exited with code 1

This can be fixed by finding the method that exists in the class declaration but that is not implemented, and removing the declaration, and rescanning the bindings.

Caveats

Python bindings for NS-3 is a work in progress, and some limitations are know by developers. Some of these limitations (not all) are listed here.

Incomplete Coverage

First of all keep in mind that not 100% of the API is supported in Python. Some of the reasons are:

  1. some of the APIs involve pointers, which require knowledge of what kind of memory passing semantics (who owns what memory). Such knowledge is not part of the function signatures, and is either documented or sometimes not even documented. Annotations are needed to bind those functions;
  2. Sometimes a unusual fundamental data type or C++ construct is used which is not yet supported by PyBindGen;
  3. GCC-XML does not report template based classes unless they are instantiated.

Most of the missing APIs can be wrapped, given enough time, patience, and expertise, and will likely be wrapped if bug reports are submitted. However, don't file a bug report saying "bindings are incomplete", because we do not have manpower to complete 100% of the bindings.

Conversion Constructors

Conversion constructors are not fully supported yet by PyBindGen, and they always act as explicit constructors when translating an API into Python. For example, in C++ you can do this:

Ipv4AddressHelper ipAddrs;
ipAddrs.SetBase ("192.168.0.0", "255.255.255.0");
ipAddrs.Assign (backboneDevices);

In Python, for the time being you have to do:

ipAddrs = ns3.Ipv4AddressHelper()
ipAddrs.SetBase(ns3.Ipv4Address("192.168.0.0"), ns3.Ipv4Mask("255.255.255.0"))
ipAddrs.Assign(backboneDevices)

CommandLine

CommandLine::AddValue works differently in Python than it does in NS-3. In Python, the first parameter is a string that represents the command-line option name. When the option is set, an attribute with the same name as the option name is set on the CommandLine object. Example:

    NUM_NODES_SIDE_DEFAULT = 3

    cmd = ns3.CommandLine()

    cmd.NumNodesSide = None
    cmd.AddValue("NumNodesSide", "Grid side number of nodes (total number of nodes will be this number squared)")

    cmd.Parse(argv)

    [...]

    if cmd.NumNodesSide is None:
        num_nodes_side = NUM_NODES_SIDE_DEFAULT
    else:
        num_nodes_side = int(cmd.NumNodesSide)

Tracing

Callback based tracing is not yet properly supported for Python, as new NS-3 API needs to be provided for this to be supported. Bug #127 for details.

Pcap file writing is supported via the normal API.

Ascii tracing is supported since NS-3.4 via the normal C++ API translated to Python. However, ascii tracing requires the creation of an ostream object to pass into the ascii tracing methods. In Python, the C++ std::ofstream has been minimally wrapped to allow this. For example:

   ascii = ns3.ofstream("wifi-ap.tr") # create the file
   ns3.YansWifiPhyHelper.EnableAsciiAll(ascii)
   ns3.Simulator.Run()
   ns3.Simulator.Destroy()
   ascii.close() # close the file

There is one caveat: you must not allow the file object to be garbage collected while NS-3 is still using it. That means that the 'ascii' variable above must not be allowed to go out of scope or else the program will crash.

Cygwin limitation

Python bindings do not work on Cygwin. This is due to a gccxml bug: http://www.gccxml.org/Bug/view.php?id=7572

You might get away with it by re-scanning API definitions from within the cygwin environment (./waf --python-scan). However the most likely solution will probably have to be that we disable python bindings in CygWin.

If you really care about Python bindings on Windows, try building with mingw and native python instead. Or else, to build without python bindings, disable python bindings in the configuration stage:

 ./waf configure --disable-python

Troubleshooting

OS X problem and resolution

Problem: I installed pygccxml 1.0.0 and pybindgen 0.14.0, patched bindings/python/wscript to use the newer versions, installed gccxml-devel from ports, and succesfully compiled ns-3. All tests (./test.py) work fine.

However, when I try to run a python-example I get a python crash:

 $ ./waf --pyrun examples/wireless/mixed-wireless.py
  Waf: Entering directory `/Volumes/Shared/Projects/ns-3/ns-3-dev/build'
  Waf: Leaving directory `/Volumes/Shared/Projects/ns-3/ns-3-dev/build'
  'build' finished successfully (1.114s)
   Fatal Python error: Interpreter not initialized (version mismatch?)
   Command ['/usr/bin/python', 'examples/wireless/mixed-wireless.py'] exited with code -6

Solution: This can be due to the version used to build ns3 is not the same as the interpreter, possibly due to multiple versions of Python on the system. However, it was resolved for this specific OS X case as follows:

"We found the problem, it was in Mac OS X "port" system, causing confusion with ns-3 configure script. Adding a symlink /opt/local/bin/python -> /opt/local/bin/python2.6 fixed the issue."

Modular Python Bindings

Introduction

Some experimental work has started to make ns-3 Python bindings more modular. With modular Python bindings:

  1. There is one separate Python extension module for each ns-3 module;
  2. Scanning API definitions (apidefs) is done on a per ns- module basis;
  3. Each module's apidefs files are stored in a 'bindings' subdirectory of the module directory;

Usage

Since ns 3.11, the modular bindings are being added, in parallel to the old monolithic bindings. They are disabled by default. To enable, use the --bindings-type waf option (e.g. ./waf configure --bindings-type=modular).

The new python bindings are generated into a 'ns' namespace, instead of 'ns3' for the old bindings. Example:

 from ns.network import Node
 n1 = Node()

Adding modular bindings to a module

To begin adding support for modular bindings to an existing ns-3 module, begin by adding the following line to its wscript build() function:

   bld.ns3_python_bindings()

Then, you just need scan for apidefs of that module:

 ./waf --apiscan=mymodule

Modular bindings files layout

The src/<module>/bindings directory may contain the following files, some of them optional:

  • callbacks_list.py: this is a scanned file, DO NOT TOUCH. Contains a list of Callback<...> template instances found in the scanned headers;
  • modulegen__gcc_LP64.py: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, LP64 architecture (64-bit)
  • modulegen__gcc_ILP32.py: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, ILP32 architecture (32-bit)
  • modulegen_customizations.py: you may optionally add this file in order to customize the pybindgen code generation
  • scan-header.h: you may optionally add this file to customize what header file is scanned for the module. Basically this file is scanned instead of ns3/<module>-module.h. Typically, the first statement is #include "ns3/<module>-module.h", plus some other stuff to force template instantiations;
  • module_helpers.cc: you may add additional files, such as this, to be linked to python extension module, but they have to be registered in the wscript. Look at src/core/wscript for an example of how to do so;
  • <module>.py: if this file exists, it becomes the "frontend" python module for the ns3 module, and the extension module (.so file) becomes _<module>.so instead of <module>.so. The <module>.py file has to import all symbols from the module _<module> (this is more tricky than it sounds, see src/core/bindings/core.py for an example), and then can add some additional pure-python definitions.



Open issues

List of issues to be resolved with modular bindings (for developers):

  1. In the modular bindings there was custom code for wrapping ns-3 Callbacks, with a callbacks_list.py file holding the list of detected Callback types. This stuff wasn't yet replicated in the modular bindings;
  2. In the old bindings, some code was injected in the scanned header file to force some template instantiations (such as Time operators); this is missing for now in the new bindings;
  3. When a module includes a header of another module, all the types of the other module are imported by its python module. Maybe it is possible to detect which types are actually needed by the current module's bindings and import just those types, to speed up the import.