Python bindings

Note: This wiki page is somewhat historical/background with stale information. If you plan to work with bindings or rescan them for any ns-3 C++ changes you might make, please read instead the chapter in the manual on this topic.

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

Overview

Note: This wiki page is somewhat historical/background with stale information. If you plan to work with bindings or rescan them for any ns-3 C++ changes you might make, please read instead the chapter in the manual on this topic.

The goal of Python bindings for NS-3 are two fold:

1. Allow the programmer to write complete simulation scripts in Python;
2. Prototype new models (e.g. routing protocols).

For the time being, the primary focus of the bindings is the first goal, but the second goal will eventually be supported as well. Python bindings for NS-3 are being developed using a new tool called PyBindGen.

The process by which Python bindings are handled is the following:

1. Periodically a developer uses a GCC-XML based API scanning script, which saves the scanned API definition as bindings/python/ns3_module_*.py files. These files are kept under version control in the main NS-3 repository;
2. Other developers clone the repository and use the already scanned API definitions;
3. When configuring NS-3, pybindgen will be automatically downloaded if not already installed. Released NS-3 tarballs will ship a copy of pybindgen.

Build instructions

Just make sure Python development headers are installed (e.g., in debian/ubuntu: sudo apt-get install python-dev), then waf configure and waf build ns-3 as usual. If you are building a ns-3 repository version (rather than official release), you will also need the program bazaar in order to automatically donwload pybindgen (e.g., in debian/ubuntu: sudo apt-get install bzr).

If something goes wrong with compiling Python bindings and you just want to ignore them and move on with C++, you can disable Python with:

./waf --disable-python

Note: to select a specific python version, run waf with that version's python interpreter, e.g.:

/usr/bin/python2.6 ./waf configure

Testing

"./waf check" now also runs some Python unit tests, at the end.

To debug the python bindings themselves, Gustavo reports this: "Unfortunately, Python does some memory allocation tricks that valgrind reports as errors. When I want to check Python bindings memory errors, I used a debug python build (configure python with ./configure --with-pydebug --without-pymalloc), and then valgrind works. But with a standard Python binary valgrind will always detect phony errors and we don't want that."

Running programs

waf contains some options that automatically update the python path to find the ns3 module. To run example programs, there are two ways to use waf to take care of this. One is to run a waf shell; e.g.:

./waf shell
python examples/mixed-wireless.py

and the other is to use the --pyrun option to waf:

./waf --pyrun examples/mixed-wireless.py

To run a python script under the C debugger:

./waf shell
gdb --args python examples/mixed-wireless.py

Instructions for wrapping new or changed APIs

The manual way

You will need to read some PyBindGen documentation.

API of existing module changed or added

So you have been changing existing NS-3 APIs and Python bindings no longer compile? Do not despair, you can manually edit the bindings API definitions to make them reflect the new changed API.

The API definitions are organized as follows. For each ns-3 module <name>, the file bindings/python/ns3_module_<name>.py describes its API. Each of those files have 3 toplevel functions:

1. def register_types(module): this function takes care of registering new types (e.g. C++ classes, enums) that are defined in tha module;
2. def register_methods(module): this function calls, for each class <name>, another function register_methods_Ns3<name>(module). These latter functions add method definitions for each class;
3. def register_functions(module): this function registers ns-3 functions that belong to that module.

New module

If you are adding a new module, Python bindings will continue to compile but will not cover the new module. To cover a new module, you have to create a bindings/python/ns3_module_<name>.py file, similar to the what is described in the previous section, and register it in the variable LOCAL_MODULES in bindings/python/ns3modulegen.py

The semi-automatic way

Note: The below information is stale; see instead https://www.nsnam.org/docs/manual/html/python.html#working-with-python-bindings

It is possible to use GCC-XML and PyGccXml to scan API definitions automatically. The API scanner is not perfect, but it gets most of the job done.

First you need to install the necessary tools. Instructions for installing these tools on your local machine:

1. Get CMake (can either build from source or download binaries for windows, linux i386);
2. Get GCC-XML (must download from git). Instructions are to be found here. Note that, as with all development versions, stability may vary. This git changeset is known to work: b040a46352e4d5c11a0304e4fcb6f7842008942a.
   1. Additionally, if you are running Ubuntu 64-bit version, you also need to install the Debian package: g++-multilib
   2. Additionally, if you are running Fedora 64-bit version, you will want to install glibc-devel.i686 package (in addition to glibc-devel.x86_64 to be able to generate 32-bit bindings.
   3. For Ubuntu 16.04, gccxml will not build for gcc-5 or gcc-6 series; described here: https://www.nsnam.org/bugzilla/show_bug.cgi?id=2451. Instead, you must install and use g++-4.9 (sudo apt-get install g++-4.9)
3. Download and install pygccxml; (To install: python setup.py install);If you use Ubuntu's apt-get then the command is "apt-get install python-pygccxml"

Alternatively, you can install these tools as part of the 'bake' build system, which will just install them to a local directory. See the ns-3 tutorial.

Once installed, run ./waf configure, to let ns-3 detect the python scanning tools. You should see something along these lines; if not, something is wrong:

[...]
Checking for Python module pygccxml                                      : ok  
Checking for pygccxml version                                            : ok 1.0.0 
Checking for program gccxml                                              : ok /usr/local/bin/gccxml 
Checking for gccxml version                                              : ok 0.9.0 
[...]
Python Bindings               : enabled
Python API Scanning Support   : enabled

Now you are ready to scan; run ./waf --apiscan=all to scan all bindings, or ./waf --apiscan=modulename for a specific modulename. If all goes well, the files in bindings/python/* should have been updated with new API definitions, which you can review with hg diff command. Now you can build ns-3 as usual, and new Python bindings are compiled.

Pitfalls

lots of 'did not match' warnings

If you see lots of errors of this type:

'ns3::LogTimePrinter' did not match
'void ( * ) (  ) *' did not match
'std::ostream *' did not match
'std::_List_const_iterator< std::basic_string< char, std::char_traits< char >, std::allocator< char > > >' did not match

and the apiscan fails such as:

Build failed
 -> task in '' failed (exit status 1): 
	{task 139671078693840: apiscan_task core-module.h -> modulegen__gcc_ILP32.py}
''

This is usually indicative of missing libraries, especially 32-bit libraries on a 64-bit machine. Make sure that you have installed the prerequisite packages listed on the Installation page for making modified python bindings.

waf --apiscan hangs

Sometimes when you run waf --apiscan the scanning eventually stops and waf hangs. In this case, the bindings are correctly scanned, and it's just a bug that causes WAF to hang after the scanning is completed. In this case, just kill WAF (e.g. hit Ctrl-Q).

"invalid use of incomplete type"

If you get a "invalid use of incomplete type" compilation error message, like this:

[722/796] cxx: build/debug/bindings/python/ns3_module_internet_stack.cc -> build/debug/bindings/python/ns3_module_internet_stack_3.o
debug/ns3/ptr.h: In destructor ‘ns3::Ptr<T>::~Ptr() [with T = ns3::Ipv4Interface]’:
debug/ns3/arp-cache.h:50:   instantiated from here
debug/ns3/ptr.h:439: error: invalid use of incomplete type ‘struct ns3::Ipv4Interface’
debug/ns3/arp-cache.h:40: error: forward declaration of ‘struct ns3::Ipv4Interface’

It means that PyBindGen is generating code that makes use of the implicit copy constructor, and the copy constructor does not work with incomplete types. The solution is to make the copy constructor private, then rescan the API.

class ArpCache : public Object
{
private:
  ArpCache (ArpCache const &);
  ArpCache& operator= (ArpCache const &);
[...] 
}

The other way that this error can arise is if you forward declare a class but don't otherwise include the header somewhere else in at least one of the module's header files. The file "scan-header.h" that can be found in some of the modules is designed to allow customization of what headers are included for bindings compilation, but if that is not working as expected, the simplest solution is to avoid the forward declaration.

KeyError

If you get a KeyError during bindings code generation, like:

   File "/home/gjc/projects/ns/ns-3-allinone/ns-3-distributed/bindings/python/apidefs/gcc-LP64/ns3_module_mpi.py", line 13, in register_types
     module.add_class('DistributedSimulatorImpl', parent=root_module['ns3::SimulatorImpl'])
 KeyError: 'ns3::SimulatorImpl'

It is possible that the order of registering module types is incorrect. You should correct the inter-module dependencies in the module's wscript file:

 -  sim = bld.create_ns3_module('mpi', ['core'])
 +  sim = bld.create_ns3_module('mpi', ['core', 'simulator'])

Then re-scan the python bingins (waf --python-rescan).

RuntimeError: pygccxml error: unable to find fundamental type with name '__int128_t'.

Recent optimizations in the Time class make use of a __int128_t type, which pygccxml is not written to recognize. A simple workaround for this problem is to configure with --high-precision-as-double, scan, then switch back to normal options:

./waf configure --high-precision-as-double
./waf --python-scan
./waf configure [normal options]

Alternatively, update pygccxml to svn revision 1844.

ImportError: undefined symbol

If you declare but don't implement a public method, and rescan the bindings, compilation may succeed but you may get a cryptic runtime error such as:

 ./waf --pyrun examples/tutorial/first.py
 ImportError: /home/user/ns-3-allinone/ns-3-dev/build/debug/bindings/python/ns3/_ns3.so: undefined   symbol: _ZN3ns34dsdv12RoutingTable23InvalidateRoutesWithDstERKSt3mapINS_11Ipv4AddressEjSt4lessIS3_ESaISt4pairIKS3_jEEE
 Command ['/usr/bin/python', 'examples/tutorial/first.py'] exited with code 1

This can be fixed by finding the method that exists in the class declaration but that is not implemented, and removing the declaration, and rescanning the bindings.

To decode what is the missing symbol, c++filt command:

 c++filt -n _ZN3ns34dsdv12RoutingTable23InvalidateRoutesWithDstERKSt3mapINS_11Ipv4AddressEjSt4lessIS3_ESaISt4pairIKS3_jEEE

While would output:

 ns3::dsdv::RoutingTable::InvalidateRoutesWithDst(std::map<ns3::Ipv4Address, unsigned int, std::less<ns3::Ipv4Address>, std::allocator<std::pair<ns3::Ipv4Address const, unsigned int> > > const&)

On some systems (e.g., OSX), you may get a slightly different message, which including missing symbol with two leading underscore characters, like this:

 Symbol not found: __ZN3ns36PacketC1ERKSs

To decode this, either manually remove first underscore character or remove -n command line option for c++filt:

 c++filt __ZN3ns36PacketC1ERKSs

or

 c++filt -n _ZN3ns36PacketC1ERKSs

Which should return:

 ns3::Packet::Packet(std::string const&)

Containers of a type with no default constructor

Pybindgen does not support containers whose elements are of a type that has no default constructor. The error may look like:

 [1307/1949] cxx: build/src/dsr/bindings/ns3module.cc ->
 build/src/dsr/bindings/ns3module.cc.7.o
 src/dsr/bindings/ns3module.cc: In function ‘int
 _wrap_convert_py2c__std__vector__lt___ns3__dsr__RouteCache__Neighbor___gt__(PyObject*,
 std::vector<ns3::dsr::RouteCache::Neighbor>*)’:
 src/dsr/bindings/ns3module.cc:25982:44: error: no matching function for call to
 ‘ns3::dsr::RouteCache::Neighbor::Neighbor()’

See this bug and resolution: https://www.nsnam.org/bugzilla/show_bug.cgi?id=1427

./waf --apiscan complains about gnu/stubs-32.h

If when performing an API scan on a 64-bit machine:

 ./waf --apiscan=all

you have an error "gnu/stubs-32.h: No such file or directory", this is because you need the 32-bit libc dev package to build 32-bit bindings (apiscan will try to build 32- and 64-bit bindings on a 64-bit machine). On Ubuntu, install:

 apt-get install libc6-dev-i386

On Fedora/RHEL, try:

 yum install glibc-devel.i686

and then try to scan again.

Caveats

Python bindings for NS-3 is a work in progress, and some limitations are know by developers. Some of these limitations (not all) are listed here.

Incomplete Coverage

First of all keep in mind that not 100% of the API is supported in Python. Some of the reasons are:

1. some of the APIs involve pointers, which require knowledge of what kind of memory passing semantics (who owns what memory). Such knowledge is not part of the function signatures, and is either documented or sometimes not even documented. Annotations are needed to bind those functions;
2. Sometimes a unusual fundamental data type or C++ construct is used which is not yet supported by PyBindGen;
3. GCC-XML does not report template based classes unless they are instantiated.

Most of the missing APIs can be wrapped, given enough time, patience, and expertise, and will likely be wrapped if bug reports are submitted. However, don't file a bug report saying "bindings are incomplete", because we do not have manpower to complete 100% of the bindings.

Conversion Constructors

Conversion constructors are not fully supported yet by PyBindGen, and they always act as explicit constructors when translating an API into Python. For example, in C++ you can do this:

Ipv4AddressHelper ipAddrs;
ipAddrs.SetBase ("192.168.0.0", "255.255.255.0");
ipAddrs.Assign (backboneDevices);

In Python, for the time being you have to do:

ipAddrs = ns3.Ipv4AddressHelper()
ipAddrs.SetBase(ns3.Ipv4Address("192.168.0.0"), ns3.Ipv4Mask("255.255.255.0"))
ipAddrs.Assign(backboneDevices)

CommandLine

CommandLine::AddValue works differently in Python than it does in NS-3. In Python, the first parameter is a string that represents the command-line option name. When the option is set, an attribute with the same name as the option name is set on the CommandLine object. Example:

    NUM_NODES_SIDE_DEFAULT = 3

    cmd = ns3.CommandLine()

    cmd.NumNodesSide = None
    cmd.AddValue("NumNodesSide", "Grid side number of nodes (total number of nodes will be this number squared)")

    cmd.Parse(argv)

    [...]

    if cmd.NumNodesSide is None:
        num_nodes_side = NUM_NODES_SIDE_DEFAULT
    else:
        num_nodes_side = int(cmd.NumNodesSide)

Tracing

Callback based tracing is not yet properly supported for Python, as new NS-3 API needs to be provided for this to be supported. Bug #127 for details.

Pcap file writing is supported via the normal API.

Ascii tracing is supported since NS-3.4 via the normal C++ API translated to Python. However, ascii tracing requires the creation of an ostream object to pass into the ascii tracing methods. In Python, the C++ std::ofstream has been minimally wrapped to allow this. For example:

   ascii = ns3.ofstream("wifi-ap.tr") # create the file
   ns3.YansWifiPhyHelper.EnableAsciiAll(ascii)
   ns3.Simulator.Run()
   ns3.Simulator.Destroy()
   ascii.close() # close the file

There is one caveat: you must not allow the file object to be garbage collected while NS-3 is still using it. That means that the 'ascii' variable above must not be allowed to go out of scope or else the program will crash.

Cygwin limitation

Python bindings do not work on Cygwin. This is due to a gccxml bug: http://www.gccxml.org/Bug/view.php?id=7572

You might get away with it by re-scanning API definitions from within the cygwin environment (./waf --python-scan). However the most likely solution will probably have to be that we disable python bindings in CygWin.

If you really care about Python bindings on Windows, try building with mingw and native python instead. Or else, to build without python bindings, disable python bindings in the configuration stage:

 ./waf configure --disable-python

Troubleshooting

32-bit bindings deleted and not regenerated

If you run the apiscan process on a recent Fedora or Ubuntu machine (64 bits) and you find that either you have the 32-bits bindings deleted or you take errors such as:

  Build failed
  -> task in  failed (exit status 1):
    {task 23417296: apiscan_task bridge-module.h -> 
    modulegen__gcc_ILP32.py}

This is probably a sign that either glibc-devel.i686 package is missing (Fedora) or g++-multilib is missing (Ubuntu).

OS X problem and resolution

Problem: I installed pygccxml 1.0.0 and pybindgen 0.14.0, patched bindings/python/wscript to use the newer versions, installed gccxml-devel from ports, and succesfully compiled ns-3. All tests (./test.py) work fine.

However, when I try to run a python-example I get a python crash:

 $ ./waf --pyrun examples/wireless/mixed-wireless.py
  Waf: Entering directory `/Volumes/Shared/Projects/ns-3/ns-3-dev/build'
  Waf: Leaving directory `/Volumes/Shared/Projects/ns-3/ns-3-dev/build'
  'build' finished successfully (1.114s)
   Fatal Python error: Interpreter not initialized (version mismatch?)
   Command ['/usr/bin/python', 'examples/wireless/mixed-wireless.py'] exited with code -6

Solution: This can be due to the version used to build ns3 is not the same as the interpreter, possibly due to multiple versions of Python on the system. However, it was resolved for this specific OS X case as follows:

"We found the problem, it was in Mac OS X "port" system, causing confusion with ns-3 configure script. Adding a symlink /opt/local/bin/python -> /opt/local/bin/python2.6 fixed the issue."

Note: While the above symlink will work, a less intrusive fix is to tell waf which version of python to use, by specifying './waf configure --with-python=/opt/local/bin/python2.6'. There may be other reasons why your system may not want a permanent symlink there (in case other applications on the Mac rely on finding the frameworks version of python first).

This type of problem (multiple Python versions installed on a Mac) manifests itself slightly differently in OS X Lion:

 ./waf configure --enable-examples 
 ./waf build
 ./waf --pyrun examples/tutorial/first.py

yields:

 Fatal Python error: PyThreadState_Get: no current thread

Again, the solution may be to specify '--with-python=/opt/local/bin/python2.7' when configuring waf.

The rationale is: there is a libpython2.7.dylib installed by macports in /opt/local/lib and it's loaded instead of the one found automatically by the configure system. Depending on the actual python version installed by macports, you have to point to a different python location. the exact procedure is the following:

• Check if there is a "libpythonX.Y.dylib" in /opt/local/lib by executing 'ls -l /opt/local/lib/libpyton*' in a terminal.
• The output is something like: 'lrwxr-xr-x 1 root admin 66 3 Dec 19:39 libpython2.7.dylib -> /opt/local/Library/Frameworks/Python.framework/Versions/2.7/Python'
• Find the X.Y numbers.
• Add the option '--with-python=/opt/local/bin/pythonX.Y' to waf configure, e.g., './waf configure --with-python=/opt/local/bin/pythonX.Y'

Modular Python Bindings

Introduction

Some experimental work has started to make ns-3 Python bindings more modular. With modular Python bindings:

1. There is one separate Python extension module for each ns-3 module;
2. Scanning API definitions (apidefs) is done on a per ns- module basis;
3. Each module's apidefs files are stored in a 'bindings' subdirectory of the module directory;

Usage

Since ns 3.11, the modular bindings are being added, in parallel to the old monolithic bindings. They are disabled by default. To enable, use the --bindings-type waf option (e.g. ./waf configure --bindings-type=modular).

The new python bindings are generated into a 'ns' namespace, instead of 'ns3' for the old bindings. Example:

 from ns.network import Node
 n1 = Node()

Adding modular bindings to a module

To begin adding support for modular bindings to an existing ns-3 module, begin by adding the following line to its wscript build() function:

   bld.ns3_python_bindings()

Then, you just need scan for apidefs of that module:

 ./waf --apiscan=mymodule

Modular bindings files layout

The src/<module>/bindings directory may contain the following files, some of them optional:

• callbacks_list.py: this is a scanned file, DO NOT TOUCH. Contains a list of Callback<...> template instances found in the scanned headers;
• modulegen__gcc_LP64.py: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, LP64 architecture (64-bit)
• modulegen__gcc_ILP32.py: this is a scanned file, DO NOT TOUCH. Scanned API definitions for the GCC, ILP32 architecture (32-bit)
• modulegen_customizations.py: you may optionally add this file in order to customize the pybindgen code generation
• scan-header.h: you may optionally add this file to customize what header file is scanned for the module. Basically this file is scanned instead of ns3/<module>-module.h. Typically, the first statement is #include "ns3/<module>-module.h", plus some other stuff to force template instantiations;
• module_helpers.cc: you may add additional files, such as this, to be linked to python extension module, but they have to be registered in the wscript. Look at src/core/wscript for an example of how to do so;
• <module>.py: if this file exists, it becomes the "frontend" python module for the ns3 module, and the extension module (.so file) becomes _<module>.so instead of <module>.so. The <module>.py file has to import all symbols from the module _<module> (this is more tricky than it sounds, see src/core/bindings/core.py for an example), and then can add some additional pure-python definitions.

Open issues

List of issues to be resolved with modular bindings (for developers):

1. In the modular bindings there was custom code for wrapping ns-3 Callbacks, with a callbacks_list.py file holding the list of detected Callback types. This stuff wasn't yet replicated in the modular bindings;
2. In the old bindings, some code was injected in the scanned header file to force some template instantiations (such as Time operators); this is missing for now in the new bindings;
3. When a module includes a header of another module, all the types of the other module are imported by its python module. Maybe it is possible to detect which types are actually needed by the current module's bindings and import just those types, to speed up the import.