10.2 Running Distributed Simulations

10.2.1 Prerequisites

Ensure that MPI is installed, as well as mpic++. In Ubuntu repositories, these are openmpi-bin, openmpi-common, openmpi-doc, libopenmpi-dev. In Fedora, these are openmpi and openmpi-devel.

Note: There is a conflict on some Fedora systems between libotf and openmpi. A possible "quick-fix" is to yum remove libotf before installing openmpi. This will remove conflict, but it will also remove emacs. Alternatively, these steps could be followed to resolve the conflict:

1) Rename the tiny otfdump which emacs says it needs:

     mv /usr/bin/otfdump /usr/bin/otfdump.emacs-version

2) Manually resolve openmpi dependencies:

     sudo yum install libgfortran libtorque numactl

3) Download rpm packages: 

     openmpi-1.3.1-1.fc11.i586.rpm
     openmpi-devel-1.3.1-1.fc11.i586.rpm
     openmpi-libs-1.3.1-1.fc11.i586.rpm
     openmpi-vt-1.3.1-1.fc11.i586.rpm

     from

     http://mirrors.kernel.org/fedora/releases/11/Everything/i386/os/Packages/

4) Force the packages in:

     sudo rpm -ivh --force openmpi-1.3.1-1.fc11.i586.rpm
     openmpi-libs-1.3.1-1.fc11.i586.rpm openmpi-devel-1.3.1-1.fc11.i586.rpm
     openmpi-vt-1.3.1-1.fc11.i586.rpm

Also, it may be necessary to add the openmpi bin directory to PATH in order to execute mpic++ and mpirun from the command line. Alternatively, the full path to these executables can be used. Finally, if openmpi complains about the inability to open shared libraries, such as libmpi_cxx.so.0, it may be necessary to add the openmpi lib directory to LD_LIBRARY_PATH.

10.2.2 Building and Running Examples

If you already built ns-3 without MPI enabled, you must re-build:

./waf distclean

Configure ns-3 with the –enable-mpi option:

./waf -d debug configure --enable-mpi

Ensure that MPI is enabled by checking the optional features shown from the output of configure.

Next, build ns-3:

./waf

After building ns-3 with mpi enabled, the example programs are now ready to run with mpirun. Here are a few examples (from the root ns-3 directory):

mpirun -np 2 ./waf --run simple-distributed
mpirun -np 4 -machinefile mpihosts ./waf --run 'nms-udp-nix --LAN=2 --CN=4 --nix=1 --tracing=0'

The np switch is the number of logical processors to use. The machinefile switch is which machines to use. In order to use machinefile, the target file must exist (in this case mpihosts). This can simply contain something like:

localhost
localhost
localhost
...

Or if you have a cluster of machines, you can name them.

** NOTE: Some users have experienced issues using mpirun and waf together. An alternative way to run distributed examples is shown below:

./waf shell
cd build/debug
mpirun -np 2 examples/mpi/simple-distributed

10.2.3 Creating custom topologies

The example programs in examples/mpi give a good idea of how to create different topologies for distributed simulation. The main points are assigning system ids to individual nodes, creating point-to-point links where the simulation should be divided, and installing applications only on the LP associated with the target node.

Assigning system ids to nodes is simple and can be handled two different ways. First, a NodeContainer can be used to create the nodes and assign system ids:

NodeContainer nodes;
nodes.Create (5, 1); // Creates 5 nodes with system id 1.

Alternatively, nodes can be created individually, assigned system ids, and added to a NodeContainer. This is useful if a NodeContainer holds nodes with different system ids:

NodeContainer nodes;
Ptr<Node> node1 = CreateObject<Node> (0); // Create node1 with system id 0
Ptr<Node> node2 = CreateObject<Node> (1); // Create node2 with system id 1
nodes.Add (node1);
nodes.Add (node2);

Next, where the simulation is divided is determined by the placement of point-to-point links. If a point-to-point link is created between two nodes with different system ids, a remote point-to-point link is created, as described in Current Implementation Details.

Finally, installing applications only on the LP associated with the target node is very important. For example, if a traffic generator is to be placed on node 0, which is on LP0, only LP0 should install this application. This is easily accomplished by first checking the simulator system id, and ensuring that it matches the system id of the target node before installing the application.

This document was generated on April 21, 2010 using texi2html 1.82.