Difference between revisions of "HOWTO Use Linux Containers to set up virtual networks"

From Nsnam
Jump to: navigation, search
m (HOWTO Use Linux Containers to set up virtual networks: Typos corrected)
(update HOWTO)
 
(One intermediate revision by one other user not shown)
Line 1: Line 1:
 
{{TOC}}
 
{{TOC}}
  
One of the interesing ways ns-3 simulations can be integrated with real hardware is to create a simulated network and use "real" hosts to drive the network.  Often, the number of real hosts is relatively large, and so acquiring that number of real computers would be prohibitively expensive.  It is possible to use virtualization technology to create software implementations, called virtual machines, that execute software as if they were real hosts.  If you have not done so, we recommend going over [[HOWTO make ns-3 interact with the real world]] to review your options.
+
One of the interesting ways ns-3 simulations can be integrated with real hardware is to create a simulated network and use "real" hosts to drive the network.  Often, the number of real hosts is relatively large, and so acquiring that number of real computers would be prohibitively expensive.  It is possible to use virtualization technology to create software implementations, called virtual machines, that execute software as if they were real hosts.  If you have not done so, we recommend going over [[HOWTO make ns-3 interact with the real world]] to review your options.
  
This HOWTO discusses how one would configure and use a modern Linux system (e.g., Debian 6.0+, Fedora 16+, Ubuntu 12.04+) to run ns-3 systems using paravirtualization of resources with Linux Containers (lxc).
+
This HOWTO discusses how one would configure and use a modern Linux system (e.g., Fedora 26+, Ubuntu 16.04+) to run ns-3 systems using paravirtualization of resources with Linux Containers (lxc).  This HOWTO focuses on Fedora 26 sample configuration.
  
 
'''Note:''' A separate project focused on [https://github.com/buzz66boy/ns3-lxc/releases LXC and ns-3 integration] has been [https://groups.google.com/forum/#!topic/ns-3-users/9DUn4AODb7c announced in 2017] by Mike Albert.
 
'''Note:''' A separate project focused on [https://github.com/buzz66boy/ns3-lxc/releases LXC and ns-3 integration] has been [https://groups.google.com/forum/#!topic/ns-3-users/9DUn4AODb7c announced in 2017] by Mike Albert.
Line 9: Line 9:
 
== Background ==
 
== Background ==
  
Always know your tools.  In this context, the first step for those unfamiliar with Linux Containers is to get an idea of what it is we are talking about and where they live.  Linux Containers, also called "lxc tools," are a relatively new feature of Linux as of this writing.  They are an offshoot of what are called "chroot jails."  A quick explanation of the FreeBSD version can be found [http://en.wikipedia.org/wiki/FreeBSD_jail here].  It may be worth a quick read before continuing to help you understand some background and terminology better.
+
The first step for those unfamiliar with Linux Containers is to get an idea of what it is we are talking about and where they live.  Linux Containers, also called "lxc tools," are an offshoot of what are called "chroot jails."  A quick explanation of the FreeBSD version can be found [http://en.wikipedia.org/wiki/FreeBSD_jail here].  It may be worth a quick read before continuing to help you understand some background and terminology better. Other useful documentation links:
  
If you peruse the web looking for references to Linux Containers, you will find that much of the existing documentation is based on [http://lxc.teegra.net/ this].  This is a rather low-level document, more of a reference than a HOWTO, and makes Linux Containers actually look more complicated than they are to get started with.  [http://www.ibm.com/developerworks/linux/library/l-lxc-containers/ IBM DeveloperWorks] provides what they call a "tour and setup" document; but I find this document a little too abstract and high-level.  
+
* https://www.cyberciti.biz/faq/how-to-install-and-setup-lxc-linux-container-on-fedora-linux-26/
 +
* https://linuxcontainers.org/
  
In this HOWTO, we will attempt to bridge the gap and give you a "just-right" presentation of Linux Containers and ns-3 that should get you started doing useful work without damaging too many brain cells.
+
=== Get and Understand the Linux Container Tools ===
  
'''We assume that you have a kernel that supports Linux Containers installed and configured on your host system.  Newer versions of Fedora come with Linux Containers enabled "out of the box" and we assume you are running that systemDetails of the HOWTO may vary if you are using a system other than Fedora.'''
+
The first step in the process is to get the Linux Container Tools onto your system.  Although Fedora systems come with Linux Containers enabled, they do not necessarily have the user tools installedYou will need to install these tools, and also make sure you have the ability to create and configure bridges, and the ability to create and configure tap devices.
  
=== Get and Understand the Linux Container Tools ===
+
Fedora packages:
  
The first step in the process is to get the Linux Container Tools onto your system. Although Fedora systems come with Linux Containers enabled, they do not necessarily have the user tools installed.  You will need to install these tools, and also make sure you have the ability to create and configure bridges, and the ability to create and configure tap devices.  If you are a sudoer, you can type the following to install the required tools.
+
  $ sudo dnf install lxc lxc-templates lxc-extra debootstrap libvirt perl gpg tunctl bridge-utils
  
'''IMPORTANT:  Linux Containers do not work on Fedora 15.  Don't even try.  The recommended solution is to upgrade to Fedora 16, 17 or 18 before continuing.'''
+
Next, start some essential services:
  
  $ sudo yum install lxc bridge-utils tunctl
+
  $ sudo systemctl start libvirtd.service
 +
$ sudo systemctl start lxc.service
 +
$ sudo systemctl enable lxc.service
  
For Ubuntu- or Debian-based systems, try the following:
+
Sample output:
 
    
 
    
$ sudo apt-get install lxc uml-utilities vtun
+
  Created symlink /etc/systemd/system/multi-user.target.wants/lxc.service ? /usr/lib/systemd/system/lxc.service.
  
 
To see a summary of the Linux Container tool (lxc), you can now begin looking at man pages.  The place to start is with the '''lxc''' man page.  Give that a read, keeping in mind that some details are not relevant at this time.  Feel free to ignore the parts about kernel requirements and configuration and ssh daemons.  So go ahead and do,
 
To see a summary of the Linux Container tool (lxc), you can now begin looking at man pages.  The place to start is with the '''lxc''' man page.  Give that a read, keeping in mind that some details are not relevant at this time.  Feel free to ignore the parts about kernel requirements and configuration and ssh daemons.  So go ahead and do,
Line 34: Line 37:
  
 
and have a read.
 
and have a read.
 
The next thing to understand is the container configuration.  There is a man page dedicated to this subject.  Again, we are tying to keep this simple and centered on ns-3 applications, so feel free to disregard the bits about pty setup for now.  Go ahead and do,
 
 
$ man lxc.conf
 
 
and have a quick read.  We are going to be interested in the '''veth''' configuration type for reasons that will soon become clear.  You may recall from the '''lxc''' manual entry that there are a number of predefined configuration templates provided with the lxc tools.  Let's take a quick look at the one for veth configuration:
 
 
$ less /usr/share/doc/lxc/examples/lxc-veth.conf
 
 
If that file exists, you should see the following file contents, or its close equivalent:
 
 
# Container with network virtualized using a pre-configured bridge named br0 and
 
# veth pair virtual network devices
 
lxc.utsname = beta
 
lxc.network.type = veth
 
lxc.network.flags = up
 
lxc.network.link = br0
 
lxc.network.hwaddr = 4a:49:43:49:79:bf
 
lxc.network.ipv4 = 1.2.3.5/24
 
lxc.network.ipv6 = 2003:db8:1:0:214:1234:fe0b:3597
 
 
From the manual entry for '''lxc.conf''' recall that the veth network type means
 
 
veth:  a new network stack is created, a peer network device is created with
 
one side assigned to the container and the other side attached to a bridge
 
specified by the lxc.network.link. The bridge has to be setup before on the
 
system, lxc won’t handle configuration outside of the container.
 
 
So the configuration means that we are setting up a Linux Container, which will appear to code running in the container to have the hostname '''beta'''.  This containter will get its own network stack.  It is not clear from the documentation, but it turns out that the "peer network device" will be named '''eth0''' in the container.  We must create a Linux (brctl) bridge before creating a container using this configuration, since the created network device/interface will be added to that bridge, which is named '''br0''' in this example.  The '''lxc.network.flags''' indicate that this interface should be activated and the '''lxc.network.ipv4''' indicates the IP address for the interface.
 
  
 
=== The Big Picture ===
 
=== The Big Picture ===
Line 158: Line 132:
 
  br-right      8000.a6915397f8cb    no            tap-right
 
  br-right      8000.a6915397f8cb    no            tap-right
  
8. ('''Note:''' On ''most'' modern distributions, including Fedora 16, 17 and 18, this step is ''not'' necessary because cgroup directory does exist and is mounted by default. Please check using
 
  
  $ ls /cgroup
+
8. Next, create a Fedora 26 container based on the 'left' configuration file found in 'src/tap-bridge/examples/':
  
if it shows the contents of the directory, this step is not necessary.)  You now have the network plumbing ready.  '''On earlier Fedora systems''' you may have to create and mount the cgroup directory.  This directory does not exist by default, so if you are the fist user to need that support you will have to create the directory and mount it.
+
  $ cd src/tap-bridge/examples
 +
  $ sudo lxc-create -f lxc-left.conf -t download -n left -- -d fedora -r 26 -a amd64
  
$ sudo mkdir /cgroup
+
You should see some output like this:
$ sudo mount -t cgroup cgroup /cgroup
+
  
If you want to make this mount persistent, you will have to edit '''fstab'''.
+
  Setting up the GPG keyring
 +
  Downloading the image index
 +
  Downloading the rootfs
 +
  Downloading the metadata
 +
  The image cache is now ready
 +
  Unpacking the rootfs
  
9. You will also have to make sure that your kernel has ethernet filtering (ebtables, bridge-nf, arptables) disabled.  If you do not do this, only STP and ARP traffic will be allowed to flow across your bridge and your whole scenario will not work.
+
  ---
 +
  You just created a Fedora container (release=26, arch=amd64, variant=default)
  
$ cd /proc/sys/net/bridge
+
  To enable sshd, run: dnf install openssh-server
$ sudo -s
+
# for f in bridge-nf-*; do echo 0 > $f; done
+
# exit
+
$ cd -
+
  
10.  Now it is time to actually start the containers.  The first step is to create the containers.  First create the left container.  The name you use will be the handle used by the lxc tools to refer to the container you create.  To keep things consistent, let's use the same name as the hostname in the conf file.
+
  For security reason, container images ship without user accounts
 +
  and without a root password.
  
$ sudo lxc-create -n left -f lxc-left.conf
+
  Use lxc-attach or chroot directly into the rootfs to set a root password
$ sudo lxc-create -n right -f lxc-right.conf
+
  or create user accounts.
  
11. Make sure that you have created your containers.  The lxc tools provide an '''lxc-ls''' command to list the created containers.
+
Then do the following:
  
  $ sudo lxc-ls
+
  $ sudo chroot /var/lib/lxc/left/rootfs/ passwd
 +
  $ sudo lxc-start -n left
 +
 
 +
9. Repeat; create a Fedora 26 container based on the 'right' configuration file found in 'src/tap-bridge/examples/':
 +
 
 +
  $ cd src/tap-bridge/examples
 +
  $ sudo lxc-create -f lxc-right.conf -t download -n right -- -d fedora -r 26 -a amd6
 +
 
 +
  $ sudo chroot /var/lib/lxc/right/rootfs/ passwd
 +
  $ sudo lxc-start -n right
 +
 
 +
10. Make sure that you have created your containers.  The lxc tools provide an '''lxc-ls''' command to list the created containers.
 +
 
 +
  $ sudo lxc-ls -f
  
 
You should see the containers you just created listed:
 
You should see the containers you just created listed:
  
  left  right
+
NAME      STATE  AUTOSTART GROUPS IPV4    IPV6
 +
  left     RUNNING 0        -      10.0.0.1 -   
 +
  right     RUNNING 0        -      10.0.0.2 -   
 +
 
  
12. To make life easier, start three terminal sessions and place one to the upper left of your screen (this will become the "left" container shell), one to the right of your screen (this will become the "right" container shell), and one at the bottom center of your screen (this will become the shell you use to start the ns-3 simulation).
+
11. To make life easier, start three terminal sessions and place one to the upper left of your screen (this will become the "left" container shell), one to the right of your screen (this will become the "right" container shell), and one at the bottom center of your screen (this will become the shell you use to start the ns-3 simulation).
  
13. Now, let's start the container for the "left" host.  Select the upper left shell window you created in step 11, take a deep breath and start your first container.
+
12. Now, let's attach to the container for the "left" host.  Select the upper left shell window you created in step 11, and type:
  
  $ sudo lxc-start -n left /bin/bash
+
  $ sudo lxc-attach -n left
  
 
At first it may appear that nothing has happened, but look closely at the prompt.  If all has gone well, it has changed.  The hostname is now "left" indicating your have a shell "inside" your newly created container.  If you used the prompt suggested above, you should now see
 
At first it may appear that nothing has happened, but look closely at the prompt.  If all has gone well, it has changed.  The hostname is now "left" indicating your have a shell "inside" your newly created container.  If you used the prompt suggested above, you should now see
  
  # root@left:~#
+
  [root@left /]#
  
 
You can verify that everything has gone well by looking at the network interface configuration.
 
You can verify that everything has gone well by looking at the network interface configuration.
  
  # ifconfig
+
  $ ip addr show dev eth0
  
 
You should see an "eth0" interface, with an IP address of 10.0.0.1, and a MAC address of 08:00:2e:00:00:01 that is up and running.
 
You should see an "eth0" interface, with an IP address of 10.0.0.1, and a MAC address of 08:00:2e:00:00:01 that is up and running.
  
   eth0     Link encap:Ethernet  HWaddr 08:00:2E:00:00:01 
+
   20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
            inet addr:10.0.0.1 Bcast:0.0.0.0  Mask:255.255.255.0
+
    link/ether ea:0f:69:8e:c6:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
            inet6 addr: fe80::a00:2eff:fe00:1/64 Scope:Link
+
    inet 10.0.0.1/24 brd 10.0.0.255 scope global eth0
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+
      valid_lft forever preferred_lft forever
            RX packets:4 errors:0 dropped:0 overruns:0 frame:0
+
    inet6 fe80::e80f:69ff:fe8e:c6e2/64 scope link
            TX packets:3 errors:0 dropped:0 overruns:0 carrier:0
+
      valid_lft forever preferred_lft forever
 +
 
 
             collisions:0 txqueuelen:1000  
 
             collisions:0 txqueuelen:1000  
 
             RX bytes:272 (272.0 b)  TX bytes:196 (196.0 b)
 
             RX bytes:272 (272.0 b)  TX bytes:196 (196.0 b)
  
14. Now, let's start the container for the "right" host.  Select the upper right shell window you created in step 11, take another deep breath and start your second container.
+
13. Repeat for the "right" host.  Select the upper right shell window you created in step 11, and type:
  
  $ sudo lxc-start -n right /bin/bash
+
  $ sudo lxc-attach -n right
  
 
Look closely at the prompt and notice it has changed.  The hostname is now "right" indicating your have a shell "inside" your newly created container.  If you used the prompt suggested above, you should now see
 
Look closely at the prompt and notice it has changed.  The hostname is now "right" indicating your have a shell "inside" your newly created container.  If you used the prompt suggested above, you should now see
  
  # root@right:~#
+
  [root@right /]#
  
 
You can verify that everything has gone well by looking at the network interface configuration.
 
You can verify that everything has gone well by looking at the network interface configuration.
  
  # ifconfig
+
  $ ip addr show dev eth0
 
+
You should see an "eth0" interface, with an IP address of 10.0.0.2, and a MAC address of 08:00:2e:00:00:02 that is up and running.
+
 
+
  eth0      Link encap:Ethernet  HWaddr 08:00:2E:00:00:02 
+
            inet addr:10.0.0.2  Bcast:0.0.0.0  Mask:255.255.255.0
+
            inet6 addr: fe80::a00:2eff:fe00:2/64 Scope:Link
+
            UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
+
            RX packets:4 errors:0 dropped:0 overruns:0 frame:0
+
            TX packets:4 errors:0 dropped:0 overruns:0 carrier:0
+
            collisions:0 txqueuelen:1000
+
            RX bytes:272 (272.0 b)  TX bytes:272 (272.0 b)
+
  
You are now done with the lxc configuration.  I had an old math professor that used to say, "I feel joy", in situations like this.  Feel free to celebrate with "high fives" all around.  The hard part is done.
 
  
 
== Running an ns-3 Simulated CSMA Network ==
 
== Running an ns-3 Simulated CSMA Network ==
  
We provide an ns-3 simulation script that is usable for different kinds of virtualization that end up with tap devices present in the system.  We did configure the system to have two tap devices, one called "tap-left" and one called "tap-right" and we are expecting ns-3 to provide a network between them.  In a spectacular coincidence, provided ns-3 simulation script implements a CSMA network between two tap devices called "tap-left" and "tap-right"!
+
We provide an ns-3 simulation script that is usable for different kinds of virtualization that end up with tap devices present in the system.  We did configure the system to have two tap devices, one called "tap-left" and one called "tap-right" and we are expecting ns-3 to provide a network between them.  The provided ns-3 simulation script implements a CSMA network between two tap devices called "tap-left" and "tap-right"!
  
 
Let's go ahead and wire in the network by running the ns-3 script.
 
Let's go ahead and wire in the network by running the ns-3 script.
  
15.  Select in the lower of the three terminal windows you created in step 11 and change into the ns-3 distribution you created back in step 1 (e.g., ~/repos/ns-3-allinone/ns-3-dev).  By default, waf will build the code required to create the tap devices, but will not try to set the suid bit.  This allows "normal" users to build the system without requiring root privileges.  You are going to have to enable building with this bit set.  Do the following:
+
14.  Select in the lower of the three terminal windows you created in step 11 and change into the ns-3 distribution you created back in step 1 (e.g., ~/repos/ns-3-allinone/ns-3-dev).  By default, waf will build the code required to create the tap devices, but will not try to set the suid bit.  This allows "normal" users to build the system without requiring root privileges.  You are going to have to enable building with this bit set.  Do the following:
  
 
  $ ./waf configure --enable-examples --enable-tests --enable-sudo
 
  $ ./waf configure --enable-examples --enable-tests --enable-sudo
Line 257: Line 238:
 
  $ ./waf --run tap-csma-virtual-machine
 
  $ ./waf --run tap-csma-virtual-machine
  
16. Ping from one container to the other.  Remember that you now have a ten minute clock running before your simulated network will be torn down.  Go to the left container and enter
+
15. Ping from one container to the other.  Remember that you now have a ten minute clock running before your simulated network will be torn down.  Go to the left container and enter
  
 
  # ping -c 4 10.0.0.2
 
  # ping -c 4 10.0.0.2
Line 265: Line 246:
 
== Running an ns-3 Simulated Ad-Hoc Wireless Network ==
 
== Running an ns-3 Simulated Ad-Hoc Wireless Network ==
  
17. If that's not enough, we can swap in an ad-hoc wireless network by simply running a different ns-3 script.  If the previous script has not stopped (it will run for ten minutes), go ahead and control-C out of it (in the lower window).  Then run the wireless ns-3 network script.
+
16. If that's not enough, we can swap in an ad-hoc wireless network by simply running a different ns-3 script.  If the previous script has not stopped (it will run for ten minutes), go ahead and control-C out of it (in the lower window).  Then run the wireless ns-3 network script.
  
 
  $ ./waf --run tap-wifi-virtual-machine
 
  $ ./waf --run tap-wifi-virtual-machine
  
18. Ping from one container to the other over a wireless ad-hoc network simulated on your computer.  Again, go to the left container and enter
+
17. Ping from one container to the other over a wireless ad-hoc network simulated on your computer.  Again, go to the left container and enter
  
 
  # ping -c 4 10.0.0.2
 
  # ping -c 4 10.0.0.2
Line 277: Line 258:
 
== Tearing it all Down ==
 
== Tearing it all Down ==
  
19. The first thing to do is to exit from the bash shell you have running in both containers.  These are the shells in the upper left and upper right windows you created in step 11.  Do the following in both windows.
+
18. The first thing to do is to exit from the bash shell you have running in both containers.  These are the shells in the upper left and upper right windows you created in step 11.  Do the following in both windows.
  
 
  # exit
 
  # exit
Line 283: Line 264:
 
You should see the prompts in those windows return to their normal state.
 
You should see the prompts in those windows return to their normal state.
  
20. Pick a window to work in and type enter the following to destroy the two Linux Containers you created.
+
19. Pick a window to work in and type enter the following to kill the two Linux Containers you created.
  
  $ sudo lxc-destroy -n left
+
  $ sudo lxc-stop -n left -k
  $ sudo lxc-destroy -n right
+
  $ sudo lxc-stop -n right -k
  
21. Take both of the bridges down
+
20. Take both of the bridges down
  
 
  $ sudo ifconfig br-left down
 
  $ sudo ifconfig br-left down
 
  $ sudo ifconfig br-right down
 
  $ sudo ifconfig br-right down
  
22. Remove the taps from the bridges
+
21. Remove the taps from the bridges
  
 
  $ sudo brctl delif br-left tap-left
 
  $ sudo brctl delif br-left tap-left
 
  $ sudo brctl delif br-right tap-right
 
  $ sudo brctl delif br-right tap-right
  
23. Destroy the bridges
+
22. Destroy the bridges
  
 
  $ sudo brctl delbr br-left
 
  $ sudo brctl delbr br-left
 
  $ sudo brctl delbr br-right
 
  $ sudo brctl delbr br-right
  
24. Bring down the taps
+
23. Bring down the taps
  
 
  $ sudo ifconfig tap-left down
 
  $ sudo ifconfig tap-left down
 
  $ sudo ifconfig tap-right down
 
  $ sudo ifconfig tap-right down
  
25. Delete the taps
+
24. Delete the taps
  
 
  $ sudo tunctl -d tap-left
 
  $ sudo tunctl -d tap-left
 
  $ sudo tunctl -d tap-right
 
  $ sudo tunctl -d tap-right
 +
 +
25.  Destroy the containers
 +
 +
$ sudo lxc-destroy -n right
 +
$ sudo lxc-destroy -n left
  
 
Now if you do an ifconfig, you should only see the devices you started with, and if you do an lxc-ls you should see no containers.
 
Now if you do an ifconfig, you should only see the devices you started with, and if you do an lxc-ls you should see no containers.
Line 318: Line 304:
 
[[User:Craigdo|Craigdo]] 01:19, 20 February 2010 (UTC)
 
[[User:Craigdo|Craigdo]] 01:19, 20 February 2010 (UTC)
  
Fixes and cleanups by [[User:Vedranm|Vedranm]] 10:32, 29 January 2013 (UTC)
+
Updated by [[User:Vedranm|Vedranm]] 29 January 2013 (UTC) and Tom Henderson, 2017.

Latest revision as of 06:08, 25 November 2017

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

One of the interesting ways ns-3 simulations can be integrated with real hardware is to create a simulated network and use "real" hosts to drive the network. Often, the number of real hosts is relatively large, and so acquiring that number of real computers would be prohibitively expensive. It is possible to use virtualization technology to create software implementations, called virtual machines, that execute software as if they were real hosts. If you have not done so, we recommend going over HOWTO make ns-3 interact with the real world to review your options.

This HOWTO discusses how one would configure and use a modern Linux system (e.g., Fedora 26+, Ubuntu 16.04+) to run ns-3 systems using paravirtualization of resources with Linux Containers (lxc). This HOWTO focuses on Fedora 26 sample configuration.

Note: A separate project focused on LXC and ns-3 integration has been announced in 2017 by Mike Albert.

Background

The first step for those unfamiliar with Linux Containers is to get an idea of what it is we are talking about and where they live. Linux Containers, also called "lxc tools," are an offshoot of what are called "chroot jails." A quick explanation of the FreeBSD version can be found here. It may be worth a quick read before continuing to help you understand some background and terminology better. Other useful documentation links:

Get and Understand the Linux Container Tools

The first step in the process is to get the Linux Container Tools onto your system. Although Fedora systems come with Linux Containers enabled, they do not necessarily have the user tools installed. You will need to install these tools, and also make sure you have the ability to create and configure bridges, and the ability to create and configure tap devices.

Fedora packages:

$ sudo dnf install lxc lxc-templates lxc-extra debootstrap libvirt perl gpg tunctl bridge-utils

Next, start some essential services:

$ sudo systemctl start libvirtd.service
$ sudo systemctl start lxc.service
$ sudo systemctl enable lxc.service

Sample output:

 Created symlink /etc/systemd/system/multi-user.target.wants/lxc.service ? /usr/lib/systemd/system/lxc.service.

To see a summary of the Linux Container tool (lxc), you can now begin looking at man pages. The place to start is with the lxc man page. Give that a read, keeping in mind that some details are not relevant at this time. Feel free to ignore the parts about kernel requirements and configuration and ssh daemons. So go ahead and do,

$ man lxc

and have a read.

The Big Picture

At this time, it will be helpful to take a look at "the big picture" of what it is we are going to accomplish. Let's take a look at an actual big picture.

Container and ns-3 Block Diagram.png

The dotted-dashed lines demark the containers and the host OS. You can see that in this example, we created two containers, located at the top of the picture. They are each running an application called "Your App." As far as these applications are concerned, they are running as if they were in their own Linux systems, each having their own network stack and talking to a net device named "eth0."

These net devices are actually connected to Linux bridges that form the connection to the host OS. There is a tap device also connected to each of these bridges. These tap devices bring the packets flowing through them into user space where ns-3 can get hold of them. A special ns-3 NetDevice attaches to the network tap and passes packets on to an ns-3 "ghost node."

The ns-3 ghost node acts as a proxy for the container in the ns-3 simulation. Packets that come in through the network tap are forwarded out the corresponding CSMA net device; and packets that come in through the CSMA net device are forwarded out the network tap. This gives the illusion to the container (and its application) that it is connected to the ns-3 CSMA network.

The following is a screenshot of where we're going with all of this. It shows a Windows XP machine running VirtualBox. VirtualBox is running a Fedora 12 virtual machine with the Guest Additions installed to give a big, beautiful display. There are three windows open. The top left window is running a bash shell in a Linux container that corresponds to the upper left container in the "big picture." The top right window is running a bash shell in a Linux container that corresonds to the upper right container in the "big picture." The lower window is running an ns-3 script that implements the virtual network tying the two containers together. You may be able to make out that each container has pinged the other.

Lxc-screenshot-2.PNG

HOWTO Use Linux Containers to set up virtual networks

With the previous background matierial in hand, developing the actual use-case should be almost self-explanatory.

1. Download and build ns-3. The following are roughly the steps used in the ns-3 tutorial. If you already have a current ns-3 distribution (version > ns-3.7) you don't have to bother with this step.

 $ cd
 $ mkdir repos
 $ cd repos
 $ hg clone http://code.nsnam.org/ns-3-allinone
 $ cd ns-3-allinone
 $ ./download.py
 $ ./build.py --enable-examples --enable-tests
 $ cd ns-3-dev
 $ ./test.py

After the above steps, you should have a cleanly built ns-3 distribution that passes all of the various unit and system tests.

2. Locate the configuration files for the two Linux Containers we will be creating -- one for the container at the upper left of the "big picture," and one at the upper right. We call the container on the upper left "left" and the container on the upper right "right." We have provided two configuration files in the src/tap-bridge/examples directory to use with this HOWTO. The paths below are relative to the distribution (e.g., ~/repos/ns-3-allinone/ns-3-dev).

$ less src/tap-bridge/examples/lxc-left.conf

You should see the following in the configuration file for the "left" container:

# Container with network virtualized using a pre-configured bridge named br0 and
# veth pair virtual network devices
lxc.utsname = left
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br-left
lxc.network.ipv4 = 10.0.0.1/24

Now take a look at the configuration file for the "right" container. You should see the following in the configuration file:

$ less src/tap-bridge/examples/lxc-right.conf
# Container with network virtualized using a pre-configured bridge named br0 and
# veth pair virtual network devices
lxc.utsname = right
lxc.network.type = veth
lxc.network.flags = up
lxc.network.link = br-right
lxc.network.ipv4 = 10.0.0.2/24

3. You must create the bridges we referenced in the conf files before creating the containers. These are going to be the "signal paths" to get packets in and out of the Linux containers. Getting pretty much all of the configuration for this HOWTO done requires root privileges, so we will assume you are able to su or sudo and know what that means.

$ sudo brctl addbr br-left
$ sudo brctl addbr br-right

4. You must create the tap devices that ns-3 will use to get packets from the bridges into its process. We are going to continue with the left and right naming convention. You will use these names when you get to the ns-3 simulation later.

$ sudo tunctl -t tap-left
$ sudo tunctl -t tap-right

The system will respond with "Set 'tap-left' persistent and owned by uid 0" which is normal and not an error message.

5. Before adding the tap devices to the bridges, you must set their IP addresses to 0.0.0.0 and bring them up.

$ sudo ifconfig tap-left 0.0.0.0 promisc up
$ sudo ifconfig tap-right 0.0.0.0 promisc up

6. Now you have got to add the tap devices you just created to their respective bridges, assign IP addresses to the bridges and bring them up.

$ sudo brctl addif br-left tap-left
$ sudo ifconfig br-left up
$ sudo brctl addif br-right tap-right
$ sudo ifconfig br-right up

7. Double-check that your have your bridges and taps configured correctly

$ sudo brctl show

This should show two bridges, br-left and br-right, with the tap devices (tap-left and tap-right) as interfaces. You should see something like:

bridge name    bridge id            STP enabled    interfaces
br-left        8000.3ef6a7f07182    no             tap-left
br-right       8000.a6915397f8cb    no             tap-right


8. Next, create a Fedora 26 container based on the 'left' configuration file found in 'src/tap-bridge/examples/':

 $ cd src/tap-bridge/examples
 $ sudo lxc-create -f lxc-left.conf -t download -n left -- -d fedora -r 26 -a amd64

You should see some output like this:

 Setting up the GPG keyring
 Downloading the image index
 Downloading the rootfs
 Downloading the metadata
 The image cache is now ready
 Unpacking the rootfs
 ---
 You just created a Fedora container (release=26, arch=amd64, variant=default)
 To enable sshd, run: dnf install openssh-server
 For security reason, container images ship without user accounts
 and without a root password.
 Use lxc-attach or chroot directly into the rootfs to set a root password
 or create user accounts.

Then do the following:

 $ sudo chroot /var/lib/lxc/left/rootfs/ passwd
 $ sudo lxc-start -n left

9. Repeat; create a Fedora 26 container based on the 'right' configuration file found in 'src/tap-bridge/examples/':

 $ cd src/tap-bridge/examples
 $ sudo lxc-create -f lxc-right.conf -t download -n right -- -d fedora -r 26 -a amd6
 $ sudo chroot /var/lib/lxc/right/rootfs/ passwd
 $ sudo lxc-start -n right

10. Make sure that you have created your containers. The lxc tools provide an lxc-ls command to list the created containers.

$ sudo lxc-ls -f

You should see the containers you just created listed:

NAME      STATE   AUTOSTART GROUPS IPV4     IPV6 
left      RUNNING 0         -      10.0.0.1 -    
right     RUNNING 0         -      10.0.0.2 -    


11. To make life easier, start three terminal sessions and place one to the upper left of your screen (this will become the "left" container shell), one to the right of your screen (this will become the "right" container shell), and one at the bottom center of your screen (this will become the shell you use to start the ns-3 simulation).

12. Now, let's attach to the container for the "left" host. Select the upper left shell window you created in step 11, and type:

$ sudo lxc-attach -n left

At first it may appear that nothing has happened, but look closely at the prompt. If all has gone well, it has changed. The hostname is now "left" indicating your have a shell "inside" your newly created container. If you used the prompt suggested above, you should now see

[root@left /]#

You can verify that everything has gone well by looking at the network interface configuration.

$ ip addr show dev eth0

You should see an "eth0" interface, with an IP address of 10.0.0.1, and a MAC address of 08:00:2e:00:00:01 that is up and running.

 20: eth0@if21: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
   link/ether ea:0f:69:8e:c6:e2 brd ff:ff:ff:ff:ff:ff link-netnsid 0
   inet 10.0.0.1/24 brd 10.0.0.255 scope global eth0
      valid_lft forever preferred_lft forever
   inet6 fe80::e80f:69ff:fe8e:c6e2/64 scope link 
      valid_lft forever preferred_lft forever
           collisions:0 txqueuelen:1000 
           RX bytes:272 (272.0 b)  TX bytes:196 (196.0 b)

13. Repeat for the "right" host. Select the upper right shell window you created in step 11, and type:

$ sudo lxc-attach -n right

Look closely at the prompt and notice it has changed. The hostname is now "right" indicating your have a shell "inside" your newly created container. If you used the prompt suggested above, you should now see

[root@right /]#

You can verify that everything has gone well by looking at the network interface configuration.

$ ip addr show dev eth0


Running an ns-3 Simulated CSMA Network

We provide an ns-3 simulation script that is usable for different kinds of virtualization that end up with tap devices present in the system. We did configure the system to have two tap devices, one called "tap-left" and one called "tap-right" and we are expecting ns-3 to provide a network between them. The provided ns-3 simulation script implements a CSMA network between two tap devices called "tap-left" and "tap-right"!

Let's go ahead and wire in the network by running the ns-3 script.

14. Select in the lower of the three terminal windows you created in step 11 and change into the ns-3 distribution you created back in step 1 (e.g., ~/repos/ns-3-allinone/ns-3-dev). By default, waf will build the code required to create the tap devices, but will not try to set the suid bit. This allows "normal" users to build the system without requiring root privileges. You are going to have to enable building with this bit set. Do the following:

$ ./waf configure --enable-examples --enable-tests --enable-sudo
$ ./waf build

You may now be asked for your password by sudo when you build in order to do the suid for the tap creator.

The ns-3 simulation uses the real-time simualtor and will run for ten minutes to give you time to play around in your new environment. Go ahead and run the simulation.

$ ./waf --run tap-csma-virtual-machine

15. Ping from one container to the other. Remember that you now have a ten minute clock running before your simulated network will be torn down. Go to the left container and enter

# ping -c 4 10.0.0.2

Watch the packets flow.

Running an ns-3 Simulated Ad-Hoc Wireless Network

16. If that's not enough, we can swap in an ad-hoc wireless network by simply running a different ns-3 script. If the previous script has not stopped (it will run for ten minutes), go ahead and control-C out of it (in the lower window). Then run the wireless ns-3 network script.

$ ./waf --run tap-wifi-virtual-machine

17. Ping from one container to the other over a wireless ad-hoc network simulated on your computer. Again, go to the left container and enter

# ping -c 4 10.0.0.2

Pretty cool.

Tearing it all Down

18. The first thing to do is to exit from the bash shell you have running in both containers. These are the shells in the upper left and upper right windows you created in step 11. Do the following in both windows.

# exit

You should see the prompts in those windows return to their normal state.

19. Pick a window to work in and type enter the following to kill the two Linux Containers you created.

$ sudo lxc-stop -n left -k
$ sudo lxc-stop -n right -k

20. Take both of the bridges down

$ sudo ifconfig br-left down
$ sudo ifconfig br-right down

21. Remove the taps from the bridges

$ sudo brctl delif br-left tap-left
$ sudo brctl delif br-right tap-right

22. Destroy the bridges

$ sudo brctl delbr br-left
$ sudo brctl delbr br-right

23. Bring down the taps

$ sudo ifconfig tap-left down
$ sudo ifconfig tap-right down

24. Delete the taps

$ sudo tunctl -d tap-left
$ sudo tunctl -d tap-right

25. Destroy the containers

$ sudo lxc-destroy -n right
$ sudo lxc-destroy -n left

Now if you do an ifconfig, you should only see the devices you started with, and if you do an lxc-ls you should see no containers.


Craigdo 01:19, 20 February 2010 (UTC)

Updated by Vedranm 29 January 2013 (UTC) and Tom Henderson, 2017.