HOWTO use distcc and ccache

From Nsnam
Revision as of 16:40, 14 November 2018 by Natale (Talk | contribs)

Jump to: navigation, search

How to use distcc and ccache for speed up building time

Ns-3 has a lot of files to compile. With the expected growth of different branches and repositories, time spent in compiling is affecting the development time in a considerable way. At the end of this HOWTO, you will be able to store the intermediate result of a compilation (the object files) and to distribute new compilation unit (i.e., when the cached object file does not exist or it has to be updated). You could also selectively disable one (or both) options just by switching an environment variable.

Linux: setup distcc

distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network to speed up the building time. It should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile. Use your favorite package manager to install the distcc package.

In the following, we will use the term master to indicate the computer which initiates the compilation and which contains the source copy of ns-3, while we will use the term slave to indicate a computer which accepts compilation requests sent by the master. First of all, in the slaves, follow your distribution guide to setup a working distcc server. For instance, for Archlinux (that is the most easy distribution you can find out there -- forget all the complexity of Ubuntu, multiple package versions, libraries, -dev, and everything that could make you hate the Linux ecosystem) you can refer to [1]. Basically, the /etc/conf.d/distcc should look like:

# Parameters to be passed to distccd
# You must explicitly add IPs (or subnets) that are allowed to connect,
# using the --allow switch.  See the distccd manpage for more info.

DISTCC_ARGS="--allow MASTER_IP --log-level debug --log-file /tmp/distccd.log"

Then, in the master, install the distcc package without running the server. Then, configure it (following your distribution howto) to have the slave ip in the DISTCC_HOST configuration variable. For instance, for Archlinux, this is easy as having a file /etc/distcc/host with the following content:

# --- /etc/distcc/hosts -----------------------
# See the "Hosts Specification" section of
# "man distcc" for the format of this file.
# By default, just test that it works in loopback mode.

Just replace the SLAVE_IP with the IP address of the slave. Please note that this configuration is insecure because sources and objects are traveling the network with a simple, unencrypted, TCP connection. Please set up a more advanced environment by using SSH if you have to cross the Internet or another untrusted network.

In the master computer, you have to specify now to use pump distcc as your compiler. The most obvious way to use it for ns-3 is the following:

$ CXX="pump distcc g++" ./waf --your-options

But before doing so, let test if distcc generally works for you by creating a file named "" that contains the following:

int main ()
  return 0;

In the slave let's see what is happening by doing

$ tail -f /tmp/distccd.log

and then running the following:

$ pump distcc gcc -c test.o

You should see that the slave receives the compilation and compile successfully the file. The most important line is at the end:

distccd[686] (dcc_check_client) connection from
distccd[686] (check_address_inet) match client 0x3410010a, value 0x3410010a, mask 0xffffffff
distccd[686] (dcc_r_token_int) got DIST00000002
distccd[686] (dcc_r_token_int) got ARGC00000005
distccd[686] job complete
distccd[686] (dcc_cleanup_tempfiles_inner) deleted 5 temporary files
distccd[686] (dcc_job_summary) client: COMPILE_OK exit:0 sig:0 core:0 ret:0 time:61ms gcc asd.c

If you see COMPILE_OK, then you are good: distcc and distccd are properly configured, and you can try to use it with ns-3:

$ CXX="pump distcc g++" ./waf --your-options
$ ./waf -j8042 # 8042 concurrent processes is my dream

Linux: Setup ccache =