Bugzilla – Bug 1562
Upgrade waf to 1.7.9+ and fix shipped wscripts
Last modified: 2014-05-05 04:47:02 EDT
We should upgrade waf to 1.7 series (eventually, at least), as we don't want to be stuck with unmaintained version. Waf 1.7 disables compatibility with 1.5 by default; this is a reason enough to clean up wscripts of it if possible. I will attach a patch.
Created attachment 1490 [details] Waf 1.7 upgrade patch Patch includes both ./waf and ./waf-1.7, so we can isolate potential waf bugs from bugs resulting from these changes. Good: * Replaces waf 1.5 API calls with 1.6+ API calls. * Removes custom pkgconfig.py utility and replaces it with waf check_cfg() calls. * Many minor cleanups all around. * Upgrades all included waf-tools. Bad: * For some reason, I managed to break scratch (seems easy to fix, but I haven't figured out how); it doesn't work with either 1.6 or 1.7. * Python bindings don't work with ./waf-1.7, and this could be a waf issue. I can't figure out where exactly does it break.
I suggest we merge this early in ns-3.17 cycle. In case issue with Python bindings is not that easy to fix, I would still propose merging wscript changes without upgrade to waf 1.7.
Alex, are you willing to take a look at this? We can discuss how to solve the Boost issue discussed on mailing list afterwards.
Sure. I'll take a look in the next couple of days.
I have a little bit coding and fixing. Now I'm able to compile everything, including python bindings. The only (at least I think that the only) problem is with apiscan. I couldn't figure out why the process is not starting... Here are my patches: Resolving compilation problem with bindings: https://github.com/cawka/ns-3-dev-ndnSIM/commit/81692e503573bb47c3280d7cfac5827b5536e596 Fix scratch: https://github.com/cawka/ns-3-dev-ndnSIM/commit/9043887de9d14ddc92abc2f4cee2400a4ff2601c Special handling of boost requirements by modules: https://github.com/cawka/ns-3-dev-ndnSIM/commit/473576bd09a7cf699408bcffa7621de665302238
Tom, do we have someone who is able and willing to look into apiscan issue?
Created attachment 1506 [details] Combined patch
(In reply to comment #7) > Created attachment 1506 [details] > Combined patch Thanks for this combined patch. I just tested on an Ubuntu 10.04-64 system set up for API scanning, and I found that './waf --apiscan=all' and './waf --apiscan=core' worked as expected (not sure what issue Alex faced). I did discover a problem, though: './waf --pyrun' seems to be broken (e.g. './waf --pyrun examples/tutorial/first.py' does not run anymore. I didn't look into this yet; help would be appreciated.
I tried pyrun (on OSX 10.8.2) and this worked ok (didn't work without specifying a a full path to python script): ./waf --pyrun src/core/examples/sample-rng-plot.py Also, I have retried --apiscan (with core and other modules) and it is working perfectly with the latest patch (finally, it finishes, not just hangs at the end of apiscan). There is one correction that I would like to have in wscript. Somehow, the current patch changes default directory for running scenarios to [./build] from current [./], and this change "broke" a couple of my scenarios... Here is the patch: diff --git a/wscript b/wscript index d48ae4b..a631a48 100644 --- a/wscript +++ b/wscript @@ -698,7 +698,7 @@ def build(bld): pass Runner.printout = null_printout - Options.cwd_launch = bld.path.get_bld().abspath() + Options.cwd_launch = bld.path.abspath() bld.create_ns3_program = types.MethodType(create_ns3_program, bld) bld.register_ns3_script = types.MethodType(register_ns3_script, bld) bld.create_suid_program = types.MethodType(create_suid_program, bld)
I can confirm pyrun issue on Debian wheezy.
Also, --enable-sudo seems to be broken.
I tried messing around with adding self.outputs = [] to SuidBuild_task, but it didn't help (it hangs indefinitely instead of returning an exception). I tried adding both @Task.cache_outputs and @Task.update_outputs and it didn't fix it on my side. Alex, could you please take a look?
For some reason I have the same issue with --enable-sudo on Waf 1.6.11 as well. But there is a difference: I'm getting this all the time with Waf 1.7.9 and only sometimes with older Waf. The message is: Build failed Traceback (most recent call last): File "/home/vedranm/workspace/ns-3-allinone/ns-3-dev/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Runner.py", line 162, in start st=tsk.runnable_status() File "/home/vedranm/workspace/ns-3-allinone/ns-3-dev/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Task.py", line 285, in runnable_status new_sig=self.signature() File "/home/vedranm/workspace/ns-3-allinone/ns-3-dev/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Task.py", line 274, in signature self.sig_implicit_deps() File "/home/vedranm/workspace/ns-3-allinone/ns-3-dev/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Task.py", line 377, in sig_implicit_deps self.are_implicit_nodes_ready() File "/home/vedranm/workspace/ns-3-allinone/ns-3-dev/.waf-1.6.11-30618c54883417962c38f5d395f83584/waflib/Task.py", line 407, in are_implicit_nodes_ready for x in tsk.outputs: AttributeError: 'SuidBuild_task' object has no attribute 'outputs' Can anyone test 1.7.9 on their machine? I'm using Debian wheezy 64-bit.
We should also fix create-module.py.
Alina found the bug with sudo, we should use Task.Task instead of Task.TaskBase as a base class for SuidBuild_task. I will post the updated patch soon.
Alina and I did some work on getting pyrun to work. It doesn't work yet, but it's probably doable. Also, I noticed that boost seems to break if you have no boost libraries at all. We should definitely fix that.
Created attachment 1541 [details] Fix for --pyrun bug
Created attachment 1542 [details] Fix for --enable-sudo bug
Created two patches: 1 - Patch "attachment 1541 [details]" is a modification to the previous 'combined patch' ("attachment 1506 [details]") that fixes the --pyrun issue. The problem was in the wscript, line 692 "Options.cwd_launch = bld.path.abspath()" should not be replaced by " Options.cwd_launch = bld.path.get_bld().abspath()" 2 - Patch "attachment 1542 [details]" has the fix we did with Vedran for the "SuidBuild_task" described in "comment 16". This patch should be applied after the other.
Created attachment 1544 [details] Fix for configure/build when boost is not installed There is a bug in waf's boost.py script. I have updated the script from the latest available waf (1.7.10), and it seem to solve the problem. The script actually got a little bit refactored (changed expected dir for --boost-includes and --boost-libs), so if we have mentioned boost-related options in the documentation (did we?), we may need to add minor corrections. I'm attaching the patch that basically replaces old waf-tools/boost.py with a new one. Btw. The original waf-tools/boost.py is with dos-style line breaks, so I kept this in the updated file as well.
OK, let me try to make the merged patch.
Created attachment 1545 [details] Rebase to ns-3-dev, merge all the patches, add waf 1.7.10, fix some more boost stuff Alina and Alex, thanks for your work. This is new combined patch, please test. Remains to be tested: system without boost (soon), FreeBSD (can't promise).
Alex, I'm afraid we haven't fixed the boost includes stuff yet. On a system with no boost headers I get: $ ./waf configure --enable-examples --enable-tests Setting top to : /home/vmiletic/workspace/ns-3-allinone/ns-3-dev-waf17 Setting out to : /home/vmiletic/workspace/ns-3-allinone/ns-3-dev-waf17/build Checking for 'gcc' (c compiler) : /usr/bin/gcc Checking for 'g++' (c++ compiler) : /usr/bin/g++ Checking for compilation flag -Wl,--soname=foo... support : ok Checking for program python : /usr/bin/python Checking for python version : (2, 6, 6, 'final', 0) Checking for library python2.6 in LIBDIR : not found Checking for library python2.6 in python_LIBPL : yes Checking for program /usr/bin/python-config,python2.6-config,python-config-2.6,python2.6m-config : not found Checking for header Python.h : :-( Asking python-config for pyembed --cflags flags : not found Checking boost includes : ---- Summary of optional NS-3 features: Python Bindings : not enabled (Python library or headers missing) BRITE Integration : not enabled (BRITE not enabled (see option --with-brite)) NS-3 Click Integration : not enabled (nsclick not enabled (see option --with-nsclick)) GtkConfigStore : not enabled (library 'gtk+-2.0 >= 2.12' not found) XmlIo : not enabled (library 'libxml-2.0 >= 2.7' not found) Threading Primitives : enabled Real Time Simulator : enabled Emulated Net Device : enabled File Descriptor Net Device : enabled Network Simulation Cradle : not enabled (NSC library liblinux2.6.26.so is missing: NSC has not been built?) MPI Support : not enabled (option --enable-mpi not selected) NS-3 OpenFlow Integration : not enabled (Required boost libraries not found) SQlite stats data output : not enabled (library 'sqlite3' not found) Tap Bridge : enabled PyViz visualizer : not enabled (Python Bindings are needed but not enabled) Use sudo to set suid bit : not enabled (option --enable-sudo not selected) Build tests : enabled Build examples : enabled GNU Scientific Library (GSL) : not enabled (GSL not found) 'configure' finished successfully (3.362s) It seems that if you do check_boost() try/except way, it prints out nothing and the rest of the configuration output is suppresed. I haven't checked deeply enough to see whether the configuration gets done or not. If you do it without try/except, it fails and the configuration doesn't continue. Perhaps we should bug boost guys to allow the check to be non-mandatory.
Created attachment 1546 [details] Fix boost on hosts without headers/libs Nailed boost using tricks Gustavo probably invented when he hit the same issue before.
Fixed. author Vedran Miletić <rivanvx@gmail.com> Mon, 01 Apr 2013 22:33:46 +0200 (3 minutes ago) changeset 9277 0f87d1cb030c parent 9276 1b5ccba50c6b child 9278 0a749c0f1afd author Alexander Afanasyev <alexander.afanasyev@ucla.edu> Mon, 01 Apr 2013 22:34:50 +0200 (2 minutes ago) changeset 9278 0a749c0f1afd parent 9277 0f87d1cb030c
Alex, did you get apiscan working at some point? It doesn't seem to work now, it doesn't even start.
Yes. I just checked with the latest code and I got the same problem that I had before - the scanning doesn't want to start. I checked against the old patch and scanning was working at that point. I didn't yet went investigating what and when went wrong. -- Btw. There is also another issue that I found related to boost libraries. I totally forgot that I made special changes to waf-tools/boost.py, which got erased after script update. I will attach the patch for that.
Created attachment 1557 [details] Fixing boost configuration (producing the list of supported modules)
Alex, if you take a look at the code, you will see that I resolved this stuff in a different way. Do you think that require_all keyword argument is still useful? Second, when we agree on what to include, I would like to upstream as much as possible of our boost.py changes.
Comment on attachment 1557 [details] Fixing boost configuration (producing the list of supported modules) not necessary
Sorry. I missed your change. It should work most of the time, but may break in some cases. For example, when static linkage is used, LIB_BOOST will be undefined and STLIB_BOOST will be defined. Though I'm not sure how many people actually statically build NS-3...
Forgot to mention. I don't really remember why I have added "require_all" keyword, so it is not really necessary. As for LIB_BOOST variable and "problems" with static compilation, I think it is ok, since all the previous code was relying on LIB_BOOST variable anyways. In other words, the way you implemented checking for missing libraries does not break anything that is currently working :)
Created attachment 1558 [details] Potential fix for apiscan failure This is not a final fix, since I do not fully understand what is going on and somebody should check if the "uid" function takes enough parameters to make it unique. But at least it made scanning process to start. Before applying this patch to the latest code, when I was running apiscan without any additional flags, the whole process was getting stuck and was going nowhere. However, when I specified -j1 option, I immediately got an exception that there is a missing uid() method in apiscan_task. After I added a simple implementation of uid() method (somebody should check its correctness), the scanning start running with and without specifying -j option.
Seems to work for me (apiscan fails for errors in gccxml since I don't have latest git, just what Debian packages). Tom, could you please try on your machine?
Mitch tested Alex's uid() patch, and it worked for us on the machine where we normally generate bindings. We still have the separate issue of cleanly exiting the apiscan task, but we can work around that by killing the process for now. Regarding whether that particular patch should be merged, we noticed that there is a uid() method defined in class Task: def uid(self): try: return self.uid_ except AttributeError: m=Utils.md5() up=m.update up(self.__class__.__name__) for x in self.inputs+self.outputs: up(x.abspath()) self.uid_=m.digest() return self.uid_ but not in TaskBase where class apiscan_task derives from. Alex's uid() is slightly different but seems to create a similar hash. So, I think these things should be tested before deciding on a specific merge: 1) what is the implication of making class apiscan_task inherit instead from Task, and using that uid()? 2) if TaskBase is needed for some other reason, should the above uid() method from class Task be instead copied to this class, instead of Alex's version?
*** Bug 1622 has been marked as a duplicate of this bug. ***
Tom, I believe that such approach would work (I haven't tried it, though), so I suggest to try it. Alina fixed the SuidBuild_task in exactly the same manner.
(In reply to comment #33) > Created attachment 1558 [details] > Potential fix for apiscan failure > > This is not a final fix, since I do not fully understand what is going on and > somebody should check if the "uid" function takes enough parameters to make it > unique. But at least it made scanning process to start. > > Before applying this patch to the latest code, when I was running apiscan > without any additional flags, the whole process was getting stuck and was going > nowhere. However, when I specified -j1 option, I immediately got an exception > that there is a missing uid() method in apiscan_task. After I added a simple > implementation of uid() method (somebody should check its correctness), the > scanning start running with and without specifying -j option. I think your uid() function should work as the tuple (self.__class__.__name__. self.curdirnode.abspath(), self.target) is unique for each apiscan_task instance. You could hash also the cflags, for completeness, but it works anyway. (In reply to comment #35) > Mitch tested Alex's uid() patch, and it worked for us on the machine where we > normally generate bindings. > We still have the separate issue of cleanly > exiting the apiscan task, but we can work around that by killing the process > for now. Sorry, no idea there, really. The same code should work as before. It might make sense to ask in the WAF list for this feature. > > Regarding whether that particular patch should be merged, we noticed that there > is a uid() method defined in class Task: > > def uid(self): > try: > return self.uid_ > except AttributeError: > m=Utils.md5() > up=m.update > up(self.__class__.__name__) > for x in self.inputs+self.outputs: > up(x.abspath()) > self.uid_=m.digest() > return self.uid_ > > > but not in TaskBase where class apiscan_task derives from. Alex's uid() is > slightly different but seems to create a similar hash. > > So, I think these things should be tested before deciding on a specific merge: > > 1) what is the implication of making class apiscan_task inherit instead from > Task, and using that uid()? To make this work we'd need to set the input files for the task, in this case the header file to be parsed. However, WAF might decide it doesn't need to run the task because, e.g. ns3/network-module.h did not change, even though a file included by ns3/network-module.h did change. Plus you need to have different uids for different targets (gcc-ILP32, etc.), which the above will not give you. To take the "target" into account you could declare an output file based on the target, I guess. But still, to me it sounds simpler to not let WAF decide whether or not to run the task, and have it always run when specified in the command-line. > > 2) if TaskBase is needed for some other reason, should the above uid() method > from class Task be instead copied to this class, instead of Alex's version? The reason why TaskBase is used instead of Task is that we do not know in advance what will be all the output files. Hm.. I am realizing now that the above statement may no longer be true. When we had monolithic bindings, we didn't know because we generated bindings for a number of modules, but now each task generates bindings for a single module and target, with well known output file names. I guess some of my comments above no longer apply. In any case, TaskBase should work. There are even examples with TaskBase in The WAF Book[1]. If only the WAF maintainer could keep the WAF APIs more stable... [1] http://docs.waf.googlecode.com/git/book_16/single.html
Here is what Gustavo did to try to fix the problem when Waf hangs: Mercurialns-3-dev / changeset Fix possible deadlock when 'waf --apiscan' finishes author Gustavo J. A. M. Carneiro <gjc@inescporto.pt> Thu, 12 Jul 2012 17:39:58 +0100 (9 months ago) bindings/python/wscript file | annotate | diff | revisions 1.1 --- a/bindings/python/wscript Thu Jul 12 16:31:42 2012 +0100 1.2 +++ b/bindings/python/wscript Thu Jul 12 17:39:58 2012 +0100 1.3 @@ -409,7 +409,6 @@ 1.4 # can't cope with it, so we have to force the user to restart 1.5 # WAF) 1.6 self.bld.producer.stop = 1 1.7 - self.bld.producer.free_task_pool() 1.8 return 0 1.9 1.10 I'm not sure why he did this because the function free_task_pool() still exists in Waf and did not need to be removed because it was no longer going to exist. I tried to put that line back in, but that did not fix the problem. That function used to be called by this class: class python_scan_task_collector(Task.TaskBase): """Tasks that waits for the python-scan-* tasks to complete and then signals WAF to exit """ after = 'apiscan' before = 'cxx' color = "BLUE" def __init__(self, curdirnode, env, bld): self.bld = bld super(python_scan_task_collector, self).__init__(generator=self) self.curdirnode = curdirnode self.env = env def display(self): return 'python-scan-collector\n' def run(self): # signal stop (we generated files into the source dir and WAF # can't cope with it, so we have to force the user to restart # WAF) self.bld.producer.stop = 1 return 0 I will look at this class more carefully to try to understand how it works and how it could be affected by migrating to the latest version of Waf.
I ran the version of ns-3 before the waf updates were made: Improve logging of InetTopologyReader author Vedran Miletić <rivanvx@gmail.com> Mon, 01 Apr 2013 21:56:36 +0200 (9 days ago) changeset 9276 1b5ccba50c6b and it hung at the end of the Python bindings scan. So, I think the problem was pre-existing, i.e. it was not introduced by the waf update. I don't think we will have time to isolate the problem before the release.
On my Mac (as well on my Ubuntu linux installation), I almost always had a problem of "waf --apiscan" hanging up after a successful binding rescanning. So, it is definitely not the problem of waf upgrade.
(In reply to comment #38) > (In reply to comment #33) > > Created attachment 1558 [details] > > Potential fix for apiscan failure > > > > This is not a final fix, since I do not fully understand what is going on and > > somebody should check if the "uid" function takes enough parameters to make it > > unique. But at least it made scanning process to start. > > > > Before applying this patch to the latest code, when I was running apiscan > > without any additional flags, the whole process was getting stuck and was going > > nowhere. However, when I specified -j1 option, I immediately got an exception > > that there is a missing uid() method in apiscan_task. After I added a simple > > implementation of uid() method (somebody should check its correctness), the > > scanning start running with and without specifying -j option. > > I think your uid() function should work as the tuple (self.__class__.__name__. > self.curdirnode.abspath(), self.target) is unique for each apiscan_task > instance. You could hash also the cflags, for completeness, but it works > anyway. I'll plan to push Alex's patch in the next day or so, and close this bug, unless there are further comments. The apiscan failure to terminate cleanly is a separate pre-existing issue that I'll probably reopen bug 1622 for since there is some discussion context already there.
Yes, apiscan hanging in the end was something that used to happen to me occasionally, and I never was able to fix it. Sorry about that.
*** Bug 1381 has been marked as a duplicate of this bug. ***
Shall we close this bug ? We're running waf 1.7.13...
Sure. A new bug should be reported if there are problems with current build system.