I remember that WAF did not have any API you can call to "exit WAF right now", which we need to do[1] at the end of --apiscan.  I did some dirty hacks, changing some waf inner data structures, to try to cause it to exit, and is not surprising that my hacks stopped working.

[1] the reason is that to keep WAF going after apiscan we would need to tell it beforehand which files a task creates or modifies, which is very complicated to accomplish in the case of --apiscan.

(In reply to comment #1)
> I remember that WAF did not have any API you can call to "exit WAF right now",
> which we need to do[1] at the end of --apiscan.  I did some dirty hacks,
> changing some waf inner data structures, to try to cause it to exit, and is not
> surprising that my hacks stopped working.
> 
> [1] the reason is that to keep WAF going after apiscan we would need to tell it
> beforehand which files a task creates or modifies, which is very complicated to
> accomplish in the case of --apiscan.

In this case, it is hanging before performing the scan.  

When I roll back to the version before waf was upgraded (1b5ccba50c6b) and I ask for an apiscan with the "-vvv" option, this kind of debugging output is generated:

08:14:17 envhash d751713988987e9331980363e24189ce []
08:14:17 envhash d751713988987e9331980363e24189ce []
08:14:17 envhash d751713988987e9331980363e24189ce []
08:14:17 envhash d751713988987e9331980363e24189ce []
api-scan-gcc_ILP32
api-scan-gcc_LP64
python-scan-collector
/usr/local/lib/python2.6/dist-packages/pygccxml/parser/declarations_cache.py:8: DeprecationWarning: the md5 module is deprecated; use hashlib instead
  import md5
...

However, same command for the tip of ns-3-dev will cause waf to hang after these statements:

08:17:39 envhash d751713988987e9331980363e24189ce []
08:17:39 envhash d751713988987e9331980363e24189ce []
08:17:39 envhash d751713988987e9331980363e24189ce []

i.e. the display() method in bindings/python/wscript:
    def display(self):
        return 'api-scan-%s\n' % (self.target,)

is not invoked, seeming to imply that the task is not launched and perhaps there is some deadlock in task dispatching

(In reply to comment #1)
> I remember that WAF did not have any API you can call to "exit WAF right now",
> which we need to do[1] at the end of --apiscan.  I did some dirty hacks,
> changing some waf inner data structures, to try to cause it to exit, and is not
> surprising that my hacks stopped working.
> 
> [1] the reason is that to keep WAF going after apiscan we would need to tell it
> beforehand which files a task creates or modifies, which is very complicated to
> accomplish in the case of --apiscan.

In light of the above, would it be easier to maintain, in the long run, a separate program to perform apiscan outside of waf?  e.g.

utils/apiscan.py --all
utils/apiscan.py --modules=lte

(In reply to comment #3)
> (In reply to comment #1)
> In light of the above, would it be easier to maintain, in the long run, a
> separate program to perform apiscan outside of waf?  e.g.
> 
> utils/apiscan.py --all
> utils/apiscan.py --modules=lte

One might think so, but then the apiscan.py script needs to know a bunch of things that is in the waf scripts in order to do its job, starting with with ns3 modules that exist, their dependencies (dependencies are very important), and compilation flags needed (gccxml is a real compiler, needs all the proper include paths set).

I tried to determine exactly which changeset introduced the problem, and
I was able to narrow it down to the changeset that contains the new version of waf or the one right after that.

The version before waf was upgraded does the binding scans fine:

    Improve logging of InetTopologyReader
    author	Vedran Miletić <rivanvx@gmail.com>
    	Mon, 01 Apr 2013 21:56:36 +0200 (9 days ago)
    changeset 9276	1b5ccba50c6b

The next change set, which upgraded waf, won't build for me:

    Upgrade waf to 1.7.10 and fix included wscripts
    author	Vedran Miletić <rivanvx@gmail.com>
    	Mon, 01 Apr 2013 22:33:46 +0200 (9 days ago)
    changeset 9277	0f87d1cb030c

The change set right after that builds but hangs during bindings scans:

    Fix Python bindings and openflow for waf 1.7.10 upgrade and 
          take care about boost requirements globally
    author	Alexander Afanasyev <alexander.afanasyev@ucla.edu>
    	Mon, 01 Apr 2013 22:34:50 +0200 (9 days ago)
    changeset 9278	0a749c0f1afd

*** This bug has been marked as a duplicate of bug 1562 ***

moving priority to P2 critical

--apiscan should no longer hang.  Instead I gave up in trying to make the WAF build exit early, and the build simply continues after the scanning.  At least it's less annoying.

(In reply to comment #8)
> --apiscan should no longer hang.  Instead I gave up in trying to make the WAF
> build exit early, and the build simply continues after the scanning.  At least
> it's less annoying.

I found a small problem with the current solution.  apiscan_task.run() will only run if there have been changes to the ns-3 source or wscripts.  It seems that perhaps this is because there is some check elsewhere in the system that suppresses build tasks if there are no code changes.  Can this be tweaked so that apiscan_task.run() is always called if --apiscan is passed as a waf argument?

To reproduce:  './waf --apiscan=all' will either scan zero or one modules.

Created attachment 1705 [details]
patch to fix

This works for me and will add to ns-3.18 repository for now; OK to commit to ns-3-dev?

(In reply to Tom Henderson from comment #10)
> Created attachment 1705 [details]
> patch to fix
> 
> This works for me and will add to ns-3.18 repository for now; OK to commit
> to ns-3-dev?

Looks great.

this patch was applied in 10398:458687e0055d