Bugzilla – Bug 2451
replacement of gccxml with castxml
Last modified: 2018-07-21 19:33:58 EDT
gccxml is no longer maintained, having been replaced by castxml. For the moment, until castxml is tested and supported in our Python bindings toolchain, we can still use gccxml. However, gccxml does not build on Ubuntu 16.04. One patch and one build flag are needed: Patch: http://pkgs.fedoraproject.org/cgit/rpms/gccxml.git/tree/gccxml-gcc5.patch?h=f22 build flag: -DCMAKE_C_FLAGS=-fgnu89-inline as in: cmake ../gccxml -DCMAKE_INSTALL_PREFIX:PATH=../../build -DCMAKE_C_FLAGS=-fgnu89-inline These mods must be made manually until future revision of bakeconf.xml automates them: - download gccxml from git - patch with the above patch from Fedora project - follow the build instructions in the gccxml README, but modify the 'cmake' statement as above
Ubuntu being pretty popular, it should be cheaper to fix this before the release than providing support afterwards. https://groups.google.com/forum/#!topic/ns-3-users/2xwbyrQcQCc I've used castxml for dce and updating ns3 to use castxml instead of gccxml should be pretty straightforward (pretty much drop in).
(In reply to Matthieu Coudron from comment #1) > Ubuntu being pretty popular, it should be cheaper to fix this before the > release than providing support afterwards. > https://groups.google.com/forum/#!topic/ns-3-users/2xwbyrQcQCc > > I've used castxml for dce and updating ns3 to use castxml instead of gccxml > should be pretty straightforward (pretty much drop in). Do you have any patch to ns-3 wscript for castxml? I gave castxml a try. It has dependencies on LLVM/clang. It doesn't appear to have packages for Ubuntu earlier than 16.04. I gave this a try also: https://github.com/thewtex/CastXMLSuperbuild but it is taking all day to build. It seems to me that the most direct way to fix it for the immediate future may be to provide a patched gccxml for now (we could provide from one of our repositories). Once system packages for our supported distributions are available, we could consider to move this to a system_install requirement and cut over.
(In reply to Tom Henderson from comment #2) > (In reply to Matthieu Coudron from comment #1) > > Ubuntu being pretty popular, it should be cheaper to fix this before the > > release than providing support afterwards. > > https://groups.google.com/forum/#!topic/ns-3-users/2xwbyrQcQCc > > > > I've used castxml for dce and updating ns3 to use castxml instead of gccxml > > should be pretty straightforward (pretty much drop in). > > Do you have any patch to ns-3 wscript for castxml? > > I gave castxml a try. It has dependencies on LLVM/clang. It doesn't appear > to have packages for Ubuntu earlier than 16.04. > > I gave this a try also: > > https://github.com/thewtex/CastXMLSuperbuild > > but it is taking all day to build. > > It seems to me that the most direct way to fix it for the immediate future > may be to provide a patched gccxml for now (we could provide from one of our > repositories). Once system packages for our supported distributions are > available, we could consider to move this to a system_install requirement > and cut over. I tried to fix this for the ns-3.26 release today. I was able to instrument bakeconf.xml to build gccxml for gcc 5 and 6: diff -r b2d9a273011e -r 668b9be8c476 bakeconf.xml --- a/bakeconf.xml Tue Sep 27 21:07:38 2016 -0700 +++ b/bakeconf.xml Mon Oct 03 10:59:25 2016 -0700 @@ -667,9 +667,12 @@ <module name="gccxml-ns3"> <source type="git"> <attribute name="url" value="git://github.com/gccxml/gccxml.git"/> + <attribute name="post_download" value="curl http://pkgs.fedoraproject.org/cgit/rpms/gccxml.git/plain/gccxml-gcc5.patch?h=f22 > $SRCDIR/gccxml-gcc5.patch"/> <attribute name="module_directory" value="gccxml"/> </source> <build type="cmake" objdir="yes"> + <attribute name="patch" value="$SRCDIR/gccxml-gcc5.patch"/> + <attribute name="cmake_arguments" value="-DCMAKE_C_FLAGS=-fgnu89-inline"/> </build> </module> However, this unfortunately is not sufficient, as there are gccxml runtime errors due to lack of support for gcc more recent than version 4.9. The run time error (observed when running ./waf --apiscan) is: pygccxml.parser.source_reader.gccxml_runtime_error_t: Error occured while running GCC-XML: gccxml_cc1plus: error: gccxml_builtins.h: No such file or directory The reason is that gccxml_builtins.h is version-specific header, and support for this header ended at gcc-4.9. So, I think the current state is to post the ns-3.26 release with python scanning support limited to gcc-4, and expedite a transition to newer pybindgen that uses castxml; ns-3.26 could perhaps have a maintenance release when castxml is supported.
Ankit Deepak is working on the transition to CastXML. For Ubuntu 16.04 users, the workaround is to install gcc/g++ 4.9 and use that to build gccxml.
We also have a new limitation with gcc-xml; it does not support C++-11 syntax: from /home/buildslave/scratch/bake/source/ns-3-dev/src/network/bindings/scan-header.h:3: /home/buildslave/scratch/bake/source/ns-3-dev/build/ns3/net-device-queue-interface.h: In member function 'void ns3::NetDeviceQueueInterface::ConnectQueueTraces(ns3::Ptr<ns3::Queue<Item> >, uint8_t)': /home/buildslave/scratch/bake/source/ns-3-dev/build/ns3/net-device-queue-interface.h:360: error: 'class std::map<ns3::Ptr<ns3::QueueBase>, std::vector<ns3::CallbackBase, std::allocator<ns3::CallbackBase> >, std::less<ns3::Ptr<ns3::QueueBase> >, std::allocator<std::pair<const ns3::Ptr<ns3::QueueBase>, std::vector<ns3::CallbackBase, std::allocator<ns3::CallbackBase> > > > >' has no member named 'emplace' /home/buildslave/scratch/bake/source/ns-3-dev/build/ns3/net-device-queue-interface.h:360: error: 'initializer_list' is not a member of 'std' We cannot presently scan the network module as a result.
Here is a brief update on status (Ankit Deepak is working this bug). In the ns-3-allinone directory generated by bake, gccxml must be replaced by CastXML, and pygccxml should be replaced by recent version. clang must also be supported. The recent testing has been of pygccxml v1.9.0, castxml 0.1-gfab9c47 and clang version 3.8.1, on ubuntu 16.04. 0) Install development version of clang. It can be done from source but is lengthy, so consider using the binary packages. 1) fetch CastXML from its git repository, and build: Note: I had to follow this recipe on Ubuntu 16.04 to build it against the version of clang installed by Ubuntu: https://github.com/iovisor/bcc/issues/492 2) install pygccxml-1.9 (or current tip of repository) to your python path (I installed to /usr/local/lib). Check that you can import 'pygccxml' from python. 3) provide castxmlparser.py to pybindgen (attached to this issue) 4) patch ns-3-dev with two patches attached to this issue: - bindings.diff - remove-anonymous-typedefs.patch This gets us most of the way there (to running ./waf --apiscan successfully), but there are lingering issues with how typedefs for ns-3 abstract classes are resolved, and handling of typedeffed types in the global namespace such as 'uint##_t'
Created attachment 2881 [details] ns-3 patch to remove anonymous typedefs
Created attachment 2882 [details] parser for pybindgen
Created attachment 2883 [details] patch to ns-3-dev bindings code
Update: the ns-3 changes to support CastXML can be found in this repository: https://github.com/adeepkit01/castxml-changes The pybindgen changes are found in this repository: https://github.com/adeepkit01/pybindgen Steps to complete: 1) merge changes to ns-3-dev (outside of the 'temporary fix for wifi queue' commit, for which we seek a better solution) 2) work with Gustavo to finalize the castxmlparser.xml file for pybindgen 3) move away from separate 32-bit and 64-bit bindings to a single bindings file 4) prepare bake changes to install prerequisite packages, and provide new documentation 5) test on MacOS and Linux 32- and 64-bit systems
MacOS: In file included from src/network/bindings/ns3module.cc:1: src/network/bindings/ns3module.h:2355:23: error: no member named 'DoNsLog' in 'ns3::QueueBase' { ns3::QueueBase::DoNsLog(level, str); } ~~~~~~~~~~~~~~~~^ 1 error generated. Can we sync with the latest ns-3-dev ? I think that this error was fixed some time ago (In reply to Tom Henderson from comment #10) > Update: the ns-3 changes to support CastXML can be found in this repository: > > https://github.com/adeepkit01/castxml-changes > > The pybindgen changes are found in this repository: > > https://github.com/adeepkit01/pybindgen > > Steps to complete: > > 1) merge changes to ns-3-dev (outside of the 'temporary fix for wifi queue' > commit, for which we seek a better solution) > > 2) work with Gustavo to finalize the castxmlparser.xml file for pybindgen > > 3) move away from separate 32-bit and 64-bit bindings to a single bindings > file > > 4) prepare bake changes to install prerequisite packages, and provide new > documentation > > 5) test on MacOS and Linux 32- and 64-bit systems
(In reply to Tommaso Pecorella from comment #11) > MacOS: > > In file included from src/network/bindings/ns3module.cc:1: > src/network/bindings/ns3module.h:2355:23: error: no member named 'DoNsLog' > in 'ns3::QueueBase' > { ns3::QueueBase::DoNsLog(level, str); } > ~~~~~~~~~~~~~~~~^ > 1 error generated. > > Can we sync with the latest ns-3-dev ? > I think that this error was fixed some time ago There is a separate issue that MacOS is deciding that it is a 32-bit system (has been for some time), and the CastXML bindings update that I checked in is for 64-bit only because Ankit's current framework doesn't support 32-bit scans. If you do a './waf configure' on MacOS, it reports: Checking for the apidefs that can be used for Python bindings : gcc-ILP32 This is because this test program compiles on MacOS: #include <stdint.h> #include <vector> int main () { std::vector< uint64_t > t = std::vector< unsigned long long > (); return 0; } i.e. uint64_t is equivalent to unsigned long long on MacOS, while it is equivalent to unsigned long on Linux 64-bit machines. So waf concludes that MacOS is 32-bit and uses the 32-bit bindings, which have not been updated. We are trying to move away from this distinction, and support a single bindings file (modulegen__clang_LP64.py). The problem of handling uint64_t types should not arise in the CastXML framework (we hope) because, unlike the gccxml scanning process, uint64_t types should not be resolved to 'unsigned long' or 'unsigned long long'. But as of this morning, we don't have tested fixes for MacOS in place in ns-3-dev; only Linux 64-bit will compile (sorry...).
I am on ArchLinux 64 bit with: - g++ 7.1.1 - clang 4.0.1 - castxml, pygccxml from git master - pybindgen from Ankit's github repo - python-cxxfilt (new requirement?) from github repo - commits 723becb and c544761 applied on top of current ns-3-dev [stefano@deauville ns-3-dev-git]$ python2 ./waf configure --enable-tests --enable-examples Checking for 'gcc' (C compiler) : /usr/bin/gcc Checking for cc version : 7.1.1 Checking for python version : (2, 7, 13, 'final', 0) Checking for python module 'pybindgen' : 0.18.0.post3+g3eade76 Checking for pybindgen version : 0.18.0.post3+g3eade76 Checking for types uint64_t and unsigned long equivalence : no Checking for types uint64_t and unsigned long long equivalence : yes Checking for the apidefs that can be used for Python bindings : gcc-LP64 Checking for python module 'pygccxml' : 1.9.0 Checking for pygccxml version : 1.9.0 Checking for program 'castxml' : /usr/bin/castxml LLVM ERROR: IO failure on output stream. Checking high precision implementation : 128-bit integer (default) Then, python2 ./waf --apiscan=all terminates with some errors: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/antenna-module.h:10: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/angles.h:25: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/vector.h:23: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/attribute.h:23: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/string:41: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/bits/allocator.h:46: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/x86_64-pc-linux-gnu/bits/c++allocator.h:33: In file included from /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/ext/new_allocator.h:33: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:120:7: error: 'operator new' takes type size_t ('unsigned int') as first parameter void* operator new(std::size_t) _GLIBCXX_THROW (std::bad_alloc) ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:122:7: error: 'operator new[]' takes type size_t ('unsigned int') as first parameter void* operator new[](std::size_t) _GLIBCXX_THROW (std::bad_alloc) ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:134:7: error: 'operator new' takes type size_t ('unsigned int') as first parameter void* operator new(std::size_t, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:136:7: error: 'operator new[]' takes type size_t ('unsigned int') as first parameter void* operator new[](std::size_t, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:168:14: error: 'operator new' takes type size_t ('unsigned int') as first parameter inline void* operator new(std::size_t, void* __p) _GLIBCXX_USE_NOEXCEPT ^ /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1.1/new:170:14: error: 'operator new[]' takes type size_t ('unsigned int') as first parameter inline void* operator new[](std::size_t, void* __p) _GLIBCXX_USE_NOEXCEPT and In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-module.h:10: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-dpd.h:26: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-id-cache.h:33: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/simulator.h:27: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/nstime.h:26: In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64.h:30: /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:29:9: error: unknown type name '__uint128_t' typedef __uint128_t uint128_t; ^ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:30:9: error: unknown type name '__int128_t' typedef __int128_t int128_t; ^ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:48:62: warning: shift count >= width of type [-Wshift-count-overflow] static const uint128_t HP128_MASK_HI_BIT = (((int128_t)1)<<127); ^ ~~~ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:50:41: warning: implicit conversion from 'unsigned long long' to 'const uint64_t' (aka 'const unsigned long') changes value from 18446744073709551615 to 4294967295 [-Wconstant-conversion] static const uint64_t HP_MASK_LO = 0xffffffffffffffffULL; ~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:336:10: error: constructor cannot be redeclared inline int64x64_t (const int128_t v) ^ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:135:10: note: previous declaration is here inline int64x64_t (const int v) ^ /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:123:13: warning: shift count >= width of type [-Wshift-count-overflow] _v = hi << 64; ^ ~~ Did you ever encounter such errors?
(In reply to Stefano Avallone from comment #13) > I am on ArchLinux 64 bit with: > > - g++ 7.1.1 > - clang 4.0.1 > - castxml, pygccxml from git master > - pybindgen from Ankit's github repo > - python-cxxfilt (new requirement?) from github repo > - commits 723becb and c544761 applied on top of current ns-3-dev > > [stefano@deauville ns-3-dev-git]$ python2 ./waf configure --enable-tests > --enable-examples > Checking for 'gcc' (C compiler) : /usr/bin/gcc > Checking for cc version : 7.1.1 > Checking for python version : (2, 7, 13, 'final', > 0) > Checking for python module 'pybindgen' : > 0.18.0.post3+g3eade76 > Checking for pybindgen version : > 0.18.0.post3+g3eade76 > Checking for types uint64_t and unsigned long equivalence : no > Checking for types uint64_t and unsigned long long equivalence : yes > Checking for the apidefs that can be used for Python bindings : > gcc-LP64 > Checking for python module 'pygccxml' : 1.9.0 > Checking for pygccxml version : 1.9.0 > Checking for program 'castxml' : > /usr/bin/castxml > LLVM ERROR: IO failure on output stream. > Checking high precision implementation : 128-bit > integer (default) > > Then, python2 ./waf --apiscan=all terminates with some errors: > > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/antenna-module.h:10: > In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/angles.h:25: > In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/vector.h:23: > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/attribute.h:23: > In file included from > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/string:41: > In file included from > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/bits/allocator.h:46: > In file included from > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/x86_64-pc-linux-gnu/bits/c++allocator.h:33: > In file included from > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/ext/new_allocator.h:33: > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:120:7: error: 'operator new' takes type size_t ('unsigned int') as > first parameter > void* operator new(std::size_t) _GLIBCXX_THROW (std::bad_alloc) > ^ > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:122:7: error: 'operator new[]' takes type size_t ('unsigned int') as > first parameter > void* operator new[](std::size_t) _GLIBCXX_THROW (std::bad_alloc) > ^ > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:134:7: error: 'operator new' takes type size_t ('unsigned int') as > first parameter > void* operator new(std::size_t, const std::nothrow_t&) _GLIBCXX_USE_NOEXCEPT > ^ > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:136:7: error: 'operator new[]' takes type size_t ('unsigned int') as > first parameter > void* operator new[](std::size_t, const std::nothrow_t&) > _GLIBCXX_USE_NOEXCEPT > ^ > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:168:14: error: 'operator new' takes type size_t ('unsigned int') as > first parameter > inline void* operator new(std::size_t, void* __p) _GLIBCXX_USE_NOEXCEPT > ^ > /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/7.1.1/../../../../include/c++/7.1. > 1/new:170:14: error: 'operator new[]' takes type size_t ('unsigned int') as > first parameter > inline void* operator new[](std::size_t, void* __p) _GLIBCXX_USE_NOEXCEPT > > > and > > > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-module.h:10: > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-dpd.h:26: > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/aodv-id-cache.h:33: > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/simulator.h:27: > In file included from /home/stefano/Tools/ns-3-dev-git/build/ns3/nstime.h:26: > In file included from > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64.h:30: > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:29:9: error: > unknown type name '__uint128_t' > typedef __uint128_t uint128_t; > ^ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:30:9: error: > unknown type name '__int128_t' > typedef __int128_t int128_t; > ^ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:48:62: warning: > shift count >= width of type [-Wshift-count-overflow] > static const uint128_t HP128_MASK_HI_BIT = (((int128_t)1)<<127); > ^ ~~~ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:50:41: warning: > implicit conversion from 'unsigned long long' to 'const uint64_t' (aka > 'const unsigned long') changes value from > 18446744073709551615 to 4294967295 [-Wconstant-conversion] > static const uint64_t HP_MASK_LO = 0xffffffffffffffffULL; > ~~~~~~~~~~ ^~~~~~~~~~~~~~~~~~~~~ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:336:10: error: > constructor cannot be redeclared > inline int64x64_t (const int128_t v) > ^ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:135:10: note: > previous declaration is here > inline int64x64_t (const int v) > ^ > /home/stefano/Tools/ns-3-dev-git/build/ns3/int64x64-128.h:123:13: warning: > shift count >= width of type [-Wshift-count-overflow] > _v = hi << 64; > ^ ~~ > > Did you ever encounter such errors? This is the error from 32 bit bindings, add "--no32bit-scan" to the apiscan command. By default it still tries to scan both 32 and 64 bits
> This is the error from 32 bit bindings, add "--no32bit-scan" to the apiscan > command. By default it still tries to scan both 32 and 64 bits Ahhh, thanks! I finally managed to rescan all the python bindings (64-bit only). And I managed to fix the issue with wifi mac queues. From what I can see, the issue was due to multiple instantiations of Queue<WifiMacQueueItem> (due to wifi-mac-queue.h being included in multiple modules): the second instance tried to register the same uid again. The issue is solved by suppressing implicit instantiations of Queue<WifiMacQueueItem>, in much the same way I did for Queue<Packet> and Queue<QueueDiscItem>. Sorry that I didn't spot it before. Shall I push: - the commit to make WifiMacQueue a subclass of Queue<WifiMacQueueItem>, which includes the fix described above - the (64-bit only) bindings generated by using castxml to ns-3-dev or do you want to have a look at it before?
(In reply to Stefano Avallone from comment #15) > > This is the error from 32 bit bindings, add "--no32bit-scan" to the apiscan > > command. By default it still tries to scan both 32 and 64 bits > > Ahhh, thanks! I finally managed to rescan all the python bindings (64-bit > only). > > And I managed to fix the issue with wifi mac queues. > > From what I can see, the issue was due to multiple instantiations of > Queue<WifiMacQueueItem> (due to wifi-mac-queue.h being included in multiple > modules): the second instance tried to register the same uid again. The > issue is solved by suppressing implicit instantiations of > Queue<WifiMacQueueItem>, in much the same way I did for Queue<Packet> and > Queue<QueueDiscItem>. Sorry that I didn't spot it before. > > Shall I push: > > - the commit to make WifiMacQueue a subclass of Queue<WifiMacQueueItem>, > which includes the fix described above > > - the (64-bit only) bindings generated by using castxml > > to ns-3-dev or do you want to have a look at it before? I think you should push both, also when I worked around the wifi issue, there were 3 wifi related python tests failing.
> I think you should push both Thanks, I'll wait for the green flag from Tom as well. > also when I worked around the wifi issue, > there were 3 wifi related python tests failing. I checked that my fix makes tutorial/third.py work. Running test.py still shows 3 failing python tests, but I think failures are (now) unrelated to the wifi issue: [stefano@localhost ns-3-dev-git]$ python2 ./waf --pyrun examples/wireless/mixed-wired-wireless.py Waf: Entering directory `/home/stefano/Tools/ns-3-dev-git/build' Waf: Leaving directory `/home/stefano/Tools/ns-3-dev-git/build' Build commands will be stored in build/compile_commands.json 'build' finished successfully (1.552s) Enabling OLSR routing on all backbone nodes Configuring local area network for backbone node 0 Configuring local area network for backbone node 1 Configuring local area network for backbone node 2 Configuring local area network for backbone node 3 Configuring local area network for backbone node 4 Configuring local area network for backbone node 5 Configuring local area network for backbone node 6 Configuring local area network for backbone node 7 Configuring local area network for backbone node 8 Configuring local area network for backbone node 9 Configuring wireless network for backbone node 0 Configuring wireless network for backbone node 1 Configuring wireless network for backbone node 2 Configuring wireless network for backbone node 3 Configuring wireless network for backbone node 4 Configuring wireless network for backbone node 5 Configuring wireless network for backbone node 6 Configuring wireless network for backbone node 7 Configuring wireless network for backbone node 8 Configuring wireless network for backbone node 9 Create Applications. Configure Tracing. Traceback (most recent call last): File "examples/wireless/mixed-wired-wireless.py", line 380, in <module> main(sys.argv) File "examples/wireless/mixed-wired-wireless.py", line 349, in main stream = ascii.CreateFileStream("mixed-wireless.tr"); AttributeError: 'network.AsciiTraceHelper' object has no attribute 'CreateFileStream' Command ['/usr/bin/python2', 'examples/wireless/mixed-wired-wireless.py'] exited with code 1 [stefano@localhost ns-3-dev-git]$ python2 ./waf --pyrun src/flow-monitor/examples/wifi-olsr-flowmon.py Waf: Entering directory `/home/stefano/Tools/ns-3-dev-git/build' Waf: Leaving directory `/home/stefano/Tools/ns-3-dev-git/build' Build commands will be stored in build/compile_commands.json 'build' finished successfully (1.577s) Traceback (most recent call last): File "src/flow-monitor/examples/wifi-olsr-flowmon.py", line 185, in <module> sys.exit(main(sys.argv)) File "src/flow-monitor/examples/wifi-olsr-flowmon.py", line 158, in main for flow_id, flow_stats in monitor.GetFlowStats(): AttributeError: 'flow_monitor.FlowMonitor' object has no attribute 'GetFlowStats' Command ['/usr/bin/python2', 'src/flow-monitor/examples/wifi-olsr-flowmon.py'] exited with code 1 [stefano@localhost ns-3-dev-git]$ python2 ./waf --pyrun examples/routing/simple-routing-ping6.py Waf: Entering directory `/home/stefano/Tools/ns-3-dev-git/build' Waf: Leaving directory `/home/stefano/Tools/ns-3-dev-git/build' Build commands will be stored in build/compile_commands.json 'build' finished successfully (1.565s) Create nodes Addressing Application Tracing Traceback (most recent call last): File "examples/routing/simple-routing-ping6.py", line 111, in <module> main(sys.argv) File "examples/routing/simple-routing-ping6.py", line 102, in main csma.EnableAsciiAll(ascii.CreateFileStream("simple-routing-ping6.tr")) AttributeError: 'network.AsciiTraceHelper' object has no attribute 'CreateFileStream' Command ['/usr/bin/python2', 'examples/routing/simple-routing-ping6.py'] exited with code 1
(In reply to Stefano Avallone from comment #17) Looks like error due to bindings (python is unable to find some of the C++ functions). I will investigate and update with my findings
> > Shall I push: > > - the commit to make WifiMacQueue a subclass of Queue<WifiMacQueueItem>, > which includes the fix described above > > - the (64-bit only) bindings generated by using castxml > > to ns-3-dev or do you want to have a look at it before? Yes, please push.
Done, thanks.
Update: the ns-3 changes to support CastXML are in ns-3-dev; the following repo is no longer necessary: https://github.com/adeepkit01/castxml-changes The pybindgen changes are found in this repository: https://github.com/adeepkit01/pybindgen but the only file needed from there is the castxmlparser.xml file. We can now mostly scan the 64-bit bindings. There are a few hiccups: - cannot scan anonymous typedefs (there are two instances of these, one in wifi, one in mesh module) - has trouble with AsciiTraceHelper::CreateFileStream (std::string filename, std::ios::openmode filemode) due to std::ios::openmode The main problem at the moment is that MacOS behaves as a 32-bit Linux machine with respect to 'unsigned long' vs. 'unsigned long long'. This may be possibly overcome by just doing away with the 32- and 64-bit distinction. So the next step I think is to try to remove modulegen__gcc_LP64.py and modulegen__gcc_ILP32.py everywhere, and replace with modulegen__clang.py, and see if Linux and MacOS both can compile and use the bindings. Additional steps to complete: 0) resolve the small remaining ssues found when rescanning all bindings again 1) work with Gustavo to finalize the castxmlparser.xml file for pybindgen 2) prepare bake changes to install prerequisite packages, and provide new documentation
Current status as of the release of ns-3.27: 1) CastXML and recent pygccxml are now installed by the bake build tool. If a development version of clang is installed, this may go smoothly, but some intervention may be required on some systems (e.g. Ubuntu). 2) a castxmlparser.py file contributed by Ankit Deepak is needed for pybindgen (has not been upstreamed yet to pybindgen). Upstreaming is in process. The version of pybindgen installed by bake as part of ns-allinone-3.27 is the forked version with this additional file. 3) Once installed, bindings may be scanned in an automated way as in the past, but for 64-bit (LP64) only. That is, the --no32bit-scan argument must be passed when calling for an API scan. 4) In general, the 32-bit variants (ILP32) can be obtained by copying the LP64 file, and making a global substitution of 'unsigned long long' for 'unsigned long'. For some modules, there is no difference, but for others, a number of substitutions of this nature must be made. 5) There is also an issue when scanning the network module; it has trouble with AsciiTraceHelper::CreateFileStream (std::string filename, std::ios::openmode filemode) due to std::ios::openmode handling. The current situation is not ideal and we will continue to try to further automate the process.
Quick question to the developers: what exact versions of castxml, pygccxml, pybindgen are used for the bindings generation? Current versions from git repos generates: module.add_container('std::list< ns3::ArpCache::Entry * >', 'ns3::ArpCache::Entry', container_type=u'list') instead of: module.add_container('std::list< ns3::ArpCache::Entry * >', 'ns3::ArpCache::Entry *', container_type=u'list') in aodv module, for example. Resulting build/src/aodv/bindings/ns3module.cc fails to compile. Checked everything, can't see what is wrong with my setup. This problem blocks next steps I'd like to make.
(In reply to Nick Krylov from comment #23) > Quick question to the developers: what exact versions of castxml, pygccxml, > pybindgen are used for the bindings generation? > > Current versions from git repos generates: > > module.add_container('std::list< ns3::ArpCache::Entry * >', > 'ns3::ArpCache::Entry', container_type=u'list') > > instead of: > > module.add_container('std::list< ns3::ArpCache::Entry * >', > 'ns3::ArpCache::Entry *', container_type=u'list') > > in aodv module, for example. Resulting build/src/aodv/bindings/ns3module.cc > fails to compile. > > Checked everything, can't see what is wrong with my setup. This problem > blocks next steps I'd like to make. Nick, sorry I missed replying to this earlier. In my working setup (Ubuntu 16.04), I am using castxml: commit 367e90c1ecd92adb0071b4ec1ddf684213805afc (Oct 13) pygccxml: version 1.9.1 release pybindgen: commit 3eade7687ca15ac7e2c907f7c6a2065b5e6f8eb2 from Ankit Deepak's fork I will try to test now using a recent castxml
> > I will try to test now using a recent castxml The most recent castxml (commit 90d4abc1 from 15 Dec) worked for me. If you can provide more information about your setup, I could try to reproduce.
Here is my understanding of the latest status. Ankit Deepak is working on the remaining problems, although I'll test a potential solution to 4) below that has been suggested by Gustavo. 1) castxmlparser.py. I sent a pull request to Gustavo for pybindgen so that we could avoid the forked pybindgen repository, and he is looking at it; he wants to convert the tests that presently use gccxmlparser.py and reports that he needs more time. 2) Ankit is looking at how to suppress (or convert via some kind of script) the conversions of int64_t and uint64_t to unsigned long or unsigned long long, to either automate the generation of ILP and LP bindings or to remove the distinction altogether. 3) We need to test whether MacOs can install the toolchain generate the bindings. 4) There is a small problem requiring a couple of manual conversions for the network module; we scan and get std::ios::openmode and convert to std::_Ios_Openmode due to a customization (below) but we really need it to be std::ios_base::openmode. class IosOpenmodeParam(Parameter): DIRECTIONS = [Parameter.DIRECTION_IN] CTYPES = ['std::ios::openmode', 'std::_Ios_Openmode']
> > 4) There is a small problem requiring a couple of manual conversions for the > network module; we scan and get std::ios::openmode and convert to > std::_Ios_Openmode due to a customization (below) but we really need it to > be std::ios_base::openmode. > > class IosOpenmodeParam(Parameter): > > DIRECTIONS = [Parameter.DIRECTION_IN] > CTYPES = ['std::ios::openmode', 'std::_Ios_Openmode'] I'm not getting this error any more, for some reason, but I'll stash Gustavo's comment in case it pops up again: In bindings/python/ns3_modulegen_core_customizations.py: class IosOpenmodeParam(Parameter): DIRECTIONS = [Parameter.DIRECTION_IN] CTYPES = ['std::ios::openmode', 'std::_Ios_Openmode'] The "CTYPES" attribute controls which C types our custom type handler matches. We are trying to match std::_Ios_Openmode but the scanner is giving us std::ios_base::openmode. Perhaps a change like below would fix it, but I didn't test. class IosOpenmodeParam(Parameter): DIRECTIONS = [Parameter.DIRECTION_IN] - CTYPES = ['std::ios::openmode', 'std::_Ios_Openmode'] + CTYPES = ['std::ios::openmode', 'std::_Ios_Openmode', 'std::ios_base::openmode']
(In reply to Tom Henderson from comment #27) > > > > 4) There is a small problem requiring a couple of manual conversions for the > > network module; we scan and get std::ios::openmode and convert to > > std::_Ios_Openmode due to a customization (below) but we really need it to > > be std::ios_base::openmode. Changeset 13238:12733477b074 has now fixed this specific issue; other issues remain: 1) upstreaming castxmlparser.py to pybindgen 2) handling of 32-bit scans (LP64 vs ILP32 differences) 3) whether scanning works on macOS
As of ns-3.29 (changeset 13700:df091cb07fcb), one item remains: - support for rescanning from MacOS Moving to patch wanted state.