Difference between revisions of "Python3 Project"

From Nsnam
Jump to: navigation, search
(Older updates)
 
(15 intermediate revisions by the same user not shown)
Line 1: Line 1:
 
{{TOC}}
 
{{TOC}}
 
= Project Overview  =
 
= Project Overview  =
* Student:  student:  [mailto:siddharth.santurkar@ieee.org  Siddharth Santurkar]
+
* Student:  [mailto:siddharth.santurkar@ieee.org  Siddharth Santurkar]
 
* Mentors:  Vedran Miletic and Gustavo Carneiro
 
* Mentors:  Vedran Miletic and Gustavo Carneiro
 
* Abstract:  Linux distributions are expected to transition away from default Python 2.7 support to default Python 3 support.  ns-3 has several components that rely on Python, and not all are Python 3.3+ compatible.  This project focuses on updating our bindings generation process (pygccxml), PyViz visualizer, and wscript files (used by Waf) to support Python 3.3+.  Regarding wscript files in particular, student should also propose a refactoring and reorganization to make them easier to maintain in the future.
 
* Abstract:  Linux distributions are expected to transition away from default Python 2.7 support to default Python 3 support.  ns-3 has several components that rely on Python, and not all are Python 3.3+ compatible.  This project focuses on updating our bindings generation process (pygccxml), PyViz visualizer, and wscript files (used by Waf) to support Python 3.3+.  Regarding wscript files in particular, student should also propose a refactoring and reorganization to make them easier to maintain in the future.
 
* Recommended reading: pygccxml:  http://sourceforge.net/projects/pygccxml/
 
* Recommended reading: pygccxml:  http://sourceforge.net/projects/pygccxml/
 
* Recommended reading:  Discussion on ns-developers list:  http://mailman.isi.edu/pipermail/ns-developers/2014-March/011794.html
 
* Recommended reading:  Discussion on ns-developers list:  http://mailman.isi.edu/pipermail/ns-developers/2014-March/011794.html
 +
* Recommended reading: The Waf Book: http://docs.waf.googlecode.com/git/book_17/single.html
 +
* Recommended reading: Porting to Python 3: An in-depth guide : http://python3porting.com/
  
 
= Code =
 
= Code =
  
 
Mercurial repository located at https://bitbucket.org/ssanturkar/ns-3-python-port
 
Mercurial repository located at https://bitbucket.org/ssanturkar/ns-3-python-port
 +
 +
= Software versions =
 +
 +
* Python 2.7.3
 +
* Python 3.3.5
 +
* GCC 4.6.3 on linux2
 +
* Waf 1.7.13
  
 
= Plan =
 
= Plan =
  
Based on [http://python3porting.com/ porting books] and review of the code, the following possible strategies were discussed with mentors:
+
Based on the [http://python3porting.com/ porting book] and review of the code, the following possible strategies were discussed with mentors:
  
 
(a) Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.
 
(a) Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.
Line 23: Line 32:
 
changes in the Python 2 branch can be built to generate up-to-date Python 3 code.
 
changes in the Python 2 branch can be built to generate up-to-date Python 3 code.
  
(d) Providing Python2 and Python 3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.
+
(d) Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.
  
 
The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.
 
The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.
  
 
Based on discussion with mentors, option (d) (rewrite of code, use of six Python module was selected.
 
Based on discussion with mentors, option (d) (rewrite of code, use of six Python module was selected.
 +
 +
'''The rest of the project covers the following deliverables:'''
 +
 +
(a) Using 2to3 converter to generate scripts for compatible with Python3. Apply further changes as discussed in the Python Porting book to provide backward compatibility to Python2.
 +
 +
(b) Test these new set of scripts for complete functionality. Develop an automated testing tool to simplify the testing process.
 +
 +
(c) Completing the above 2 steps in multiple iterations of small updates to the build scripts.
 +
 +
(d) Reviewing and merging the ported code to the [http://code.nsnam.org/ns-3-dev ns-3-dev] repository.
  
 
= Status =
 
= Status =
Line 33: Line 52:
 
== July 18, 2014 ==
 
== July 18, 2014 ==
  
'''Summary of tasks completed till now:'''
+
# Completed reading and understanding the functioning of the Waf buildtool from the Waf book guide.
1. Completed reading and understanding the functioning of the Waf buildtool from the Waf book guide.
+
# Completed reading the the Python2 to Python3 Porting book.
2. Completed reading the the Python2 to Python3 Porting book.
+
# Completed porting ns-3 wscripts for the build and configure operations.Latest changes committed to the GitHub repository. Following is a link to the repository: https://github.com/sid1607/NS3-wscript-port/
3. Completed porting ns-3 wscripts for the build and con�figure operations.Latest changes committed to the GitHub repository. Following is a link to the repository:
+
The repository contains only those files that had to be ported and are placed in their respective directories.
https://github.com/sid1607/NS3-wscript-port/
+
The repository contains only those �files that had to be ported and are placed in their respective directories.
+
4. Ported test.py. Any feedback on this would be highly appreciated.https://github.com/sid1607/NS3-wscript-port/blob/master/ns-3.19/test.py
+
  
'''Observations:'''The porting book discusses multiple strategies that could be used to carryout the porting process:
+
'''Observations:'''
1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python 3 branch. And the Python 3 branch has to be manually updated as and when the Python 2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup fi�le can be confi�gured to run 2to3 during the build process. This way, any changes in the Python 2 branch can be built to generate up-to-date Python 3 code.4. Providing Python2 and Python 3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.'''Tasks to be done:'''
+
1.The build, con�figure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.2. Selecting one of the four approaches mentioned above for porting.
+
  
 +
The [http://python3porting.com/ Python3 porting book] discusses multiple strategies that could be used to carry out the porting process:
  
== July 27, 2014 ==
+
# Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python2 support is discontinued.
 +
# Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python3 branch. And the Python3 branch has to be manually updated as and when the Python2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.
 +
# Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python2 branch can be built to generate up-to-date Python3 code.
 +
# Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.
  
1. Porting of scripts to support both Python 2 and Python 3.  This process consists of the following steps:-
+
'''Tasks to be done:'''
* Running the 2to3 converter on all existing Python2 files.
+
* Providing Python 2 compatibility to the new Python 3 scripts. The following had to be done for this:
+
** Handling the print function using the __future__ module
+
** Rectifying ImportError by importing the correct package depending on the interpreter being used
+
The six module has not been used so far.
+
  
2. Performed the above process for all wscript and wutils.py files in the ns-3.19 directory and all python files in the ns-3.19/waf-tools directory.
+
# The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.
 +
# Selecting one of the four approaches mentioned above for porting.
  
3. Modified test.py to launch waf with the suitable interpreter.
+
== July 27, 2014 ==
  
4. Pushed all these changes to an initial GitHub repository (in the test-port branch).
+
* Porting of scripts to support both Python 2 and Python 3. This process consists of the following steps:-
 +
** Running the 2to3 converter on all existing Python2 files.
 +
** Providing Python 2 compatibility to the new Python 3 scripts. The following had to be done for this:
 +
** Handling the print function using the __future__ module
 +
** Rectifying ImportError by importing the correct package depending on the interpreter being used
 +
The six module has not been used so far.
 +
* Performed the above process for all wscript and wutils.py files in the ns-3.19 directory and all python files in the  ns-3.19/waf-tools directory.
 +
* Modified test.py to launch waf with the suitable interpreter.
 +
* Pushed all these changes to an initial GitHub repository (in the test-port branch).
 
As of now the configure and build commands and test.py are functioning properly, but they still have to be tested to ensure that they support all the options.
 
As of now the configure and build commands and test.py are functioning properly, but they still have to be tested to ensure that they support all the options.
  
 
'''Next steps:'''
 
'''Next steps:'''
  
1. Testing all the options provided by build, configure and test.py in both Python 2 and Python 3.
+
# Testing all the options provided by build, configure and test.py in both Python 2 and Python 3.
 
+
# Continuing the porting process for the other commands supported by the main wscript.
2. Continuing the porting process for the other commands supported by the main wscript.
+
  
 
== August 25, 2014 ==
 
== August 25, 2014 ==
Summary of tasks completed till now:
 
  
1. Cloning of the ns-3-dev mercurial repository and committing all the changes on this repository in order to provide 2 and 3 compatibility.
+
# Cloning of the ns-3-dev mercurial repository and committing all the changes on this repository in order to provide 2 and 3 compatibility.
 
+
# Fixed some of the Unicode String and Bytes interchangeability problem in Python 3 encountered while testing some of the different options in the test.py script.
2. Fixed some of the Unicode String and Bytes interchangeability problem in Python 3 encountered while testing some of the different options in the test.py script.
+
  
 
'''Observations'''
 
'''Observations'''
Line 88: Line 107:
 
'''Tasks to be done'''
 
'''Tasks to be done'''
  
1. Complete the testing for the waf build and configure commands and test.py to ensure that they work correctly in all cases.
+
# Complete the testing for the waf build and configure commands and test.py to ensure that they work correctly in all cases.
 +
# Pushing these changes to the main repository and commencing the next iteration of porting.
 +
 
 +
== September 22, 2014 ==
 +
 
 +
# The updated ns-3-dev repository has been hosted on BitBucket.
 +
# Manual tests on ported scripts have passed successfully for '''waf configure''' and '''waf build'''.
 +
 
 +
''' Tasks to be done '''
 +
 
 +
# Develop an automated testing script for the wscripts.
 +
# Testing of the ported scripts on the buildslaves by the mentors.
 +
 
 +
''' Issues Faced '''
 +
 
 +
# '''waf build''' re-builds certain packages (even if they have already been built) when the Python3 interpreter is used. This does not happen with Python2.
 +
 
 +
== October 3, 2014 ==
 +
 
 +
# Manual tests passed for '''waf build''' and '''waf configure'''
 +
# The automated test script '''test_waf.py''' has been pushed to BitBucket. It currently uses a hard-coded list of options that are provided by waf, as test cases. '''waf build''' and '''waf configure''' passed for both Python2 and Python3.
 +
 
 +
''' Bugs Fixed '''
 +
 
 +
# Appearance of the message: ''Don't know how to configure dynamic library path for the platform 'linux'; assuming it's LD_LIBRARY_PATH.''. Fixed by modifying sys.platform in wutils
 +
 
 +
''' Issues Faced '''
 +
 
 +
# Possible issue with Waf when used with the Python 3.4.2rc1 interpreter. '''waf configure''' fails with _pickle.PicklingError.
 +
 
 +
''' Tasks to be Done '''
 +
 
 +
# Resolving the above issue with the mentors.
 +
# Continuing the testing process for other waf commands and test.py.
 +
# Fixing the Python3 issue discussed in the [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 previous status update]
 +
 
 +
== November 10, 2014 ==
 +
 
 +
# '''test_waf.py''' has been used to check the porting of waf successfully for the options ''build'', ''configure'', ''step'', ''clean'', ''dist'' and ''list''.
 +
# test_waf.py has been upgraded to conceal all the waf related output and direct stderr to the failed test cases log file.
 +
# test_waf.py provides information about the progress and test cases passed.
 +
 
 +
''' Bugs Fixed '''
 +
 
 +
# Type casting of the NS3_MODULE_PATH environment variable to String for proc_env.
 +
 
 +
''' Tasks to be Done '''
 +
 
 +
# Upgrading test_waf.py to support the remaining waf options.
 +
# Developing an automated test script for test.py
 +
# Fixing the Python3 issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here].
 +
 
 +
== December 9, 2014 ==
 +
 
 +
# New automated script ''test_test.py'' to validate the Python 2 to 3 porting of ''test.py''.
 +
# Python 2 porting of test.py validated by the above script
 +
# Porting of Waf commands ''sphinx'' and ''install'' validated by ''test_waf.py''.
 +
 
 +
''' Tasks to be Done '''
 +
 
 +
# Upgrading ''test_waf.py'' to support the remaining waf options.
 +
# Using the automated script for test.py to validate the Python 3 porting.
 +
# Fixing the Python3 issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here].
 +
# Pushing updates to main ns-3-dev repository.
 +
 
 +
== February 10, 2015 ==
 +
 
 +
# Portability issue for the wscript used by pybindgen has been rectified.
 +
# Test automation for waf shifted to the ''utils/PortTesting'' directory.
 +
# Fully qualified path in CLI rectified.
 +
# Return of non-zero error code in case of test script failure.
 +
# Cleaning of by-product files and directories.
 +
# Automated test-cases generated with appropriate pre and post conditions for each command.
 +
# Failed test cases are tracked and displayed on exit.
 +
 
 +
''' Tasks to be Done '''
 +
# Fixing the Python3 issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here].
 +
# Exploring issues related to waf for Python versions >= 3.4
 +
# Pushing updates to main ns-3-dev repository.
 +
 
 +
== March 14, 2015 ==
 +
 
 +
# Following leads to the issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here] have been identified:-
 +
## During the first build phase, waf generates signatures for every task. Upon successful completion of the task, the signature of the task is stored into a hidden file in the build directory. The task signatures are serialized using the Pickle library.
 +
## Following builds after this would load the task signatures from the hidden file. For every task, it generates its signature and checks if it is present in the dictionary of the loaded task signatures. In case of Python3, for some tasks, the newly generated signature fails to match with the loaded signature, causing a KeyError and consequent re-building of that task. The issue possibly lies in the signature generation in Python3.
 +
# ''waf configure'' was found to work correctly on Python 3.3.5 and not on Python 3.4.1
  
2. Pushing these changes to the main repository and commencing the next iteration of porting.
+
''' Tasks to be Done '''
 +
# Validating that the issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here] is indeed due to signature generation approach used in Python3.
 +
# Exploring issues related to waf for Python versions >= 3.4
 +
# Pushing updates to main ns-3-dev repository.

Latest revision as of 07:19, 3 May 2015

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Project Overview

Code

Mercurial repository located at https://bitbucket.org/ssanturkar/ns-3-python-port

Software versions

  • Python 2.7.3
  • Python 3.3.5
  • GCC 4.6.3 on linux2
  • Waf 1.7.13

Plan

Based on the porting book and review of the code, the following possible strategies were discussed with mentors:

(a) Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.

(b) Maintaining different development branches for Python 2 and 3 support in the framework’s code repository. The 2to3 converter script is used to obtain the Python 3 branch. And the Python 3 branch has to be manually updated as and when the Python 2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.

(c) Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python 2 branch can be built to generate up-to-date Python 3 code.

(d) Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.

Based on discussion with mentors, option (d) (rewrite of code, use of six Python module was selected.

The rest of the project covers the following deliverables:

(a) Using 2to3 converter to generate scripts for compatible with Python3. Apply further changes as discussed in the Python Porting book to provide backward compatibility to Python2.

(b) Test these new set of scripts for complete functionality. Develop an automated testing tool to simplify the testing process.

(c) Completing the above 2 steps in multiple iterations of small updates to the build scripts.

(d) Reviewing and merging the ported code to the ns-3-dev repository.

Status

July 18, 2014

  1. Completed reading and understanding the functioning of the Waf buildtool from the Waf book guide.
  2. Completed reading the the Python2 to Python3 Porting book.
  3. Completed porting ns-3 wscripts for the build and configure operations.Latest changes committed to the GitHub repository. Following is a link to the repository: https://github.com/sid1607/NS3-wscript-port/

The repository contains only those files that had to be ported and are placed in their respective directories.

Observations:

The Python3 porting book discusses multiple strategies that could be used to carry out the porting process:

  1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python2 support is discontinued.
  2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python3 branch. And the Python3 branch has to be manually updated as and when the Python2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.
  3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python2 branch can be built to generate up-to-date Python3 code.
  4. Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

Tasks to be done:

  1. The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.
  2. Selecting one of the four approaches mentioned above for porting.

July 27, 2014

  • Porting of scripts to support both Python 2 and Python 3. This process consists of the following steps:-
    • Running the 2to3 converter on all existing Python2 files.
    • Providing Python 2 compatibility to the new Python 3 scripts. The following had to be done for this:
    • Handling the print function using the __future__ module
    • Rectifying ImportError by importing the correct package depending on the interpreter being used

The six module has not been used so far.

  • Performed the above process for all wscript and wutils.py files in the ns-3.19 directory and all python files in the ns-3.19/waf-tools directory.
  • Modified test.py to launch waf with the suitable interpreter.
  • Pushed all these changes to an initial GitHub repository (in the test-port branch).

As of now the configure and build commands and test.py are functioning properly, but they still have to be tested to ensure that they support all the options.

Next steps:

  1. Testing all the options provided by build, configure and test.py in both Python 2 and Python 3.
  2. Continuing the porting process for the other commands supported by the main wscript.

August 25, 2014

  1. Cloning of the ns-3-dev mercurial repository and committing all the changes on this repository in order to provide 2 and 3 compatibility.
  2. Fixed some of the Unicode String and Bytes interchangeability problem in Python 3 encountered while testing some of the different options in the test.py script.

Observations

The Unicode String and Bytes problem seems to be the most frequently occurring issue for the Python3 scripts, and needs to be fixed for all possible options that waf and test.py support.

Issues being faced

Currently, the ported scripts work only for some of the options that are supported by them. I am looking for an automated technique that could be used to test all the command-line options provided by Waf and test.py to find and rectify the compatibility issues. I went though the NS-3 documentation, but it only describes how to write tests for new modules being updated/contributed to NS-3 (http://www.nsnam.org/docs/manual/html/tests.html).

I would like to know how I could go about writing the tests for these ported scripts. Once this testing phase is completed, I can push some of these changes to the main repository and begin the next iteration of porting.

Tasks to be done

  1. Complete the testing for the waf build and configure commands and test.py to ensure that they work correctly in all cases.
  2. Pushing these changes to the main repository and commencing the next iteration of porting.

September 22, 2014

  1. The updated ns-3-dev repository has been hosted on BitBucket.
  2. Manual tests on ported scripts have passed successfully for waf configure and waf build.

Tasks to be done

  1. Develop an automated testing script for the wscripts.
  2. Testing of the ported scripts on the buildslaves by the mentors.

Issues Faced

  1. waf build re-builds certain packages (even if they have already been built) when the Python3 interpreter is used. This does not happen with Python2.

October 3, 2014

  1. Manual tests passed for waf build and waf configure
  2. The automated test script test_waf.py has been pushed to BitBucket. It currently uses a hard-coded list of options that are provided by waf, as test cases. waf build and waf configure passed for both Python2 and Python3.

Bugs Fixed

  1. Appearance of the message: Don't know how to configure dynamic library path for the platform 'linux'; assuming it's LD_LIBRARY_PATH.. Fixed by modifying sys.platform in wutils

Issues Faced

  1. Possible issue with Waf when used with the Python 3.4.2rc1 interpreter. waf configure fails with _pickle.PicklingError.

Tasks to be Done

  1. Resolving the above issue with the mentors.
  2. Continuing the testing process for other waf commands and test.py.
  3. Fixing the Python3 issue discussed in the previous status update

November 10, 2014

  1. test_waf.py has been used to check the porting of waf successfully for the options build, configure, step, clean, dist and list.
  2. test_waf.py has been upgraded to conceal all the waf related output and direct stderr to the failed test cases log file.
  3. test_waf.py provides information about the progress and test cases passed.

Bugs Fixed

  1. Type casting of the NS3_MODULE_PATH environment variable to String for proc_env.

Tasks to be Done

  1. Upgrading test_waf.py to support the remaining waf options.
  2. Developing an automated test script for test.py
  3. Fixing the Python3 issue discussed here.

December 9, 2014

  1. New automated script test_test.py to validate the Python 2 to 3 porting of test.py.
  2. Python 2 porting of test.py validated by the above script
  3. Porting of Waf commands sphinx and install validated by test_waf.py.

Tasks to be Done

  1. Upgrading test_waf.py to support the remaining waf options.
  2. Using the automated script for test.py to validate the Python 3 porting.
  3. Fixing the Python3 issue discussed here.
  4. Pushing updates to main ns-3-dev repository.

February 10, 2015

  1. Portability issue for the wscript used by pybindgen has been rectified.
  2. Test automation for waf shifted to the utils/PortTesting directory.
  3. Fully qualified path in CLI rectified.
  4. Return of non-zero error code in case of test script failure.
  5. Cleaning of by-product files and directories.
  6. Automated test-cases generated with appropriate pre and post conditions for each command.
  7. Failed test cases are tracked and displayed on exit.

Tasks to be Done

  1. Fixing the Python3 issue discussed here.
  2. Exploring issues related to waf for Python versions >= 3.4
  3. Pushing updates to main ns-3-dev repository.

March 14, 2015

  1. Following leads to the issue discussed here have been identified:-
    1. During the first build phase, waf generates signatures for every task. Upon successful completion of the task, the signature of the task is stored into a hidden file in the build directory. The task signatures are serialized using the Pickle library.
    2. Following builds after this would load the task signatures from the hidden file. For every task, it generates its signature and checks if it is present in the dictionary of the loaded task signatures. In case of Python3, for some tasks, the newly generated signature fails to match with the loaded signature, causing a KeyError and consequent re-building of that task. The issue possibly lies in the signature generation in Python3.
  2. waf configure was found to work correctly on Python 3.3.5 and not on Python 3.4.1

Tasks to be Done

  1. Validating that the issue discussed here is indeed due to signature generation approach used in Python3.
  2. Exploring issues related to waf for Python versions >= 3.4
  3. Pushing updates to main ns-3-dev repository.