Difference between revisions of "Python3 Project"

From Nsnam
Jump to: navigation, search
(Reading update)
(Status)
Line 159: Line 159:
  
 
3. Fixing the Python3 issue discussed in the [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 previous status update]
 
3. Fixing the Python3 issue discussed in the [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 previous status update]
 +
 +
== November 10, 2014 ==
 +
 +
1. '''test_waf.py''' has been used to check the porting of waf successfully for the options ''build'', ''configure'', ''step'', ''clean'', ''dist'' and ''list''.
 +
 +
2. test_waf.py has been upgraded to conceal all the waf related output and direct stderr to the failed test cases log file.
 +
 +
3. test_waf.py provides information about the progress and test cases passed.
 +
 +
''' Bugs Fixed '''
 +
 +
1. Appearance of the message: ''Don't know how to configure dynamic library path for the platform 'linux'; assuming it's LD_LIBRARY_PATH.''. Fixed by modifying sys.platform in wutils
 +
 +
''' Tasks to be Done '''
 +
 +
1. Upgrading test_waf.py to support the remaining waf options.
 +
 +
2. Developing an automated test script for test.py
 +
 +
3. Fixing the Python3 issue discussed [http://www.nsnam.org/wiki/Python3_Project#September_22.2C_2014 here].

Revision as of 13:17, 10 November 2014

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Project Overview

Code

Mercurial repository located at https://bitbucket.org/ssanturkar/ns-3-python-port

Software versions

  • Python 2.7.3
  • Python 3.3.5
  • GCC 4.6.3 on linux2
  • Waf 1.7.13

Plan

Based on the porting book and review of the code, the following possible strategies were discussed with mentors:

(a) Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.

(b) Maintaining different development branches for Python 2 and 3 support in the framework’s code repository. The 2to3 converter script is used to obtain the Python 3 branch. And the Python 3 branch has to be manually updated as and when the Python 2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.

(c) Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python 2 branch can be built to generate up-to-date Python 3 code.

(d) Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.

Based on discussion with mentors, option (d) (rewrite of code, use of six Python module was selected.

The rest of the project covers the following deliverables:

(a) Using 2to3 converter to generate scripts for compatible with Python3. Apply further changes as discussed in the Python Porting book to provide backward compatibility to Python2.

(b) Test these new set of scripts for complete functionality. Develop an automated testing tool to simplify the testing process.

(c) Completing the above 2 steps in multiple iterations of small updates to the build scripts.

(d) Reviewing and merging the ported code to the ns-3-dev repository.

Status

July 18, 2014

1. Completed reading and understanding the functioning of the Waf buildtool from the Waf book guide.

2. Completed reading the the Python2 to Python3 Porting book.

3. Completed porting ns-3 wscripts for the build and configure operations.Latest changes committed to the GitHub repository. Following is a link to the repository: https://github.com/sid1607/NS3-wscript-port/ The repository contains only those files that had to be ported and are placed in their respective directories.

Observations:

The Python3 porting book discusses multiple strategies that could be used to carry out the porting process:

1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python2 support is discontinued.

2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python3 branch. And the Python3 branch has to be manually updated as and when the Python2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.

3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python2 branch can be built to generate up-to-date Python3 code.

4. Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

Tasks to be done:

1.The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.

2. Selecting one of the four approaches mentioned above for porting.

July 27, 2014

1. Porting of scripts to support both Python 2 and Python 3. This process consists of the following steps:-

  • Running the 2to3 converter on all existing Python2 files.
  • Providing Python 2 compatibility to the new Python 3 scripts. The following had to be done for this:
    • Handling the print function using the __future__ module
    • Rectifying ImportError by importing the correct package depending on the interpreter being used

The six module has not been used so far.

2. Performed the above process for all wscript and wutils.py files in the ns-3.19 directory and all python files in the ns-3.19/waf-tools directory.

3. Modified test.py to launch waf with the suitable interpreter.

4. Pushed all these changes to an initial GitHub repository (in the test-port branch). As of now the configure and build commands and test.py are functioning properly, but they still have to be tested to ensure that they support all the options.

Next steps:

1. Testing all the options provided by build, configure and test.py in both Python 2 and Python 3.

2. Continuing the porting process for the other commands supported by the main wscript.

August 25, 2014

1. Cloning of the ns-3-dev mercurial repository and committing all the changes on this repository in order to provide 2 and 3 compatibility.

2. Fixed some of the Unicode String and Bytes interchangeability problem in Python 3 encountered while testing some of the different options in the test.py script.

Observations

The Unicode String and Bytes problem seems to be the most frequently occurring issue for the Python3 scripts, and needs to be fixed for all possible options that waf and test.py support.

Issues being faced

Currently, the ported scripts work only for some of the options that are supported by them. I am looking for an automated technique that could be used to test all the command-line options provided by Waf and test.py to find and rectify the compatibility issues. I went though the NS-3 documentation, but it only describes how to write tests for new modules being updated/contributed to NS-3 (http://www.nsnam.org/docs/manual/html/tests.html).

I would like to know how I could go about writing the tests for these ported scripts. Once this testing phase is completed, I can push some of these changes to the main repository and begin the next iteration of porting.

Tasks to be done

1. Complete the testing for the waf build and configure commands and test.py to ensure that they work correctly in all cases.

2. Pushing these changes to the main repository and commencing the next iteration of porting.

September 22, 2014

1. The updated ns-3-dev repository has been hosted on BitBucket.

2. Manual tests on ported scripts have passed successfully for waf configure and waf build.

Tasks to be done

1. Develop an automated testing script for the wscripts.

2. Testing of the ported scripts on the buildslaves by the mentors.

Issues Faced

1. waf build re-builds certain packages (even if they have already been built) when the Python3 interpreter is used. This does not happen with Python2.

October 3, 2014

1. Manual tests passed for waf build and waf configure

2. The automated test script test_waf.py has been pushed to BitBucket. It currently uses a hard-coded list of options that are provided by waf, as test cases. waf build and waf configure passed for both Python2 and Python3.

Bugs Fixed

1. Appearance of the message: Don't know how to configure dynamic library path for the platform 'linux'; assuming it's LD_LIBRARY_PATH.. Fixed by modifying sys.platform in wutils

Issues Faced

1. Possible issue with Waf when used with the Python 3.4.2rc1 interpreter. waf configure fails with _pickle.PicklingError.

Tasks to be Done

1. Resolving the above issue with the mentors.

2. Continuing the testing process for other waf commands and test.py.

3. Fixing the Python3 issue discussed in the previous status update

November 10, 2014

1. test_waf.py has been used to check the porting of waf successfully for the options build, configure, step, clean, dist and list.

2. test_waf.py has been upgraded to conceal all the waf related output and direct stderr to the failed test cases log file.

3. test_waf.py provides information about the progress and test cases passed.

Bugs Fixed

1. Appearance of the message: Don't know how to configure dynamic library path for the platform 'linux'; assuming it's LD_LIBRARY_PATH.. Fixed by modifying sys.platform in wutils

Tasks to be Done

1. Upgrading test_waf.py to support the remaining waf options.

2. Developing an automated test script for test.py

3. Fixing the Python3 issue discussed here.