Difference between revisions of "Python3 Project"

From Nsnam
Jump to: navigation, search
m
m
Line 42: Line 42:
 
'''Observations:'''
 
'''Observations:'''
  
The porting book discusses multiple strategies that could be used to carryout the porting process:
+
The [http://python3porting.com/ Python3 porting book] discusses multiple strategies that could be used to carry out the porting process:
  
1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.
+
1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python2 support is discontinued.
  
2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python 3 branch. And the Python 3 branch has to be manually updated as and when the Python 2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.
+
2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python3 branch. And the Python3 branch has to be manually updated as and when the Python2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.
  
3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python 2 branch can be built to generate up-to-date Python 3 code.
+
3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python2 branch can be built to generate up-to-date Python3 code.
  
4. Providing Python2 and Python 3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.
+
4. Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.
  
 
'''Tasks to be done:'''
 
'''Tasks to be done:'''

Revision as of 08:03, 3 October 2014

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Project Overview

  • Student: Siddharth Santurkar
  • Mentors: Vedran Miletic and Gustavo Carneiro
  • Abstract: Linux distributions are expected to transition away from default Python 2.7 support to default Python 3 support. ns-3 has several components that rely on Python, and not all are Python 3.3+ compatible. This project focuses on updating our bindings generation process (pygccxml), PyViz visualizer, and wscript files (used by Waf) to support Python 3.3+. Regarding wscript files in particular, student should also propose a refactoring and reorganization to make them easier to maintain in the future.
  • Recommended reading: pygccxml: http://sourceforge.net/projects/pygccxml/
  • Recommended reading: Discussion on ns-developers list: http://mailman.isi.edu/pipermail/ns-developers/2014-March/011794.html

Code

Mercurial repository located at https://bitbucket.org/ssanturkar/ns-3-python-port

Plan

Based on porting books and review of the code, the following possible strategies were discussed with mentors:

(a) Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python 2 support is discontinued.

(b) Maintaining different development branches for Python 2 and 3 support in the framework’s code repository. The 2to3 converter script is used to obtain the Python 3 branch. And the Python 3 branch has to be manually updated as and when the Python 2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.

(c) Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python 2 branch can be built to generate up-to-date Python 3 code.

(d) Providing Python2 and Python 3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.

Based on discussion with mentors, option (d) (rewrite of code, use of six Python module was selected.

Status

July 18, 2014

1. Completed reading and understanding the functioning of the Waf buildtool from the Waf book guide.

2. Completed reading the the Python2 to Python3 Porting book.

3. Completed porting ns-3 wscripts for the build and configure operations.Latest changes committed to the GitHub repository. Following is a link to the repository: https://github.com/sid1607/NS3-wscript-port/ The repository contains only those files that had to be ported and are placed in their respective directories.

Observations:

The Python3 porting book discusses multiple strategies that could be used to carry out the porting process:

1. Manually converting the existing Python2 script using the 2to3 converter tool. This could be used if only the Python3 scripts are to be maintained and the existing Python2 support is discontinued.

2. Maintaining different development branches for Python 2 and 3 support in the framework's code repository. The 2to3 converter script is used to obtain the Python3 branch. And the Python3 branch has to be manually updated as and when the Python2 branch is updated. This could be automated using a shell script, but it will be very tedious to develop and maintain such a script for large repositories.

3. Using a Python module like Setuptools to maintain the repository (or just the python codes in the repository). The setup file can be configured to run 2to3 during the build process. This way, any changes in the Python2 branch can be built to generate up-to-date Python3 code.

4. Providing Python2 and Python3 support for the framework without using conversion, i.e. modifying the existing code to be compatible with both Python2 and Python3, using the six Python module.

Tasks to be done:

1.The build, configure operations and the test.py script need to be tested with all the possible options that they support. I am currently trying to identify the test suites for this. Once I test these thoroughly, I can proceed with porting the other goals.

2. Selecting one of the four approaches mentioned above for porting.

July 27, 2014

1. Porting of scripts to support both Python 2 and Python 3. This process consists of the following steps:-

  • Running the 2to3 converter on all existing Python2 files.
  • Providing Python 2 compatibility to the new Python 3 scripts. The following had to be done for this:
    • Handling the print function using the __future__ module
    • Rectifying ImportError by importing the correct package depending on the interpreter being used

The six module has not been used so far.

2. Performed the above process for all wscript and wutils.py files in the ns-3.19 directory and all python files in the ns-3.19/waf-tools directory.

3. Modified test.py to launch waf with the suitable interpreter.

4. Pushed all these changes to an initial GitHub repository (in the test-port branch). As of now the configure and build commands and test.py are functioning properly, but they still have to be tested to ensure that they support all the options.

Next steps:

1. Testing all the options provided by build, configure and test.py in both Python 2 and Python 3.

2. Continuing the porting process for the other commands supported by the main wscript.

August 25, 2014

Summary of tasks completed till now:

1. Cloning of the ns-3-dev mercurial repository and committing all the changes on this repository in order to provide 2 and 3 compatibility.

2. Fixed some of the Unicode String and Bytes interchangeability problem in Python 3 encountered while testing some of the different options in the test.py script.

Observations

The Unicode String and Bytes problem seems to be the most frequently occurring issue for the Python3 scripts, and needs to be fixed for all possible options that waf and test.py support.

Issues being faced

Currently, the ported scripts work only for some of the options that are supported by them. I am looking for an automated technique that could be used to test all the command-line options provided by Waf and test.py to find and rectify the compatibility issues. I went though the NS-3 documentation, but it only describes how to write tests for new modules being updated/contributed to NS-3 (http://www.nsnam.org/docs/manual/html/tests.html).

I would like to know how I could go about writing the tests for these ported scripts. Once this testing phase is completed, I can push some of these changes to the main repository and begin the next iteration of porting.

Tasks to be done

1. Complete the testing for the waf build and configure commands and test.py to ensure that they work correctly in all cases.

2. Pushing these changes to the main repository and commencing the next iteration of porting.