Difference between revisions of "GSOC2020AppStore"

From Nsnam
Jump to: navigation, search
(Phase 2)
(Project Overview)
 
(21 intermediate revisions by one other user not shown)
Line 8: Line 8:
 
* '''Student:''' [mailto:shivamanipatil10@gmail.com Shivamani Patil]
 
* '''Student:''' [mailto:shivamanipatil10@gmail.com Shivamani Patil]
 
* '''Mentors:''' [mailto:abhijithabhayam@gmail.com Abhijith Anilkumar], [mailto:adadeepak8@gmail.com Ankit Deepak], [mailto:shahmishal1998@gmail.com Mishal Shah], [mailto:tomh@tomh.org Tom Henderson]
 
* '''Mentors:''' [mailto:abhijithabhayam@gmail.com Abhijith Anilkumar], [mailto:adadeepak8@gmail.com Ankit Deepak], [mailto:shahmishal1998@gmail.com Mishal Shah], [mailto:tomh@tomh.org Tom Henderson]
* '''Project Goals:''' This project aims to develop a CI/CD server and add necessary updates to ns-3 AppStore to check if available, uploaded or updated apps/modules to AppStore build and pass tests successfully for the given ns-3 versions and display that information on AppStore. It also aims to make App Store aware of new ns-3 releases for calling build/test on all available App Store modules from CI/CD server, in turn, displaying build/test status for that ns-3 release on App Store. Ideally, it is about making App Store aware and capable of interacting with the newly developed CI/CD server.   
+
* '''Project Goals:''' This project aims to develop a Jenkins automation server and add necessary updates to ns-3 AppStore to check if available, uploaded or updated apps/modules to AppStore build and pass tests successfully for the given ns-3 versions and display that information on AppStore. It also aims to make App Store aware of new ns-3 releases for calling build/test on all available App Store modules from Jenkins server, in turn, displaying build/test status for that ns-3 release on App Store. Ideally, it is about making App Store aware and capable of interacting with the newly developed Jenkins server.   
 
* '''Repository:''' https://gitlab.com/shivamanipatil/ns-3-AppStore
 
* '''Repository:''' https://gitlab.com/shivamanipatil/ns-3-AppStore
 +
* '''Final report:''' https://shivamanipatil.github.io/gsoc-2020-report/
 
* '''About Me:''' I am a second-year Information Technology undergraduate student from National Institute of Technology Karnataka(NITK), India. I am interested in computer systems, operating systems, and backend development. I have been working with Django and python for the past 1 year. I have previously worked on App Store for [https://gitlab.com/nsnam/ns-3-AppStore/-/merge_requests/61 migrating Elasticsearch to the latest version].
 
* '''About Me:''' I am a second-year Information Technology undergraduate student from National Institute of Technology Karnataka(NITK), India. I am interested in computer systems, operating systems, and backend development. I have been working with Django and python for the past 1 year. I have previously worked on App Store for [https://gitlab.com/nsnam/ns-3-AppStore/-/merge_requests/61 migrating Elasticsearch to the latest version].
  
Line 27: Line 28:
  
 
===Phase 3===
 
===Phase 3===
* to be added
+
* Gitlab webhooks for new ns releases
 +
* Deploying Jenkins on a dedicated server
 +
* Comprehensive App user and developer guide
 +
 
 +
 
 +
= Phase reports =
 +
 
 +
===Phase 1===
 +
* Adding Build model to AppStore and it's test.
 +
* Jenkins local installation and Building pipelines in Jenkins.
 +
* Merge request for phase 1 work - [https://gitlab.com/nsnam/ns-3-AppStore/-/merge_requests/69 link]
 +
* Doc for pipelines workflow and code explanations - [https://docs.google.com/document/d/1ekx4xlLK6KDj9TnFTpxVFp_7RelAYe7JvzX-Y5RRfhA/edit?usp=sharing link]
 +
* Doc regarding limitations of current pipeline scripts and guidelines for App release if they wish to use Jenkins server for build statuses - [https://docs.google.com/document/d/19xdtI-qfJmVoJJK-JN88jyNBQAyrkBvfuM_nTfdSh_s/edit?usp=sharing link] 
 +
 
 +
===Phase 2===
 +
* App Editors and Admins can now call the build on an app release from the App Store.
 +
* App Editors and Admins can now cancel the build from the App Store.
 +
* Build history for app releases is visible.
 +
* Util functions required to interact with Jenkins are added.
 +
* The code is documented.
 +
* Doc related to this phase - [https://docs.google.com/document/d/1KofTOsiA0I_HQFSQ_zc-iK2gdN2QRyYFoRiuG2GkIh4/edit?usp=sharing link]
 +
 
 +
===Phase 3===
 +
* Installation and contributing guide for ns-3 appstore.
 +
* Readme documentation for setting the Jenkins server.
 +
* Adding the pipelines to deployed Jenkins server.
 +
* Testing Jenkins pipelines with dev appstore server.
  
 
= Weekly Reports =
 
= Weekly Reports =
Line 36: Line 63:
  
 
=== Week 1: 1st June - 7th June ===
 
=== Week 1: 1st June - 7th June ===
* to be added
+
* The Build model to store information about the build info for an app/module was created.
 +
* Tests for the Build model were also added.
 +
* Commit for the work : [https://gitlab.com/shivamanipatil/ns-3-AppStore/-/commit/db059501e9537233da4f1294deb9d9039908df1d link]
  
 
=== Week 2: 8th June - 14th June ===
 
=== Week 2: 8th June - 14th June ===
* to be added
+
* Couldn't work due to examinations and submissions(had pre-notified to mentors).
  
 
=== Week 3: 15th June - 21st June ===
 
=== Week 3: 15th June - 21st June ===
* to be added
+
* Installed Jenkins on the local machine.
 +
* Read Jenkins documentation about pipeline building and its terminologies.
 +
* Explored bake for fetching and building ns-3/modules/forks.
 +
* Set up my local environment for pipeline building.
  
 
=== Week 4: 22nd June - 28th June ===
 
=== Week 4: 22nd June - 28th June ===
* to be added
+
* Added scripts that will be used by modules/forks with bakefile.
 +
* Added scripts that will be used by modules/forks providing repo link/ direct download link.
  
=== Week 5: 29th June - 5th July ===
+
=== Week 5: 2th July - 5th July ===
* to be added
+
* Planned and discussed with mentors about Build triggers, who can call build on releases and overall workflow for App Store side of pipeline triggering using Jenkins APIs.
  
 
=== Week 6: 6th July - 12th July ===
 
=== Week 6: 6th July - 12th July ===
* to be added
+
* Added functions in the App Store to call the Jenkins builds on app releases.
 +
* Added some guideline checks in the App Store itself so that unnecessary builds are not triggered which would surely fail due to pipeline requirements:-
 +
    1. Check for git repo link structure - if not in standard form modify it.
 +
    2. Check if tag/version used for that repo exists in the git repo.
 +
    3. Check if direct link response content-type can actually be extracted/uncompressed in the pipeline.
 +
* Added pipeline script support for more direct link download types zip, gzip, tar, tar.gz and tar.bz2 (first it was just tar.gz).
  
 
=== Week 7: 13th July - 19th July ===
 
=== Week 7: 13th July - 19th July ===
* to be added
+
* Added building and aborted status types and Jenkins job URL field to the Build model.
 +
* Added token-based PATCH REST API in the App Store to update the Build status, Build id and Build URL from Jenkins pipelines.
 +
* Updated the Pipeline codes to make REST API calls to the App Store to update the Build URL and build status in Post-build section (Updating build status according to build outcome i.e aborted, success or failure.)
 +
* Display the latest build - (Build status, Build URL) for a release on the app page temporarily.
 +
* Added an additional check in the App Store to check if repo exists of module/fork first before calling 'without bakefile' pipeline builds.
  
 
=== Week 8: 20th July - 26th July ===
 
=== Week 8: 20th July - 26th July ===
* to be added
+
* Added cancel build option
 +
* Added tests for PATCH REST API used by Jenkins
 +
* Added build history with pagination
 +
* Jenkins env variables loaded from settings/base.py and imported in util/build_pipeline.py
 +
* Code formating and comment/docstrings for functions
 +
* Added exception handling where it was needed
  
 
=== Week 9: 27th July - 2nd August ===
 
=== Week 9: 27th July - 2nd August ===
* to be added
+
* Made changes to the pull request according to the discussions on the pull request.
 +
* Made Release model enforce the combined unique constraint on its app and version.
 +
* Updated build history url to more human readable /build/history/app_name/release_version/.
  
 
=== Week 10: 3rd August - 9th August ===
 
=== Week 10: 3rd August - 9th August ===
* to be added
+
* Added a periodic pipeline which builds the releases using ns-3-dev.
 +
* Added a REST API that is used by this pipeline to inform AppStore to call builds on these releases.
 +
* Started writing rough documentation about my GSoC work.
  
 
=== Week 11: 10th August - 16th August ===
 
=== Week 11: 10th August - 16th August ===
* to be added
+
* Added a pipeline to deployed Jenkins Server.
 +
* Tested and debugged communication between deployed Jenkins and dev ns-3 appstore server.
  
 
=== Week 12: 17th August - 23rd August ===
 
=== Week 12: 17th August - 23rd August ===
* to be added
+
* Added contributing and installation guides.
 +
* Added README doc for setting the Jenkins server as per the project.

Latest revision as of 15:31, 7 September 2020

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

Back to GSoC 2020 projects

Project Overview

  • Project Name: App Store improvements
  • Student: Shivamani Patil
  • Mentors: Abhijith Anilkumar, Ankit Deepak, Mishal Shah, Tom Henderson
  • Project Goals: This project aims to develop a Jenkins automation server and add necessary updates to ns-3 AppStore to check if available, uploaded or updated apps/modules to AppStore build and pass tests successfully for the given ns-3 versions and display that information on AppStore. It also aims to make App Store aware of new ns-3 releases for calling build/test on all available App Store modules from Jenkins server, in turn, displaying build/test status for that ns-3 release on App Store. Ideally, it is about making App Store aware and capable of interacting with the newly developed Jenkins server.
  • Repository: https://gitlab.com/shivamanipatil/ns-3-AppStore
  • Final report: https://shivamanipatil.github.io/gsoc-2020-report/
  • About Me: I am a second-year Information Technology undergraduate student from National Institute of Technology Karnataka(NITK), India. I am interested in computer systems, operating systems, and backend development. I have been working with Django and python for the past 1 year. I have previously worked on App Store for migrating Elasticsearch to the latest version.

Milestones and Deliverables

The whole GSoC period is divided into 3 phases. The deliverables of each phase are mentioned below:

Phase 1

  • Adding Build model to AppStore, writing tests for it and its documentation
  • Jenkins local installation
  • Building pipelines in Jenkins

Phase 2

  • Build status window on App page
  • REST API creation in AppStore, adding tests and its documentation
  • Build trigger scripts/functions

Phase 3

  • Gitlab webhooks for new ns releases
  • Deploying Jenkins on a dedicated server
  • Comprehensive App user and developer guide


Phase reports

Phase 1

  • Adding Build model to AppStore and it's test.
  • Jenkins local installation and Building pipelines in Jenkins.
  • Merge request for phase 1 work - link
  • Doc for pipelines workflow and code explanations - link
  • Doc regarding limitations of current pipeline scripts and guidelines for App release if they wish to use Jenkins server for build statuses - link

Phase 2

  • App Editors and Admins can now call the build on an app release from the App Store.
  • App Editors and Admins can now cancel the build from the App Store.
  • Build history for app releases is visible.
  • Util functions required to interact with Jenkins are added.
  • The code is documented.
  • Doc related to this phase - link

Phase 3

  • Installation and contributing guide for ns-3 appstore.
  • Readme documentation for setting the Jenkins server.
  • Adding the pipelines to deployed Jenkins server.
  • Testing Jenkins pipelines with dev appstore server.

Weekly Reports

Community Bonding Period

  • Database migration for ns-3 AppStore from SQLite to Mysql/Postgres(waiting for mentors to try it on deployed site).Steps followed by me are documented in this doc.
  • Participated in the coding sprint and have provided 3 merge requests for the ns-3 app store.
  • Reviewed dependencies(doc) and provided 2 merge requests for package replacements - python-dotenv, django-admin-bootstrapped.

Week 1: 1st June - 7th June

  • The Build model to store information about the build info for an app/module was created.
  • Tests for the Build model were also added.
  • Commit for the work : link

Week 2: 8th June - 14th June

  • Couldn't work due to examinations and submissions(had pre-notified to mentors).

Week 3: 15th June - 21st June

  • Installed Jenkins on the local machine.
  • Read Jenkins documentation about pipeline building and its terminologies.
  • Explored bake for fetching and building ns-3/modules/forks.
  • Set up my local environment for pipeline building.

Week 4: 22nd June - 28th June

  • Added scripts that will be used by modules/forks with bakefile.
  • Added scripts that will be used by modules/forks providing repo link/ direct download link.

Week 5: 2th July - 5th July

  • Planned and discussed with mentors about Build triggers, who can call build on releases and overall workflow for App Store side of pipeline triggering using Jenkins APIs.

Week 6: 6th July - 12th July

  • Added functions in the App Store to call the Jenkins builds on app releases.
  • Added some guideline checks in the App Store itself so that unnecessary builds are not triggered which would surely fail due to pipeline requirements:-
   1. Check for git repo link structure - if not in standard form modify it. 
   2. Check if tag/version used for that repo exists in the git repo.
   3. Check if direct link response content-type can actually be extracted/uncompressed in the pipeline.
  • Added pipeline script support for more direct link download types zip, gzip, tar, tar.gz and tar.bz2 (first it was just tar.gz).

Week 7: 13th July - 19th July

  • Added building and aborted status types and Jenkins job URL field to the Build model.
  • Added token-based PATCH REST API in the App Store to update the Build status, Build id and Build URL from Jenkins pipelines.
  • Updated the Pipeline codes to make REST API calls to the App Store to update the Build URL and build status in Post-build section (Updating build status according to build outcome i.e aborted, success or failure.)
  • Display the latest build - (Build status, Build URL) for a release on the app page temporarily.
  • Added an additional check in the App Store to check if repo exists of module/fork first before calling 'without bakefile' pipeline builds.

Week 8: 20th July - 26th July

  • Added cancel build option
  • Added tests for PATCH REST API used by Jenkins
  • Added build history with pagination
  • Jenkins env variables loaded from settings/base.py and imported in util/build_pipeline.py
  • Code formating and comment/docstrings for functions
  • Added exception handling where it was needed

Week 9: 27th July - 2nd August

  • Made changes to the pull request according to the discussions on the pull request.
  • Made Release model enforce the combined unique constraint on its app and version.
  • Updated build history url to more human readable /build/history/app_name/release_version/.

Week 10: 3rd August - 9th August

  • Added a periodic pipeline which builds the releases using ns-3-dev.
  • Added a REST API that is used by this pipeline to inform AppStore to call builds on these releases.
  • Started writing rough documentation about my GSoC work.

Week 11: 10th August - 16th August

  • Added a pipeline to deployed Jenkins Server.
  • Tested and debugged communication between deployed Jenkins and dev ns-3 appstore server.

Week 12: 17th August - 23rd August

  • Added contributing and installation guides.
  • Added README doc for setting the Jenkins server as per the project.