GSOC2020AppStore

Main Page - Roadmap - Summer Projects - Project Ideas - Developer FAQ - Tools - Related Projects

HOWTOs - Installation - Troubleshooting - User FAQ - Samples - Models - Education - Contributed Code - Papers

Back to GSoC 2020 projects

Project Overview

• Project Name: App Store improvements
• Student: Shivamani Patil
• Mentors: Abhijith Anilkumar, Ankit Deepak, Mishal Shah, Tom Henderson
• Project Goals: This project aims to develop a Jenkins automation server and add necessary updates to ns-3 AppStore to check if available, uploaded or updated apps/modules to AppStore build and pass tests successfully for the given ns-3 versions and display that information on AppStore. It also aims to make App Store aware of new ns-3 releases for calling build/test on all available App Store modules from Jenkins server, in turn, displaying build/test status for that ns-3 release on App Store. Ideally, it is about making App Store aware and capable of interacting with the newly developed Jenkins server.
• Repository: https://gitlab.com/shivamanipatil/ns-3-AppStore
• Final report: https://shivamanipatil.github.io/gsoc-2020-report/
• About Me: I am a second-year Information Technology undergraduate student from National Institute of Technology Karnataka(NITK), India. I am interested in computer systems, operating systems, and backend development. I have been working with Django and python for the past 1 year. I have previously worked on App Store for migrating Elasticsearch to the latest version.

Milestones and Deliverables

The whole GSoC period is divided into 3 phases. The deliverables of each phase are mentioned below:

Phase 1

• Adding Build model to AppStore, writing tests for it and its documentation
• Jenkins local installation
• Building pipelines in Jenkins

Phase 2

• Build status window on App page
• REST API creation in AppStore, adding tests and its documentation
• Build trigger scripts/functions

Phase 3

• Gitlab webhooks for new ns releases
• Deploying Jenkins on a dedicated server
• Comprehensive App user and developer guide

Phase reports

Phase 1

• Adding Build model to AppStore and it's test.
• Jenkins local installation and Building pipelines in Jenkins.
• Merge request for phase 1 work - link
• Doc for pipelines workflow and code explanations - link
• Doc regarding limitations of current pipeline scripts and guidelines for App release if they wish to use Jenkins server for build statuses - link

Phase 2

• App Editors and Admins can now call the build on an app release from the App Store.
• App Editors and Admins can now cancel the build from the App Store.
• Build history for app releases is visible.
• Util functions required to interact with Jenkins are added.
• The code is documented.
• Doc related to this phase - link

Phase 3

• Installation and contributing guide for ns-3 appstore.
• Readme documentation for setting the Jenkins server.
• Adding the pipelines to deployed Jenkins server.
• Testing Jenkins pipelines with dev appstore server.

Weekly Reports

Community Bonding Period

• Database migration for ns-3 AppStore from SQLite to Mysql/Postgres(waiting for mentors to try it on deployed site).Steps followed by me are documented in this doc.
• Participated in the coding sprint and have provided 3 merge requests for the ns-3 app store.
• Reviewed dependencies(doc) and provided 2 merge requests for package replacements - python-dotenv, django-admin-bootstrapped.

Week 1: 1st June - 7th June

• The Build model to store information about the build info for an app/module was created.
• Tests for the Build model were also added.
• Commit for the work : link

Week 2: 8th June - 14th June

• Couldn't work due to examinations and submissions(had pre-notified to mentors).

Week 3: 15th June - 21st June

• Installed Jenkins on the local machine.
• Read Jenkins documentation about pipeline building and its terminologies.
• Explored bake for fetching and building ns-3/modules/forks.
• Set up my local environment for pipeline building.

Week 4: 22nd June - 28th June

• Added scripts that will be used by modules/forks with bakefile.
• Added scripts that will be used by modules/forks providing repo link/ direct download link.

Week 5: 2th July - 5th July

• Planned and discussed with mentors about Build triggers, who can call build on releases and overall workflow for App Store side of pipeline triggering using Jenkins APIs.

Week 6: 6th July - 12th July

• Added functions in the App Store to call the Jenkins builds on app releases.
• Added some guideline checks in the App Store itself so that unnecessary builds are not triggered which would surely fail due to pipeline requirements:-

   1. Check for git repo link structure - if not in standard form modify it. 
   2. Check if tag/version used for that repo exists in the git repo.
   3. Check if direct link response content-type can actually be extracted/uncompressed in the pipeline.

• Added pipeline script support for more direct link download types zip, gzip, tar, tar.gz and tar.bz2 (first it was just tar.gz).

Week 7: 13th July - 19th July

• Added building and aborted status types and Jenkins job URL field to the Build model.
• Added token-based PATCH REST API in the App Store to update the Build status, Build id and Build URL from Jenkins pipelines.
• Updated the Pipeline codes to make REST API calls to the App Store to update the Build URL and build status in Post-build section (Updating build status according to build outcome i.e aborted, success or failure.)
• Display the latest build - (Build status, Build URL) for a release on the app page temporarily.
• Added an additional check in the App Store to check if repo exists of module/fork first before calling 'without bakefile' pipeline builds.

Week 8: 20th July - 26th July

• Added cancel build option
• Added tests for PATCH REST API used by Jenkins
• Added build history with pagination
• Jenkins env variables loaded from settings/base.py and imported in util/build_pipeline.py
• Code formating and comment/docstrings for functions
• Added exception handling where it was needed

Week 9: 27th July - 2nd August

• Made changes to the pull request according to the discussions on the pull request.
• Made Release model enforce the combined unique constraint on its app and version.
• Updated build history url to more human readable /build/history/app_name/release_version/.

Week 10: 3rd August - 9th August

• Added a periodic pipeline which builds the releases using ns-3-dev.
• Added a REST API that is used by this pipeline to inform AppStore to call builds on these releases.
• Started writing rough documentation about my GSoC work.

Week 11: 10th August - 16th August

• Added a pipeline to deployed Jenkins Server.
• Tested and debugged communication between deployed Jenkins and dev ns-3 appstore server.

Week 12: 17th August - 23rd August

• Added contributing and installation guides.
• Added README doc for setting the Jenkins server as per the project.