Difference between revisions of "GSOCSelectionProcess"

From Nsnam
Jump to: navigation, search
(Scoring System)
(Pre-evaluation phase)
Line 61: Line 61:
 
== Pre-evaluation phase ==
 
== Pre-evaluation phase ==
  
The organisation admin makes a pass through all the proposals. The admin will assign a score of 1 to spam, very weak applications, applications that do not fulfill the patch requirement and applications that are copy-paste jobs. These applications '''will not be''' considered for further review.
+
The organisation admin makes a pass through all the proposals. The admin will mark as ignored the incomplete, out of scope, or very weak applications. These applications '''will not be''' considered for further review.
  
  
'''Timeframe''': March 27-April 23, 2018
+
'''Timeframe''': completed on April 14, 2019
  
 
== First review phase ==
 
== First review phase ==

Revision as of 18:07, 14 April 2019

ns-3 Google Summer of Code Selection Process

This page describes how the ns-3 organisation reviews student applications for the Google Summer of Code programme and narrows down on the final selections.

In summary, we will form a selection committee of mentors and maintainers who will review the submitted applications. In cases where we need to probe further for a candidate's technical skills, we will organise IRC interviews with the candidates. After the reviews and the interviews are completed, the committee will agree upon a final ranking of applications. The admins will then recommend the final ranked list of applications to Google.

Selection Committee

In 2019, the committee consists of:

  • Tom Henderson (org admin)
  • Tommaso Pecorella (org admin)
  • Abhijith Anilkumar
  • Zoraze Ali
  • Biljana Bojovic
  • Ankit Deepak
  • Lorenza Giupponi
  • Alexander Krotov
  • Natale Patriciello
  • Mohit Tahiliani
  • Dizhi Zhou

The selection committee will have access to all applications and be able to leave public and private comments, and to score the applications.

Review Committee Checklist

  1. Sign in to the Google Summer of Code site as a mentor, and pick the ns-3 organisation(the organisation name is "The ns-3 network simulator"). You won't be able to see the applications otherwise.
  2. If you are the concerned mentor for a particular application, please select "Wish to mentor" on the left bar for the application so that we can assign you the application.
  3. Review every active application and decide upon whether you believe that the application merits a 'star' for further discussion. Please see below for the scoring guidelines.

Scoring System

In 2019, Google has changed the review system and no longer uses or tracks numerical scoring. However, they want projects to select 'outstanding' (or at the very least, 'very good') applications; how these are defined are left up to the individual projects.

The project will likely be allocated at most four or five slots, and we will not know in advance how many we will receive. We will therefore need to come to consensus or executive decision on what ranking we make for the top applications. We can use the 'star' system to flag those applications that a reviewer believes should be in consideration for the top four or five. Additional details that will allow us to order proposals will need to be discussed outside of the Google tool.

What are we looking for in a good proposal?

Our objective is to select students who are potential future maintainers of the ns-3 project. Over the summer, the community will be investing time and effort in mentoring the students to achieve this end. That said, this is what a good proposal will look like:

  1. A detailed proposal that sets a realistic list of deliverables, demonstrates a clear path towards achieving the same, is complete, and technically sound.
  2. The proposal is an original document authored by the candidate, and is not a copy paste job.
  3. The proposal is scoped such that it has a strong chance of being successfully completed and merged by the end of the coding period.
  4. The candidate is technically very strong. This is judged through prior experience with contributing to ns-3 and/or other open-source projects, code samples, their past experience and through interviews conducted during the review process by the mentoring team.
  5. The candidate does not have any other commitments over the summer, and can devote 40 hours per-week as is required of GSOC candidates.
  6. The proposal has been discussed at length on the mailing list and/or IRC.
  7. The student has demonstrated some interest in involvement with ns-3 either before or after GSOC ends.

Signs of a bad proposal

  1. The proposal is a regurgitation of the project description and the recommended reading.
  2. Poor professional conduct in developing the application (extensive copy-pasting for instance).
  3. There is little evidence for the candidate's technical abilities.
  4. Very little mailing list and community engagement in developing the proposal.
  5. The candidate has other commitments for the summer (vacations, exams, coursework, another job etc.).
  6. Previous failures within GSOC.


Review Process

Pre-evaluation phase

The organisation admin makes a pass through all the proposals. The admin will mark as ignored the incomplete, out of scope, or very weak applications. These applications will not be considered for further review.


Timeframe: completed on April 14, 2019

First review phase

  1. All members of the selection committee will assign scores to the applications based on the scoring system mentioned above. Note, the scores of 1, 2, 11, and 12 are reserved for the admins and the mentors of the corresponding projects.
  2. All reviewers must post a private comment for each of their top 3 proposal picks explaining their score. They are free to post comments for other applications as well.

At the end of the first review phase, admins will make a pass through the applications to ensure that scores are assigned equally across all proposals. This is for cases where some mentors are more lenient with awarding scores than others, and also to avoid situations of vested interests. Admins will adjust scores accordingly to balance against such situations.

Second review phase

Admins will present a list of second round candidates for discussion by the committee. The committee may organise interviews with the candidates to probe for technical skill and to clarify aspects of the proposal.

Final phase

Outcome of second review phase is discussed, and the final ranking list of candidates is decided. The organisation admin will then lock the final set of candidates and make the recommendation to Google.

Feedback on applications

Students who do not get selected are welcome to contact the organisation admins to solicit feedback on their applications.

Confidentiality and professional responsibilities of selection process

With respect to confidentiality and professional responsibilities of the review process, the selection committee is expected to treat the process similar to a peer reviewed, technical program committee. Example guidelines are posted here.