Difference between revisions of "GSOCSelectionProcess"

From Nsnam
Jump to: navigation, search
(Review Process)
(First review phase)
Line 61: Line 61:
 
# All members of the selection committee will review the remaining applications.  If there is an application that the reviewer would like to mentor (if selected), the reviewer should select 'Want to mentor'.  Note that more than one person can select this; we can sort out details and overlaps later.  If there is an application that the reviewer thinks is high enough quality to merit consideration for a final slot, and is not yet marked with a star, then the reviewer may mark it with a star.
 
# All members of the selection committee will review the remaining applications.  If there is an application that the reviewer would like to mentor (if selected), the reviewer should select 'Want to mentor'.  Note that more than one person can select this; we can sort out details and overlaps later.  If there is an application that the reviewer thinks is high enough quality to merit consideration for a final slot, and is not yet marked with a star, then the reviewer may mark it with a star.
 
# Do not remove a star that was placed by another reviewer.
 
# Do not remove a star that was placed by another reviewer.
# All reviewers are encouraged to keep private notes of which of the starred proposals they believe are stronger than the others, for use in the next phase.  We won't use the Google Internal Review form for this phase.
+
# All reviewers are encouraged to make private notes of which of the starred proposals they believe are stronger than the others, and of strengths and weaknesses, for use in the next phase.  We won't use the Google Internal Review form for this phase.
  
 
At the end of the first review phase, the starred applications will be taken into the next review phase.
 
At the end of the first review phase, the starred applications will be taken into the next review phase.

Revision as of 18:44, 14 April 2019

ns-3 Google Summer of Code Selection Process

This page describes how the ns-3 organisation reviews student applications for the Google Summer of Code programme and narrows down on the final selections.

In summary, we have formed a selection committee of mentors and maintainers who will review the submitted applications. In cases where we need to probe further for a candidate's technical skills, we will organise chat interviews or email exchanges with the candidates. After the reviews and the interviews are completed, the committee will agree upon a final ranking of applications. The admins will then recommend the final ranked list of applications to Google.

Selection Committee

In 2019, the committee consists of:

  • Tom Henderson (org admin)
  • Tommaso Pecorella (org admin)
  • Abhijith Anilkumar
  • Zoraze Ali
  • Biljana Bojovic
  • Ankit Deepak
  • Lorenza Giupponi
  • Alexander Krotov
  • Natale Patriciello
  • Mohit Tahiliani
  • Dizhi Zhou

The selection committee will have access to all applications and be able to leave public and private comments, and to score the applications.

Scoring System

In 2019, Google has changed the review system and no longer uses or tracks numerical scoring. However, they want projects to select 'outstanding' (or at the very least, 'very good') applications; how these are defined are left up to the individual projects.

ns-3 will likely be allocated at most four or five slots, and we will not know in advance how many we will receive. We will therefore need to come to consensus or executive decision on what ranking we make for the top applications. We can use the 'star' system to flag those applications that a reviewer believes should be in consideration for the top four or five. Additional details that will allow us to order proposals will need to be discussed outside of the Google tool.

What are we looking for in a good proposal?

Our objective is to select students who have convinced us that they should be able to deliver a nice project and who are potential future contributors to the ns-3 project. Over the summer, the community will be investing time and effort in mentoring the students to achieve this end. That said, this is what a good proposal will look like:

  1. A detailed proposal that sets a realistic list of deliverables, demonstrates a clear path towards achieving the same, is complete, and technically sound.
  2. The proposal is an original document authored by the candidate, and is not a copy paste job.
  3. The proposal is scoped such that it has a strong chance of being successfully completed and merged by the end of the coding period.
  4. The candidate is technically very strong. This is judged through prior experience with contributing to ns-3 and/or other open-source projects, code samples, their past experience and through interviews conducted during the review process by the mentoring team.
  5. The candidate does not have any other commitments over the summer, and can devote 40 hours per-week as is required of GSOC candidates.
  6. The proposal has been discussed at length on the mailing list and/or IRC.
  7. The student has demonstrated some interest in involvement with ns-3 either before or after GSOC ends.

Signs of a bad proposal

  1. The proposal is a regurgitation of the project description and the recommended reading.
  2. Poor professional conduct in developing the application (extensive copy-pasting for instance).
  3. There is little evidence for the candidate's technical abilities.
  4. Very little mailing list and community engagement in developing the proposal.
  5. The candidate has other commitments for the summer (vacations, exams, coursework, another job etc.).
  6. The proposal or patch submitted for review is sloppy (in grammar, spelling, formatting, etc.)

Review Process

Pre-evaluation phase

The organisation admin makes a pass through all the proposals. The admin will mark as ignored the incomplete, out of scope, or very weak applications. These applications will not be considered for further review.

Timeframe: completed on April 14, 2019

First review phase

  1. All members of the selection committee will review the remaining applications. If there is an application that the reviewer would like to mentor (if selected), the reviewer should select 'Want to mentor'. Note that more than one person can select this; we can sort out details and overlaps later. If there is an application that the reviewer thinks is high enough quality to merit consideration for a final slot, and is not yet marked with a star, then the reviewer may mark it with a star.
  2. Do not remove a star that was placed by another reviewer.
  3. All reviewers are encouraged to make private notes of which of the starred proposals they believe are stronger than the others, and of strengths and weaknesses, for use in the next phase. We won't use the Google Internal Review form for this phase.

At the end of the first review phase, the starred applications will be taken into the next review phase.

Timeframe: April 14-16, 2019

Second review phase

Committee members will review again each starred proposal and will leave a comment on the internal review tool, indicating what they believe are strengths and weaknesses of the proposal and on the relative ranking of the proposal (with respect to other starred proposals). If any clarifications from student applicants are needed, they can be queried at this time.

Timeframe: April 17-18, 2019

Final phase

The selection committee will form a ranking of the starred proposals by consolidating inputs from the internal reviews, and try to achieve consensus or executive decision on the ranking and on how many slots to request from Google.

Timeframe: April 19-21, 2019

Feedback on applications

Students who do not get selected are welcome to contact the organisation admins to solicit feedback on their applications.

Confidentiality and professional responsibilities of selection process

With respect to confidentiality and professional responsibilities of the review process, the selection committee is expected to treat the process similar to a peer reviewed, technical program committee. Example guidelines are posted here.