Difference between revisions of "GSOCSelectionProcess"

From Nsnam
Jump to: navigation, search
(What are we looking for in a good proposal?)
(16 intermediate revisions by the same user not shown)
Line 1: Line 1:
 +
{{TOC}}
 +
 
= ns-3 Google Summer of Code Selection Process =
 
= ns-3 Google Summer of Code Selection Process =
  
 
This page describes how the ns-3 organisation reviews student applications for the Google Summer of Code programme and narrows down on the final selections.
 
This page describes how the ns-3 organisation reviews student applications for the Google Summer of Code programme and narrows down on the final selections.
  
In summary, we will form a selection committee of mentors and maintainers who will review the submitted applications. In cases where we need to probe further for a candidate's technical skills, we will organise IRC interviews with the candidates. After the reviews and the interviews are completed, the committee will agree upon a final ranking of applications. The admins will then recommend the final ranked list of applications to Google.
+
In summary, we have formed a selection committee of mentors and maintainers who will review the submitted applications. In cases where we need to probe further for a candidate's technical skills, we will organise chat interviews or email exchanges with the candidates. After the reviews and the interviews are completed, the committee will agree upon a final ranking of applications. The admins will then recommend the final ranked list of applications to Google.
  
 
== Selection Committee ==
 
== Selection Committee ==
  
In 2018, the committee consists of:
+
In 2019, the committee consists of:
 
* Tom Henderson (org admin)
 
* Tom Henderson (org admin)
 
* Tommaso Pecorella (org admin)
 
* Tommaso Pecorella (org admin)
 +
* Abhijith Anilkumar
 +
* Zoraze Ali
 +
* Biljana Bojovic
 +
* Ankit Deepak
 +
* Lorenza Giupponi
 +
* Alexander Krotov
 
* Natale Patriciello
 
* Natale Patriciello
* Dizhi Zhou
 
 
* Mohit Tahiliani
 
* Mohit Tahiliani
* Matthieu Coudron
+
* Dizhi Zhou
  
 
The selection committee will have access to all applications and be able to leave public and private comments, and to score the applications.
 
The selection committee will have access to all applications and be able to leave public and private comments, and to score the applications.
 
== Review Committee Checklist ==
 
 
# Sign in to the Google Summer of Code site as a mentor, and pick the ns-3 organisation(the organisation name is "The ns-3 network simulator"). You won't be able to see the applications otherwise.
 
# If you are the concerned mentor for a particular application, please select "Wish to mentor" on the left bar for the application so that we can assign you the application.
 
# Review *every* application and assign a score to each one. Please see below for the scoring guidelines.
 
# After you've reviewed all applications, leave behind a *private* comment on your top 3 proposal picks, indicating your reasons for doing so.
 
# If you're trying to leave a review for other reviewers to see, please make sure *you don't* leave a public comment.
 
  
 
== Scoring System ==
 
== Scoring System ==
  
On Melange, every reviewer should assign a score to every application. The scoring system is as follows
+
In 2019, Google has changed the review system and no longer uses or tracks numerical scoringHowever, they want projects to select 'outstanding' (or at the very least, 'very good') applications; how these are defined are left up to the individual projects.
 
+
* 1  : Spam application. **For mentors of the respective projects and admins ONLY**
+
* 2  : Ignore proposal. **For mentors of the respective projects and admins ONLY**
+
 
+
* 3 : One of the worst proposals, other reviewers should ignore it
+
* 4  : This definitely doesn't deserve to get into GSoC
+
* 5  : I'd rather this didn't get a slot
+
* 6* : Don't care one way or the other whether it's funded
+
* 7  : Mild enthusiasm. Fairly cool but niche, or average proposal
+
* 8  : Student seems quite good, and the project is well thought out
+
* 9  : This is one of my favourites
+
 
+
* 10 : Highly recommended. **For mentors of the respective projects and admins ONLY**
+
* 11 : Strongly recommended. **For mentors of the respective projects and admins ONLY***
+
 
+
6 should be your default score
+
  
 +
ns-3 will likely be allocated at most four or five slots, and we will not know in advance how many we will receive.  We will therefore need to come to consensus or executive decision on what ranking we make for the top applications.  We can use the 'star' system to flag those applications that a reviewer believes should be in consideration for the top four or five.  Additional details that will allow us to order proposals will need to be discussed outside of the Google tool.
  
 
== What are we looking for in a good proposal? ==
 
== What are we looking for in a good proposal? ==
  
Our objective is to select students who are potential future maintainers of the ns-3 project. Over the summer, the community will be investing time and effort in mentoring the students to achieve this end. That said, this is what a good proposal will look like:
+
Our objective is to select students who have convinced us that they should be able to deliver a nice project and who are potential future contributors to the ns-3 project. Over the summer, the community will be investing time and effort in mentoring the students to achieve this end. That said, this is what a good proposal will look like:
  
 
# A detailed proposal that sets a realistic list of deliverables, demonstrates a clear path towards achieving the same, is complete, and technically sound.
 
# A detailed proposal that sets a realistic list of deliverables, demonstrates a clear path towards achieving the same, is complete, and technically sound.
Line 59: Line 43:
  
 
== Signs of a bad proposal ==
 
== Signs of a bad proposal ==
 
  
 
# The proposal is a regurgitation of the project description and the recommended reading.
 
# The proposal is a regurgitation of the project description and the recommended reading.
Line 66: Line 49:
 
# Very little mailing list and community engagement in developing the proposal.
 
# Very little mailing list and community engagement in developing the proposal.
 
# The candidate has other commitments for the summer (vacations, exams, coursework, another job etc.).
 
# The candidate has other commitments for the summer (vacations, exams, coursework, another job etc.).
# Previous failures within GSOC.
+
# The proposal or patch submitted for review is sloppy (in grammar, spelling, formatting, etc.)
 
+
  
 
= Review Process =
 
= Review Process =
Line 73: Line 55:
 
== Pre-evaluation phase ==
 
== Pre-evaluation phase ==
  
The organisation admin makes a pass through all the proposals. The admin will assign a score of 1 to spam, very weak applications, applications that do not fulfill the patch requirement and applications that are copy-paste jobs. These applications '''will not be''' considered for further review.
+
The organisation admin makes a pass through all the proposals. The admin will mark as ignored the incomplete, out of scope, or very weak applications. These applications will not be considered for further review.
  
 
+
'''Timeframe''': completed on April 14, 2019
'''Timeframe''': March 27-April 23, 2018
+
  
 
== First review phase ==
 
== First review phase ==
  
# All members of the selection committee will assign scores to the applications based on the scoring system mentioned above. Note, the scores of 1, 2, 11, and 12 are reserved for the admins and the mentors of the corresponding projects.
+
# All members of the selection committee will review the remaining applications. If there is an application that the reviewer would like to mentor (if selected), the reviewer should select 'Want to mentor'.  Note that more than one person can select this; we can sort out details and overlaps later.
# All reviewers must post a private comment for each of their top 3 proposal picks explaining their score. They are free to post comments for other applications as well.
+
# If there is an application that the reviewer thinks is high enough quality to merit consideration for a final slot, then the reviewer may leave a comment in the Internal Review form that the application should be further considered.
  
At the end of the first review phase, admins will make a pass through the applications to ensure that scores are assigned equally across all proposals. This is for cases where some mentors are more lenient with awarding scores than others, and also to avoid situations of vested interests. Admins will adjust scores accordingly to balance against such situations.
+
At the end of the first review phase, the applications drawing interest from the committee will be taken into the next review phase.
 +
 
 +
'''Timeframe:''' April 14-16, 2019
  
 
== Second review phase ==
 
== Second review phase ==
  
Admins will present a list of second round candidates for discussion by the committee. The committee may organise interviews with the candidates to probe for technical skill and to clarify aspects of the proposal.
+
Committee members will review again each proposal still under consideration and, if they have not done so already, will leave a comment on the internal review tool, indicating what they believe are strengths and weaknesses of the proposal and on the relative ranking of the proposal (with respect to other proposals).  If any clarifications from student applicants are needed, they can be queried at this time.
 +
 
 +
'''Timeframe:''' April 17-18, 2019
  
 
== Final phase ==
 
== Final phase ==
  
Outcome of second review phase is discussed, and the final ranking list of candidates is decided. The organisation admin will then lock the final set of candidates and make the recommendation to Google.
+
The selection committee will form a ranking of the remaining proposals by consolidating inputs from the internal reviews, and try to achieve consensus or executive decision on the ranking and on how many slots to request from Google.
 +
 
 +
'''Timeframe:''' April 19-21, 2019
  
 
= Feedback on applications =
 
= Feedback on applications =

Revision as of 02:04, 15 April 2019

Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects

Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers

ns-3 Google Summer of Code Selection Process

This page describes how the ns-3 organisation reviews student applications for the Google Summer of Code programme and narrows down on the final selections.

In summary, we have formed a selection committee of mentors and maintainers who will review the submitted applications. In cases where we need to probe further for a candidate's technical skills, we will organise chat interviews or email exchanges with the candidates. After the reviews and the interviews are completed, the committee will agree upon a final ranking of applications. The admins will then recommend the final ranked list of applications to Google.

Selection Committee

In 2019, the committee consists of:

  • Tom Henderson (org admin)
  • Tommaso Pecorella (org admin)
  • Abhijith Anilkumar
  • Zoraze Ali
  • Biljana Bojovic
  • Ankit Deepak
  • Lorenza Giupponi
  • Alexander Krotov
  • Natale Patriciello
  • Mohit Tahiliani
  • Dizhi Zhou

The selection committee will have access to all applications and be able to leave public and private comments, and to score the applications.

Scoring System

In 2019, Google has changed the review system and no longer uses or tracks numerical scoring. However, they want projects to select 'outstanding' (or at the very least, 'very good') applications; how these are defined are left up to the individual projects.

ns-3 will likely be allocated at most four or five slots, and we will not know in advance how many we will receive. We will therefore need to come to consensus or executive decision on what ranking we make for the top applications. We can use the 'star' system to flag those applications that a reviewer believes should be in consideration for the top four or five. Additional details that will allow us to order proposals will need to be discussed outside of the Google tool.

What are we looking for in a good proposal?

Our objective is to select students who have convinced us that they should be able to deliver a nice project and who are potential future contributors to the ns-3 project. Over the summer, the community will be investing time and effort in mentoring the students to achieve this end. That said, this is what a good proposal will look like:

  1. A detailed proposal that sets a realistic list of deliverables, demonstrates a clear path towards achieving the same, is complete, and technically sound.
  2. The proposal is an original document authored by the candidate, and is not a copy paste job.
  3. The proposal is scoped such that it has a strong chance of being successfully completed and merged by the end of the coding period.
  4. The candidate is technically very strong. This is judged through prior experience with contributing to ns-3 and/or other open-source projects, code samples, their past experience and through interviews conducted during the review process by the mentoring team.
  5. The candidate does not have any other commitments over the summer, and can devote 40 hours per-week as is required of GSOC candidates.
  6. The proposal has been discussed at length on the mailing list and/or IRC.
  7. The student has demonstrated some interest in involvement with ns-3 either before or after GSOC ends.

Signs of a bad proposal

  1. The proposal is a regurgitation of the project description and the recommended reading.
  2. Poor professional conduct in developing the application (extensive copy-pasting for instance).
  3. There is little evidence for the candidate's technical abilities.
  4. Very little mailing list and community engagement in developing the proposal.
  5. The candidate has other commitments for the summer (vacations, exams, coursework, another job etc.).
  6. The proposal or patch submitted for review is sloppy (in grammar, spelling, formatting, etc.)

Review Process

Pre-evaluation phase

The organisation admin makes a pass through all the proposals. The admin will mark as ignored the incomplete, out of scope, or very weak applications. These applications will not be considered for further review.

Timeframe: completed on April 14, 2019

First review phase

  1. All members of the selection committee will review the remaining applications. If there is an application that the reviewer would like to mentor (if selected), the reviewer should select 'Want to mentor'. Note that more than one person can select this; we can sort out details and overlaps later.
  2. If there is an application that the reviewer thinks is high enough quality to merit consideration for a final slot, then the reviewer may leave a comment in the Internal Review form that the application should be further considered.

At the end of the first review phase, the applications drawing interest from the committee will be taken into the next review phase.

Timeframe: April 14-16, 2019

Second review phase

Committee members will review again each proposal still under consideration and, if they have not done so already, will leave a comment on the internal review tool, indicating what they believe are strengths and weaknesses of the proposal and on the relative ranking of the proposal (with respect to other proposals). If any clarifications from student applicants are needed, they can be queried at this time.

Timeframe: April 17-18, 2019

Final phase

The selection committee will form a ranking of the remaining proposals by consolidating inputs from the internal reviews, and try to achieve consensus or executive decision on the ranking and on how many slots to request from Google.

Timeframe: April 19-21, 2019

Feedback on applications

Students who do not get selected are welcome to contact the organisation admins to solicit feedback on their applications.

Confidentiality and professional responsibilities of selection process

With respect to confidentiality and professional responsibilities of the review process, the selection committee is expected to treat the process similar to a peer reviewed, technical program committee. Example guidelines are posted here.