Difference between revisions of "Reproducible papers"
(first draft of reproducible research) |
(update guidance about reproducing past results) |
||
| Line 13: | Line 13: | ||
For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers. In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.): | For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers. In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.): | ||
| − | # the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided | + | # the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided; |
| − | # the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description | + | # the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description. Of course reproducing prior work (using a not novel scenario), in order to validate a new implementation or model construction, is a valuable methodological tool, however interesting papers will go on to demonstrate or derive new results beyond the original work; |
| − | # best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained | + | # best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained; and |
# information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3. Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly. | # information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3. Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly. | ||
Latest revision as of 20:42, 3 November 2014
Main Page - Current Development - Developer FAQ - Tools - Related Projects - Project Ideas - Summer Projects
Installation - Troubleshooting - User FAQ - HOWTOs - Samples - Models - Education - Contributed Code - Papers
Back link to Workshop on ns-3
The ns-3 project encourages users to practice the principles of reproducible research when using ns-3, and in particular, when submitting academic papers that use ns-3.
"But my code is not good enough," you may protest. Please read this opinion piece in the journal Nature to see why this is likely not the case and why you should consider to release what you have published.
The ns-3 project aims to make it easy for you to release your code so that others can reproduce your results. We will even host your code on our servers if needed; please contact one of the maintainers if you desire to do this.
In the field of networking, Stanford has demonstrated how this is achievable even in a class context.
For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers. In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.):
- the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided;
- the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description. Of course reproducing prior work (using a not novel scenario), in order to validate a new implementation or model construction, is a valuable methodological tool, however interesting papers will go on to demonstrate or derive new results beyond the original work;
- best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained; and
- information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3. Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly.
Examples
Below are some examples of papers that the project considers to follow these guidelines:
- Mininet HiFi (2012): http://dl.acm.org/citation.cfm?doid=2413176.2413206
- NEPI (2011) http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6064394
- DCE (2013) http://dl.acm.org/citation.cfm?doid=2535372.2535374
