https://www.nsnam.org/mediawiki/index.php?title=Reproducible_papers&feed=atom&action=historyReproducible papers - Revision history2024-03-29T06:17:44ZRevision history for this page on the wikiMediaWiki 1.24.1https://www.nsnam.org/mediawiki/index.php?title=Reproducible_papers&diff=9124&oldid=prevTomh: update guidance about reproducing past results2014-11-03T20:42:49Z<p>update guidance about reproducing past results</p>
<table class='diff diff-contentalign-left'>
<col class='diff-marker' />
<col class='diff-content' />
<col class='diff-marker' />
<col class='diff-content' />
<tr style='vertical-align: top;'>
<td colspan='2' style="background-color: white; color:black; text-align: center;">← Older revision</td>
<td colspan='2' style="background-color: white; color:black; text-align: center;">Revision as of 20:42, 3 November 2014</td>
</tr><tr><td colspan="2" class="diff-lineno">Line 13:</td>
<td colspan="2" class="diff-lineno">Line 13:</td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers.  In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.):</div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div>For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers.  In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.):</div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div># the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided</div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div># the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided<ins class="diffchange diffchange-inline">;</ins></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div># the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description</div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div># the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description<ins class="diffchange diffchange-inline">.  Of course reproducing prior work (using a not novel scenario), in order to validate a new implementation or model construction, is a valuable methodological tool, however interesting papers will go on to demonstrate or derive new results beyond the original work;</ins></div></td></tr>
<tr><td class='diff-marker'>−</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #ffe49c; vertical-align: top; white-space: pre-wrap;"><div># best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained</div></td><td class='diff-marker'>+</td><td style="color:black; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #a3d3ff; vertical-align: top; white-space: pre-wrap;"><div># best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained<ins class="diffchange diffchange-inline">; and</ins></div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div># information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3.  Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly.  </div></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"><div># information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3.  Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly.  </div></td></tr>
<tr><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td><td class='diff-marker'> </td><td style="background-color: #f9f9f9; color: #333333; font-size: 88%; border-style: solid; border-width: 1px 1px 1px 4px; border-radius: 0.33em; border-color: #e6e6e6; vertical-align: top; white-space: pre-wrap;"></td></tr>
</table>Tomhhttps://www.nsnam.org/mediawiki/index.php?title=Reproducible_papers&diff=9117&oldid=prevTomh: first draft of reproducible research2014-10-28T17:55:41Z<p>first draft of reproducible research</p>
<p><b>New page</b></p><div>{{TOC}}<br />
<br />
Back link to [http://www.nsnam.org/overview/wns3/ Workshop on ns-3]<br />
<br />
The ns-3 project encourages users to practice the principles of [http://reproducibleresearch.net/ reproducible research] when using ns-3, and in particular, when submitting academic papers that use ns-3. <br />
<br />
''"But my code is not good enough,"'' you may protest. Please read this [http://www.nature.com/news/2010/101013/full/467753a.html opinion piece in the journal Nature] to see why this is likely not the case and why you should consider to release what you have published.<br />
<br />
The ns-3 project aims to make it easy for you to release your code so that others can reproduce your results. We will even host your code on our servers if needed; please contact one of the maintainers if you desire to do this.<br />
<br />
In the field of networking, [http://reproducingnetworkresearch.wordpress.com/ Stanford has demonstrated] how this is achievable even in a class context.<br />
<br />
For our annual Workshop on ns-3, we provide the following soft requirements to our reviewers. In other words, the more that papers exhibit these characteristics, the higher they will score (given the other considerations such as scientific evaluation of the proposed experiment, novelty of the scenario and of the results, timeliness with research in the field, technical soundness of the discussion, quality of presentation, etc.):<br />
<br />
# the simulation setup/configuration shall be clearly described; if room is insufficient, external links should be provided<br />
# the simulation scenario should be novel in some way, and described in a way that others could replicate, either by providing the code or by providing enough description<br />
# best simulation practices should be followed, e.g., the modeling assumptions/choice of models shall be motivated, the confidence of the results shall be explained<br />
# information shall be provided on how to reproduce the results, e.g,mentioning the ns-3 version, ideally posting a link to custom code, or at least describing what are the modifications that have been made to ns-3. Some people make it as easy as possible (e.g. a shell script automating the simulation and plotting) to reproduce the figures directly. <br />
<br />
== Examples ==<br />
<br />
Below are some examples of papers that the project considers to follow these guidelines:<br />
* Mininet HiFi (2012): http://dl.acm.org/citation.cfm?doid=2413176.2413206<br />
* NEPI (2011) http://ieeexplore.ieee.org/xpl/login.jsp?tp=&arnumber=6064394<br />
* DCE (2013) http://dl.acm.org/citation.cfm?doid=2535372.2535374</div>Tomh