https://www.nsnam.org/mediawiki/api.php?action=feedcontributions&user=Watrous&feedformat=atomNsnam - User contributions [en]2024-03-28T13:40:09ZUser contributionsMediaWiki 1.24.1https://www.nsnam.org/mediawiki/index.php?title=Ns-3.16&diff=7080Ns-3.162012-12-07T18:30:57Z<p>Watrous: /* Bugs being worked */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.16. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
* Wed Nov 21 -- new feature freeze<br />
** keeping open for possible BRITE and LTE scheduler inclusion<br />
* Fri Dec 7 -- code freeze on ns-3-dev<br />
* Wed Dec 12 -- ns-3.16 released<br />
<br />
= Proposed supported platforms =<br />
<br />
* Linux Ubuntu 12.10<br />
* Linux Fedora 17 (also 18 beta)<br />
* Linux Ubuntu 12.04 LTS<br />
* Ubuntu 10.04.3 LTS<br />
* Mac OS X Mountain Lion 10.8.2<br />
* FreeBSD 9<br />
* FreeBSD 8.2<br />
<br />
= Packaging =<br />
<br />
Source tarball ns-allinone-3.16.tar.bz2.<br />
<br />
Tarball unpacks to ns-allinone-3.16 which contains, at top level:<br />
<br />
bake bakefile.xml build.py doc generate-binary.py<br />
bakeconf.xml bake.py examples source test <br />
<br />
The source/ directory contains:<br />
<br />
click netanim nsc pybindgen <br />
gccxml-ns3 ns-3.16 openflow-ns3 pygccxml<br />
<br />
The new user will either call 'build.py', which will be a wrapper around bake, or else directly call 'bake.py build' and will build all ns-3 components. From that point forward, the user may recurse into the ns-3.16 directory and work from there using ./waf as usual (if all work is ns-3-based and just uses the libraries such nsc and netanim built initially from build.py). If the user wants to work with additional packages (Direct Code Execution), he/she will reconfigure at the bake.py level and download additional components. If the user wants to edit or patch the ns-3 supporting modules (e.g. nsc or netanim) he/she will need to do more work at the bake level.<br />
<br />
= new features =<br />
<br />
* bake: http://codereview.appspot.com/6761057/<br />
* LTE MAC Schedulers by Dizhi Zhou (GSoC 2012): http://codereview.appspot.com/6591047<br />
* BRITE topology generator integration: http://codereview.appspot.com/6499120/<br />
<br />
= Bugs being worked =<br />
<br />
* 555: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=555 DCF immediate access bug] Daniel L.<br />
* 582: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/ <- being held out of ns-3.16<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* 954: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=954 changing the simulation time resolution] Peter Barnes<br />
* 1342: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1342 tcp test case failure] Brian Swenson<br />
* 1456: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1456 update openflow distribution] Tom Henderson<br />
* 1501: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1501 Tcp test crash] Brian Swenson<br />
* 1502: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1502 Shutdown on tcp socket seems to misbehave] Brian Swenson<br />
* <s>1516: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1516 GtkConfigStore does not show read-only attributes] Mitch Watrous</s><br />
* 1520: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1520 config paths not documented in Doxygen for CsmaChannel (possibly others)] Mitch Watrous<br />
* 1522: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1522 jitter needed on Ipv4 ARP WaitReply timer] Daniel L.<br />
* 1525: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1525 Linker error with mpi on Mac 10.8] Peter Barnes/Brian Swenson<br />
* 1531: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1531 Crash when using NS_LOG in destructors of static objects] Peter Barnes<br />
* 1540: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1540 bake not building click correctly] Daniel Camara/Mitch Watrous<br />
* 1542: [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1542 adapt waf to find bake-built libraries] Mitch Watrous</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6883Ns-3.152012-08-06T18:01:43Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
*** '''olsr:''' http://codereview.appspot.com/6450073<br />
*** '''uan:''' http://codereview.appspot.com/6443073<br />
*** '''wifi:''' http://codereview.appspot.com/6450087<br />
*** '''wimax:''' http://codereview.appspot.com/6446093<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6882Ns-3.152012-08-03T18:29:17Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
*** '''olsr:''' http://codereview.appspot.com/6450073<br />
*** '''uan:''' http://codereview.appspot.com/6443073<br />
*** '''wifi:''' http://codereview.appspot.com/6450087<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6881Ns-3.152012-08-01T17:57:26Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
*** '''olsr:''' http://codereview.appspot.com/6450073<br />
*** '''uan:''' http://codereview.appspot.com/6443073<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6880Ns-3.152012-07-31T22:27:01Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
*** '''olsr:''' http://codereview.appspot.com/6450073<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6879Ns-3.152012-07-31T22:25:09Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
*** '''olsr:''' http://codereview.appspot.com/6450073<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6878Ns-3.152012-07-31T18:57:38Z<p>Watrous: </p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
*** '''netanim:''' http://codereview.appspot.com/6453067<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6877Ns-3.152012-07-27T18:52:50Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''csma:''' http://codereview.appspot.com/6448069<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6875Ns-3.152012-07-27T00:10:09Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''csma:''' http://codereview.appspot.com/6443053<br />
*** '''internet:''' http://codereview.appspot.com/6442056<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6874Ns-3.152012-07-26T22:24:01Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
*** '''csma:''' http://codereview.appspot.com/6443053<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6873Ns-3.152012-07-26T01:04:06Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
*** '''dsdv:''' http://codereview.appspot.com/6444050<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.15&diff=6872Ns-3.152012-07-25T23:03:29Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.15. No release manager is yet appointed. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
# Wed Jul 25 -- new feature freeze<br />
# Wed Aug 8 -- code freeze on ns-3-dev<br />
# Wed Aug 15 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
TBD<br />
<br />
= Packaging =<br />
<br />
TBD<br />
<br />
= new feature reviews =<br />
<br />
== Aiming for ns-3.15 ==<br />
<br />
* Brian Panneton's antenna model updates http://mailman.isi.edu/pipermail/ns-developers/2012-April/010322.html<br />
* [http://codereview.appspot.com/6201059/ New IEEE 802.11b indoor wireless channel models for (HMM and BEAR)]<br />
* [http://codereview.appspot.com/6192052/ Longley-Rice and ITU terrain-aware propagation models]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here], needs updating<br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
* Switched Ethernet device: http://codereview.appspot.com/5615049/<br />
* HTTP traffic generator: http://codereview.appspot.com/4940041/<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <br />
** Patchset 1: The basic change to the implementation and API: http://codereview.appspot.com/6305104/<br />
** Patchsets 2+: Changes to the models/helpers<br />
*** '''aodv:''' http://codereview.appspot.com/6436052<br />
*** '''applications:''' http://codereview.appspot.com/6421061<br />
*** '''dsr:''' http://codereview.appspot.com/6449046<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582 Tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize] Peter Barnes<br />
** Patchset 1: Add class Hash: generic hash function interface http://codereview.appspot.com/6357056/<br />
** Patchset 2: Add hashes to TypeId. http://codereview.appspot.com/6344063/<br />
** Patchset 3: Refactor PacketTagList http://codereview.appspot.com/6354061/<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* Object::Stop patch http://mailman.isi.edu/pipermail/ns-developers/2012-May/010392.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
[https://www.nsnam.org/bugzilla/buglist.cgi?bug_status=__open__&content=&product=&query_format=specific&order=bug_id%20DESC&query_based_on= Open bugs] will be worked on a best-effort basis.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.14&diff=6754Ns-3.142012-05-01T18:18:00Z<p>Watrous: /* Simulation core */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.14. Tom Henderson and Tommaso Pecorella will be managing the release. The ns-3 release process is listed [http://www.nsnam.org/developers/release-process/ here] and [[Release Process | here]].<br />
<br />
= Release schedule =<br />
<br />
<s># Wed Apr 4 -- new feature freeze, enter bug fixing stage <- considering to slip this to include additional features<br />
# Wed Apr 18 -- code freeze on ns-3-dev<br />
# Wed Apr 25 -- ns-3.14 released</s><br />
<br />
Here are the proposed new dates:<br />
<br />
# Fri Apr 27 -- new feature freeze<br />
# Wed May 9 -- code freeze on ns-3-dev<br />
# Wed May 16 -- ns-3.14 released<br />
<br />
= Proposed supported platforms =<br />
<br />
* Fedora 16, 17<br />
* Ubuntu 11.10, 12.04<br />
* Centos (5.8 and 6.2?)<br />
* OS X Snow Leopard and Lion (maybe Mountain Lion?)<br />
* FreeBSD 8.2, 9.0 (also with clang)<br />
* Cygwin?<br />
<br />
= Packaging =<br />
<br />
* traditional ns-allinone-3.14.tar.bz2 packaging<br />
* Fedora packaging as detailed by Vedran<br />
<br />
= new feature reviews =<br />
<br />
== Still aiming for ns-3.14 ==<br />
* <s>[http://codereview.appspot.com/4889046/ RED queues], [http://mailman.isi.edu/pipermail/ns-developers/2011-August/009255.html mailing list post] -> merged to ns-3-dev and is being kept there for now so as to not disturb the repository, but requires some relicensing of original BSD-licensed source before we should release it. This is expected to be done by 13 April.</s><br />
* LTE, buildings, and antenna models: several review requests are [http://mailman.isi.edu/pipermail/ns-developers/2012-March/010020.html here]<br />
** <s>Antenna patch requires resolution of comments in the Rietveld issue tracker</s><br />
** LTE merge is blocked by resolution of [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1352 this issue]<br />
** <s>Buildings merge is being refactored for 1) the separation of pathloss models, and 2) the re-use of position allocators for the buildings allocator</s><br />
* [http://codereview.appspot.com/4823051/ Dynamic Source Routing (DSR)] -> author is responding to some final comments<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=454 TCP Echo] Code review [http://codereview.appspot.com/5654053/ here]<br />
* netanim (added to download.py). Proposed patch [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1403 here]<br />
<br />
== Slipping past ns-3.14 ==<br />
* [http://codereview.appspot.com/5452045/ TCP Options (just the encoding of the options in the header)] <br />
* [http://codereview.appspot.com/4940041/ HTTP traffic generator] -> reviewed, needs revisions<br />
* [http://codereview.appspot.com/4875052/ DelayBox] -> author responding to comments<br />
* [http://codereview.appspot.com/4747046/ TDMA model] -> sliding to ns-3.15?<br />
* [http://codereview.appspot.com/4517125/ BRITE Integration] -> sliding to ns-3.15?<br />
* [http://codereview.appspot.com/4997043/ Poisson Pareto Burst Process] -> sliding to ns-3.15?, author responding to comments<br />
* [http://codereview.appspot.com/5343044/ CLWPR (Cross Layer, Weighted, Position-based, Routing)] -> sliding to future<br />
* [http://codereview.appspot.com/5314055/ CSMA module improvements] -> sliding to future<br />
* [http://codereview.appspot.com/5401042 GPSR (Greedy Perimeter Stateless Routing)] -> Posted for pre-review, presently missing test suite.<br />
* [http://codereview.appspot.com/5615049/ Switched Ethernet Device] <br />
* [http://codereview.appspot.com/5552055/ Finishing ns-3-click-mac extensions] -> Blocked by queue API resolution<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2012-January/009821.html Antenna Model patch] Author must rebase patch against Nicola's patch.<br />
* [http://code.nsnam.org/pdbarnes/packet-tag-list/ Packet Tag List]<br />
<br />
NSOC/GSOC code merges (seem to be slipping out of ns-3.14 timeframe):<br />
* [http://codereview.appspot.com/4685048/ Monitor mode support] ([http://codereview.appspot.com/5552055/ Update from Bjorn]) <br />
* [http://codereview.appspot.com/4901055/ LTE RRC extension] <br />
* [http://groups.google.com/group/ns-3-reviews/browse_thread/thread/512bf466d3cd5ec0?pli=1 UAN Mobility Model merge (from previous GSOC)] <br />
** blocked on resolving changes to WaypointMobilityModel API<br />
<br />
* [http://code.nsnam.org/safe/ns-3-rng/file/dbdb95ae0805/src/core/model/random-variable-stream.h Random Variable Stream]<br />
** need to generate code review and may slip to ns-3.15; plan is to not touch RandomVariable or any models for ns-3.14<br />
<br />
= Bugs being worked =<br />
<br />
== Simulation core ==<br />
<br />
Bugs or issues involving things that are not related to protocol or channel models.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=101 101 RandomVariable enhancements] Mitch Watrous and Tom Henderson <- likely slipping to ns-3.15<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938 Doxygen cleanup] Vedran Miletić and Tom Henderson<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=962 962 path to attributes has bogus entries] Nicola Baldo</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1170 1170 best practices for unused variables] John Abraham and Andrey Mazo</s><br />
** closing this bug is hung up on a WordPress formatting issue (introduced by trying to extend the coding style section)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] Claudio Freire<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1327 1327-1331 packaging related enhancements] Vedran Miletić and Gustavo Carneiro</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1313 1313 Stddev (average.h) returning NaN] Mitch Watrous</s> <br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback] Ruben Merz, Frederic Urbani, and Tom Henderson<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
<br />
== Bugs in ns-3 models ==<br />
<br />
* <s> [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1333 1333 Wrong initial estimation after reset in RttEstimator] Adrian Tam<br />
** Bug 1351 seems to be a duplicate </s><br />
* TCP issues: 1399, 1405, 1340-42, 1359, <br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1053 1053 Need better error diagnostics in ns2-mobility-trace example] Anirudh<br />
* Tommaso has compiled a list of other bugs being worked (454, 1384, 1308, 1345, 1385, 1362)<br />
* Alignment of IPv6 address helper API with IPv4 address helper API (no patch yet)</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Developer_FAQ&diff=6558Developer FAQ2012-02-17T22:46:45Z<p>Watrous: /* How to add new ns-3 modules */</p>
<hr />
<div>{{TOC}}<br />
<br />
== Mercurial repository layout for developers ==<br />
<br />
# In your home dir on code.nsnam.org, you will find a new directory named "<code>repositories/username</code>". e.g.: <code>/home/tomh/repositories/tomh</code>. If you create a repository in this directory, it will appear automatically on http://code.nsnam.org<br />
#* Note: To enable this for new users, edit /var/www/cgi-hg/hgweb.config<br />
# You can obviously ssh to your personal account and manage these repositories<br />
# You can push to these repositories with the command: <code>hg push ssh://tomh@code.nsnam.org/repositories/tomh/ns-3-com</code><br />
# You can pull with the usual commands: <code>hg clone http://code.nsnam.org/tomh/ns-3-com</code><br />
# If you want to allow another user to push into your repository, all you have to do is change the unix permissions of your repository to allow this user write permissions. This means that if you want to give everyone write permissions, you can "<code>chmod -R g +rw /home/tomh/repositories/tomh/ns-3-com/</code>". If you want to allow only a smaller subset of users to push, we will need to create unix group which matches this subset<br />
# The push command for the main tree is still: <code>hg push ssh://code@code.nsnam.org/repos/ns-3-dev</code><br />
# '''New developers, please read this:''' When creating a new repository, do not "hg clone" it into your directory on code.nsnam.org (which will generate a big ns-commits mail message); instead, just copy (cp -r) or rsync it to your local "/home/username/repositories/username" directory.<br />
<br />
=== Mercurial tips ===<br />
<br />
# '''How to undo a commit''': Let's suppose you are working on a private repository and you check something in, but some other files were inadvertently checked in, and you want to revert and start over. There are two ways to do this:<br />
## <code>hg revert</code>: This can be used to revert the repository to a previous revision number. For example, to revert to changeset number #1000, type <code>hg revert -r 1000 --all</code>. This does '''not''' remove your checkin from the repository history. For example, if your mistaken checkin was number 1001, and you revert back to 1000 and then commit, you will be at changeset number 1002 now even though the code matches what was in there at changeset 1000.<br />
## <code>hg rollback</code>: This can be used to completely wipe clean the last transaction only (commit, import, push, pull). Use with care-- cannot be undone. <br />
# '''How to rename a file''': <code>hg rename old-file-name new-file-name</code> This is preferable to adding the new file name and removing the old file name, because it preserves revision history. Don't forget to commit once you are done.<br />
# '''How to merge a branch''': If you have forked a branch repository, have worked on it, and are ready to merge it back to ns-3-dev, here are the steps to take (Also, read [http://hgbook.red-bean.com/hgbookch3.html this chapter] to better understand how the mercurial source tree is structured when merging is occurring):<br />
##''cd into your branch''<br />
##<code>hg pull http://code.nsnam.org/ns-3-dev</code><br />
##<code>hg merge</code><br />
##''resolve all of the merge issues, if any, and confirm that it builds and validates''<br />
##<code>hg ci -m"merge your-branch-name with tip"</code><br />
##<code>hg push ssh://code@code.nsnam.org/repos/ns-3-dev</code><br />
# '''How to create patches for circulation''': <code>hg export tip</code> Suppose you don't have write access to the main repositories, but have some patches you'd like to circulate. This command outputs out all the diffs for the latest changeset you committed into your local repository. Simply redirect this output to a file, and you can circulate your patch for consideration. This is for when you have committed changesets. If you would like to export uncommitted changes as a patch, use: <code>hg diff</code> This gets the diffs of all the uncommitted changes versus what is checked into the repository. Redirect to a file for circulation among developers, or for inclusion with a bug report, etc.<br />
# '''Save a push URL so that you don't need to enter it each time''': If you are tired of having to specify the same ssh URL each time when you go to push, you can specify a default push location in the branch's local hgrc.<br />
##''cd into your branch''<br />
##<code>vi .hg/hgrc</code><br />
##''add the following line to the [paths] section''<br />
##<code>default-push = ssh://username@code.nsnam.org/repositories/username/ns-3-reponame</code><br />
##<code>hg push</code> should now "just work" without the URL<br />
# '''How to fix two-headed repositories''': Note, please never check in code by forcing it (with hg push -f option). This will cause the repository to have multiple heads. If the repository ends up with two heads, this can fix it:<br />
##<code>hg merge</code><br />
##<code>hg commit -m"merge two heads"</code><br />
##<code>hg push ... </code><br />
<br />
== The WAF build system ==<br />
See also the [http://www.nsnam.org/wiki/index.php/User_FAQ#WAF_.28new_build_process.29 Waf User FAQ].<br />
<br />
=== Obtaining WAF ===<br />
A snapshot of WAF is distributed with ns-3 releases and mercurial branches. This snapshopt has been tested to work correctly with ns-3, although the ''trunk'' version from the main WAF repository usually works equally well.<br />
<br />
=== Documentation resources ===<br />
There is a [http://freehackers.org/~tnagy/wafbook/index.html WAF book]. Some useful tips can be found in the [http://code.google.com/p/waf/w/list WAF Wiki]. Finally, there is a plethora of examples distributed in WAF itself, in the 'demos' directory.<br />
<br />
=== How to add new ns-3 modules ===<br />
<br />
Ns-3 is organized as a set of ''modules''. Each module is built as a set of object files, has a name, may depend on other modules, and installs a specific set of ''public header files''.<br />
<br />
Starting with version 3.11, ns-3 went to a modular directory structure. The following chapter in the ns-3 Manual shows how to add a module in the new directory structure:<br />
<br />
:[http://www.nsnam.org/docs/release/3.13/manual/html/new-modules.html Adding a New Module to ns-3 (version 3.11 or later)]<br />
<br />
If you are using a version of ns-3 earlier than 3.11, then follow the instructions in this FAQ.<br />
<br />
To add a new ns-3 module to the WAF build system, begin by creating a directory under the <code>src/</code> subtree, with the source files inside. We will use ''p2p'' module as example here. Each module needs to define a <code>wscript</code> file. For instance let us see what <code>src/devices/point-to-point/wscript</code> contains:<br />
<pre><br />
## -*- Mode: python; py-indent-offset: 4; indent-tabs-mode: nil; coding: utf-8; -*-<br />
<br />
def build(bld):<br />
module = bld.create_ns3_module('point-to-point', ['node'])<br />
module.source = [<br />
'point-to-point-net-device.cc',<br />
'point-to-point-channel.cc',<br />
'point-to-point-topology.cc',<br />
]<br />
headers = bld.create_obj('ns3header')<br />
headers.module = 'point-to-point'<br />
headers.source = [<br />
'point-to-point-net-device.h',<br />
'point-to-point-channel.h',<br />
'point-to-point-topology.h',<br />
]<br />
</pre><br />
<br />
A wscript file is basically a special python module. The main entry point to this module is the <code>build(bld)</code> python function.<br />
<br />
In the code above, the ''module'' variable represents a ns-3 module; internally it is a WAF 'objects' build object that will be linked to be come part of the ns3 library. It is created by calling a special method bld.create_ns3_module, whose first parameter is the name of the module, and the second parameter is a list of other modules that this module depends on. Additionally, module.sources has to be set to the list of source files (excluding header files) that constitute the module. Warning: beware that the name of the module must match the name of the directory where it is built. In this case, the module is in 'src/devices/point-to-point', so the module name must be 'point-to-point'.<br />
<br />
There is usually also a ''headers'' object. It is used to declare public header files. These files are copied to the build directory, and can be included from any module or program with <code>#include "ns3/header-name.h"</code>.<br />
<br />
A final step, after the wscript file is created, is to register it. Open the file <code>src/wscript</code> and add the new module to the <code>all_modules</code> list variable:<br />
<pre><br />
all_modules = (<br />
'core',<br />
'common',<br />
'simulator',<br />
'node',<br />
'internet-node',<br />
'devices/point-to-point', # <---- example here<br />
'applications',<br />
)<br />
</pre><br />
<br />
=== Adding programs ===<br />
Use the special method bld.create_ns3_program(name, [...dependencies...]). Example:<br />
<pre><br />
obj = bld.create_ns3_program('main-simple',<br />
['node', 'internet-node', 'applications'])<br />
obj.source = 'main-simple.cc'<br />
</pre><br />
<br />
== Generating new python bindings ==<br />
<br />
See the [http://www.nsnam.org/wiki/index.php/NS-3_Python_Bindings ns-3 python wiki page].<br />
<br />
== Release Process ==<br />
<br />
[[Release Process]]<br />
<br />
== Mercurial ==<br />
<br />
* Adrian Tam's [http://mercurial.selenic.com/wiki/QuickReferenceCardsAndCheatSheets mercurial cheatsheet]<br />
* [http://wiki.pylonshq.com/display/pylonscookbook/Mercurial+for+Subversion+Users Mercurial for subversion users]<br />
<br />
=== Your .hgrc file ===<br />
<br />
Each mercurial checkin is made by a user. If there is no .hgrc configuration file in the user's home directory, mercurial will default to using the accountname@hostname. This leads to commit lines like the following:<br />
<br />
changeset: 7:e53ac3c458e9<br />
user: tomh@powerbook.local<br />
<br />
To avoid this, and have it print something nicer, like<br />
<br />
changeset: 7:e53ac3c458e9<br />
user: Tom Henderson <tomh@tomh.org><br />
<br />
you need to add an .hgrc file to your home directory, such as follows:<br />
<br />
[ui]<br />
username = Tom Henderson <tomh@tomh.org><br />
<br />
or, for each checkin, you will need to specify the user string manually, such as:<br />
<br />
hg commit -u"Tom Henderson <tomh@tomh.org>" -m"commit message"<br />
<br />
If you want to commit a change on behalf of another user, such as attributing a bug fix to the original author, you can use the above -u command to override what is in your .hgrc file<br />
<br />
== Coding style == <br />
<br />
Of course, you need to follow the ns-3 coding style: http://www.nsnam.org/codingstyle.html<br />
<br />
=== If you use emacs ===<br />
<br />
All ns-3 files include this:<br />
<pre><br />
/* -*- Mode:C++; c-file-style:"gnu"; indent-tabs-mode:nil; -*- */<br />
</pre><br />
<br />
The "gnu" indentation style mostly conforms to the ns3 coding guidelines, but has a few quirks. One of these is that code after "namespace ns3 {" is indented, while the ns3 coding style says it should not. One workaround is typing the following command for every open buffer:<br />
<pre><br />
C-c C-o innamespace <ret> 0 <ret><br />
</pre><br />
Fixing this issue permanently would require using a different modeline, as discussed [http://www.nabble.com/emacs-indentation-innamespace-td22802271.html in this thread].<br />
<br />
=== If you use VIM ===<br />
<br />
If you use VIM you should add the following lines to your ~/.vimrc:<br />
<pre><br />
set ts=2<br />
set sw=2<br />
set sta<br />
set et<br />
set ai<br />
set si<br />
set cin<br />
</pre><br />
<br />
And the following lines to realize white space errors (trailing white spaces):<br />
<pre><br />
let c_no_bracket_error=1<br />
let c_no_curly_error=1<br />
let c_comment_strings=1<br />
let c_gnu=1<br />
</pre><br />
<br />
== Submitting patches for consideration ==<br />
<br />
The best way to submit a patch for consideration is to request a patch review:<br />
<br />
# download http://codereview.appspot.com/static/upload.py<br />
# record the changes you want to request a review for in a mercurial repository: "hg commit ..."<br />
# within the mercurial repository, run upload.py to submit a review with http://codereview.appspot.com/. Make sure you specify ns-3-reviews@googlegroups.com as a CC.<br />
# paste your review request on http://www.nsnam.org/wiki/index.php/Reviews<br />
# announce your review request on ns-developers<br />
<br />
When you send a tree without a detailed summary of your changes, it would help if you could send a list of the changesets you want to merge. To generate it, first merge with ns-3-dev and then, from your modified directory, run "hg outgoing -p http://code.nsnam.org/ns-3-dev"<br />
<br />
Also, avoid spurious coding style and whitespace changes when preparing such a patch, as it distracts from the readability of your proposed technical changes.<br />
<br />
If you already pulled changes from ns-3-dev into your private repository after you started doing your private modifications, there is an issue to be considered. upload.py does not let you specify a range of revisions, nor a set of changesets. So if you just run upload.py in your private repository, also the changesets pulled from ns-3-dev will be published on codereview, which is of course not desirable.<br />
<br />
A possible workaround is to pull your changes into a temporary repository which is an up-to-date clone of ns-3-dev. The following code should do the trick. This code assumes that your private repository is in path DEV_BRANCH_WITH_NEW_FEATURE, and that it is in sync with ns-3-dev.<br />
<br />
<pre><br />
hg clone http://code.nsnam.org/ns-3-dev ns-3-tmp<br />
cd ns-3-tmp<br />
export REVNO=`hg tip -q | sed 's/:.*$//'`<br />
hg pull DEV_BRANCH_WITH_NEW_FEATURE<br />
hg merge<br />
hg commit -m "merged new feature"<br />
upload.py --rev=$REVNO --cc=ns-3-reviews@googlegroups.com <br />
</pre><br />
<br />
== Checking in code ==<br />
<br />
=== General guidelines ===<br />
<br />
Please always update your code to the tip of ns-3-dev before checking in. Never force a commit (which creates another head on the repository). If someone else committed something before you got a chance to perform your commit, you will need to pull those changes, resolve any possible conflicts, and commit. <br />
<br />
The command to check in code is typically:<br />
<br />
hg push ssh://code@code.nsnam.org/repos/ns-3-dev<br />
<br />
If you forgot to do an update before, you will need to:<br />
<br />
hg pull<br />
hg update<br />
hg commit -m"branch merge"<br />
./test.py (make sure you didn't break anything)<br />
hg push ssh://code@code.nsnam.org/repos/ns-3-dev<br />
<br />
=== Checking in patches from other users ===<br />
<br />
If you as a maintainer check in a patch authored by someone else, it is good practice to credit them in the commit message using the --user or -u option to hg; e.g.:<br />
<br />
hg commit -m"...commit message..." -u"A User <a.user@example.com>"<br />
<br />
Reference: http://www.gnu.org/prep/maintain/html_node/Recording-Contributors.html<br />
<br />
=== Checking that you don't introduce regressions or leaks ===<br />
<br />
just before you do a checkin you should run the ns-3 tests. Make sure that you have valgrind installed, and change into the top level directory and type<br />
<br />
./test.py --grind<br />
<br />
If you do this, you will see a bunch of passing tests and then, if something goes wrong:<br />
<br />
FAIL: TestSuite test-tcp-large-transfer<br />
<br />
=== Logging your changes in the release notes ===<br />
<br />
The project maintains two files for tracking changes to the codebase. As you check in changes to the code that fix bugs, add features, or change the existing behavior of the simulator, you should also update these files.<br />
<br />
# ''CHANGES.html'': Used to log changes to the build system, new API, changes to existing API, and changed behavior.<br />
# ''RELEASE_NOTES'': Summarize new user-visible features and bugs fixed.<br />
<br />
These files may both be updated, or maybe only one is updated for a given changeset. For instance, if you change the API on a class, you might add a simple bullet "Changed API for class X to improve ability to do Y" to RELEASE_NOTES, but add more detail about exactly which signatures changed in the CHANGES.html file. If the change is trivial (e.g. adding missing Doxygen), there is no need to edit these files. <br />
<br />
Please update these files when you make changes to ns-3-dev rather than relying on the release manager later documenting them.<br />
<br />
=== Fixing a bug from the Bugzilla tracker ===<br />
<br />
Q. What I should do when a bug will be fixed (send a mail to the mailing list, write some comment to bugzilla, other ) ?<br />
<br />
A. The typical thing to do is:<br />
<br />
* check in the bug fix to ns-3-dev with a commit message that indicates the bug number, such as "bug 903: TapBridge doesn't close cleanly"<br />
<br />
* remember what the changeset number is; e.g.<br />
<br />
changeset: 1763:4624d5aba98f<br />
^^^^^^^^^^^^<br />
it is this hexadecimal value<br />
<br />
* mark the bug as "Resolved, Fixed" in Bugzilla, with a comment such as "fixed in changset 4624d5aba98f". Bugzilla will notify the ns-bugs mailing list of the fix, and the ns-commits list will be notified of the checkin.<br />
<br />
== Using gcov and lcov code coverage tools ==<br />
<br />
Here is a brief howto for using the [http://ltp.sourceforge.net/coverage/lcov.php lcov] front-end to gcc's code coverage tool [http://gcc.gnu.org/onlinedocs/gcc/Gcov.html gcov]<br />
<br />
A custom version of lcov and its associated tools (geninfo, genhtml) is stored in the directory utils/lcov/, and this is the version that waf uses to run lcov. As of ns-3.13, the version of lcov in ns-3 is lcov-1.9, with geninfo patched to deal with [http://old.nabble.com/-Ltp-coverage--lcov-hangs-td28514226.html this branching bug]. <br />
<br />
You probably also need to have a version of lcovrc file installed somewhere (either at /etc/lcovrc or ~/.lcovrc). The easiest way to do this is to install lcov from your package manager; e.g.<br />
sudo apt-get install lcov<br />
or install it from source. But just be aware that waf will use the version that is maintained in the utils/lcov/ directory.<br />
<br />
Then, for instance, to see the coverage of the ns-3 test suite, type: <br />
<br />
./waf configure --enable-gcov --enable-examples --enable-tests<br />
./test.py<br />
./waf --lcov-report<br />
<br />
You will find a file "index.html" in the directory build/debug-gcov/lcov-report/ that you can look at with your browser.<br />
<br />
Note that if you want to test coverage of another program, you will want to zero the counters for lcov. You can't presently do this from within waf but you can run this command from your top-level ns-3 directory:<br />
<br />
utils/lcov/lcov --directory build --zerocounters<br />
<br />
and it should report:<br />
<br />
Deleting all .da files in build and subdirectories<br />
Done.<br />
<br />
== The preferred way to create a private repository ==<br />
<br />
Let's say that developer "alice" wants to create a new repository "ns-3-dev-new-feature" that will exist on the code server as alice/ns-3-dev-new-feature. Suppose she wants to fork from the tip of ns-3-dev.<br />
<br />
cd /home/alice/repositories/alice<br />
cp -r /home/code/repos/ns-3-dev ns-3-dev-new-feature<br />
cd ns-3-dev-new-feature/.hg<br />
<br />
At this point, edit the "hgrc" file to provide contact/description information:<br />
<br />
[paths]<br />
default = http://code.nsnam.org/alice/ns-3-dev-new-feature<br />
[web]<br />
description = alice's new feature<br />
contact = <alice@example.com><br />
<br />
'''Note:''' A common minor problem is if you do an "hg clone" into the new directory instead of a "cp -r", there will be a huge ns-commits mail message generated. This is why "cp -r" is preferred way to do this.<br />
<br />
== Testing code on nsnam.org hosts ==<br />
Some times a compilation error can only reproduced in certain architectures. nsnam.org has some machines that can be accessed remotely for testing purposes, if you have an appropriate account. Contact Tom Henderson if you need an account.<br />
<br />
=== Full Suite ===<br />
Note that this takes several hours to run. It puts a specified branch through its paces on several machines/architectures. You can specify a branch to test and a notification email address where you will receive a report about how the branch performed in the tests (SUCCESS or FAILURE).<br />
$ ssh ns-regression.ee.washington.edu<br />
$ sudo -u nsnam bash<br />
$ cd /usr/local/bin/<br />
$ ./ns-3-run-tests.sh -h<br />
usage: ./ns-3-run-tests.sh options<br />
<br />
This script runs the ns-3 branch through the ns-regression testbed.<br />
OPTIONS:<br />
-h Help: show this message<br />
-n ns-3 branch Default: ns-3-dev<br />
-m mailto address. Where to send the mail. If unspecified, program output will just scroll onto stdout.<br />
$ ./ns-3-run-tests.sh -n mathieu/ns-3-simu -m someone@wherever.com<br />
<br />
=== Ubuntu x86_64 ===<br />
ssh ns-regression.ee.washington.edu<br />
<br />
=== Mac OS X PowerPC ===<br />
ssh ns-regression.ee.washington.edu<br />
sudo -u nsnam bash<br />
<enter your password><br />
ssh darwin-ppc<br />
<br />
== Adding a buildslave to our BuildBot ==<br />
The ns-3 project uses BuildBot to test the ns-3-dev build daily. The buildmaster resides on a server at the University of Washington. Currently, we use several buildslaves running Fedora Core 10 with different versions of gcc as well as a Mac OS X PPC machine. For a quick snapshot of the current buildslaves in use, please see the buildbot [http://ns-regression.ee.washington.edu:8010/waterfall waterfall].<br />
<br />
If you would like to see another buildslave, for example mingw or cygwin, and you have a machine to donate daily cycles, please complete the following steps:<br />
<br />
# Install buildbot using a package manager or installing from source: [http://buildbot.net/trac/wiki/DownloadInstall download]<br />
# When creating the buildslave, you will need to provide a slavename and password. Note that you will have to send us this name and password. You also need the hostname and port of our buildmaster: ns-regression.ee.washington.edu:9989<br />
# Follow the buildbot manual to setup a buildslave: [http://buildbot.net/buildbot/docs/0.8.1/Creating-a-buildslave.html creating a buildslave]<br />
# Send John Abraham <john.abraham@gatech.edu> an email with your buildslave name and password, and we will update the buildmaster configuration to accept connections from your buildslave.<br />
<br />
To change the buildmaster master script, edit master.cfg and then 'make reconfig' in the /home/buildmaster/master directory.<br />
<br />
To restart buildbot on ns-regression:<br />
su buildmaster<br />
cd /home/buildmaster<br />
/usr/bin/buildbot start master</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=6441HOWTO get ns-3 to detect steady-state times in your data2012-02-03T23:23:03Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
=== Batch Means (BM) Test ===<br />
<br />
==== Background ====<br />
<br />
* '''White, K., Cobb, M., & Spratt, S. (2000)'''. A Comparison of Five Steady-state Truncation Heuristics for Simulation. ''Proceedings of the 32nd Winter Simulation Conference (WSC '00)'', 755–760.<br />
** In this reference, they say: "Cash, et al. (1992), Nelson (1992), and Goldsman, et al. (1994), propose a family of related tests for detecting the presence of bias in an output series. These tests generalize and extend the earlier work Schruben (1982) and Schruben, et al. (1983)."<br />
* '''Cash, C.R., Dippold, D.G., Nelson, B.L., Pollard, W.P., and Long, J.M. (1992)'''. Evaluation of Tests for Initial-condition Bias. ''Proceedings of the 1992 Winter Simulation Conference'', 577-585.<br />
* Algorithm: Bias detection test.<br />
* Reason: Steady-state detection method can detect if not in steady-state (unlike MSER-5).<br />
* Reason: Single-replicate algorithm.<br />
<br />
==== Simulated Test Data ====<br />
<br />
* Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has an exponential bias along with the steady-state times that my implementations of the Batch Mean (BM) algorithm and the MSER-5 algrothm are currently returning. The plot also shows vertical lines where the steady-state times are. [[Image:steady-state-detector-test-exponential-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The BM method is giving a steady-state time after steady-state has been reached, which would cause it to delete more values than the MSER-5 method.<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with an exponential bias. [[Image:steady-state-detector-test-exponential-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets with fewer values than are required for steady-state (to the left of the blue line) are being called steady-state by the algorithm and giving steady-state times much less than the true steady-state time (the green line).<br />
* Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has a linearly increasing bias. My implementation of the Batch Mean (BM) algorithm did not find a steady-state time for this number of data values. [[Image:steady-state-detector-test-linearly-increasing-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with a linearly increasing bias. [[Image:steady-state-detector-test-linearly-increasing-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets that are not at steady-state, i.e. all of the data values in this data set, are being called steady-state by the algorithm.<br />
* '''Note: Because the Batch Mean (BM) algorithm is detecting steady-state for data sets that are not in steady-state, it is being REJECTED.'''<br />
<br />
==== MANET Routing ExampLe Data ====<br />
<br />
* Note: There are now 2 steady-state detectors in the SAFE MANET example:<br />
** Packets Received<br />
** Receive Rate<br />
* Note: Although the plots have different numerical values for packets-received and receive-rate, they do have the same shape, which means the 2 quantities appear to be linearly related.<br />
* Note: I made the example run for 1000 seconds so that it would be in steady-state.<br />
* Note: The attached plots make it look like the MSER-5 method works well for simulations like this that have achieved steady-state.<br />
* Note: The following 2 plots show 1000 and 200 seconds from the MANET packets-received data set along with the steady-state times that my implementations of the Batch Mean (BM) algorithm and the MSER-5 algrothm are currently returning. The plots also show vertical lines where the steady-state times are. [[Image:Manet-routing-safe-steady-state-detector-packets-received-data-values-1000-seconds.png|540px|center|SAFE MANET example packets-received data set with steady-state time.]] [[Image:Manet-routing-safe-steady-state-detector-packets-received-data-values-200-seconds.png|540px|center|SAFE MANET example packets-received data set with steady-state time.]]<br />
* Note: The BM method is giving a steady-state time before steady-state has been reached, which would cause it to include unwanted values that the MSER-5 method does not.<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the MANET packets-received data set. [[Image:Manet-routing-safe-steady-state-detector-packets-received-steady-state-times-vs-number-of-values-1000-seconds.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets with fewer values than are required for steady-state for times up to around 120 seconds are being called steady-state by the algorithm and giving steady-state times much less than the true steady-state time, which is around 60 seconds.<br />
* Note: The following 2 plots show 1000 and 200 seconds from the MANET receive-rate data set along with the steady-state times that my implementations of the Batch Mean (BM) algorithm and the MSER-5 algrothm are currently returning. The plots also show vertical lines where the steady-state times are. [[Image:Manet-routing-safe-steady-state-detector-receive-rate-data-values-1000-seconds.png|540px|center|SAFE MANET example receive-rate data set with steady-state time.]] [[Image:Manet-routing-safe-steady-state-detector-receive-rate-data-values-200-seconds.png|540px|center|SAFE MANET example receive-rate data set with steady-state time.]]<br />
* Note: The BM method is giving a steady-state time before steady-state has been reached, which would cause it to include unwanted values that the MSER-5 method does not.<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the MANET receive-rate data set. [[Image:Manet-routing-safe-steady-state-detector-receive-rate-steady-state-times-vs-number-of-values-1000-seconds.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets with fewer values than are required for steady-state for times up to around 120 seconds are being called steady-state by the algorithm and giving steady-state times much less than the true steady-state time, which is around 60 seconds.<br />
* '''Note: Because the Batch Mean (BM) algorithm is detecting steady-state for data sets that are not in steady-state, it is being REJECTED.'''<br />
<br />
==== Conclusion ====<br />
<br />
'''A more robust steady-state detection algorithm than the Batch Mean (BM) algorithm should be implemented that will then be coupled with the MSER-5 method that appears to work well for simulations once they have achieved steady-state.'''<br />
<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=6440HOWTO get ns-3 to detect steady-state times in your data2012-02-03T21:40:18Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
=== Batch Means (BM) Test ===<br />
<br />
==== Background ====<br />
<br />
* '''White, K., Cobb, M., & Spratt, S. (2000)'''. A Comparison of Five Steady-state Truncation Heuristics for Simulation. ''Proceedings of the 32nd Winter Simulation Conference (WSC '00)'', 755–760.<br />
** In this reference, they say: "Cash, et al. (1992), Nelson (1992), and Goldsman, et al. (1994), propose a family of related tests for detecting the presence of bias in an output series. These tests generalize and extend the earlier work Schruben (1982) and Schruben, et al. (1983)."<br />
* '''Cash, C.R., Dippold, D.G., Nelson, B.L., Pollard, W.P., and Long, J.M. (1992)'''. Evaluation of Tests for Initial-condition Bias. ''Proceedings of the 1992 Winter Simulation Conference'', 577-585.<br />
* Algorithm: Bias detection test.<br />
* Reason: Steady-state detection method can detect if not in steady-state (unlike MSER-5).<br />
* Reason: Single-replicate algorithm.<br />
<br />
==== Simulated Test Data ====<br />
<br />
* Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has an exponential bias along with the steady-state times that my implementations of the Batch Mean (BM) algorithm and the MSER-5 algrothm are currently returning. The plot also shows vertical lines where the steady-state times are. [[Image:steady-state-detector-test-exponential-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The BM method is giving a steady-state time after steady-state has been reached, which would cause it to delete more values than the MSER-5 method.<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with an exponential bias. [[Image:steady-state-detector-test-exponential-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets with fewer values than are required for steady-state (to the left of the blue line) are being called steady-state by the algorithm and giving steady-state times much less than the true steady-state time (the green line).<br />
* Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has a linearly increasing bias. My implementation of the Batch Mean (BM) algorithm did not find a steady-state time for this number of data values. [[Image:steady-state-detector-test-linearly-increasing-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with a linearly increasing bias. [[Image:steady-state-detector-test-linearly-increasing-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
* Note: The above figure shows a problem with the BM method that data sets that are not at steady-state, i.e. all of the data values in this data set, are being called steady-state by the algorithm.<br />
* '''Note: Because the Batch Mean (BM) algorithm is detecting steady-state for data sets that are not in steady-state, it is being REJECTED.'''<br />
<br />
==== Conclusion ====<br />
<br />
A more robust steady-state detection algorithm than the Batch Mean (BM) algorithm should be implemented that will then be coupled with the MSER-5 method that appears to work well for simulations once they have achieved steady-state.<br />
<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-receive-rate-steady-state-times-vs-number-of-values-1000-seconds.png&diff=6439File:Manet-routing-safe-steady-state-detector-receive-rate-steady-state-times-vs-number-of-values-1000-seconds.png2012-02-03T21:05:38Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-receive-rate-data-values-200-seconds.png&diff=6438File:Manet-routing-safe-steady-state-detector-receive-rate-data-values-200-seconds.png2012-02-03T21:05:01Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-receive-rate-data-values-1000-seconds.png&diff=6437File:Manet-routing-safe-steady-state-detector-receive-rate-data-values-1000-seconds.png2012-02-03T21:04:14Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-packets-received-steady-state-times-vs-number-of-values-1000-seconds.png&diff=6436File:Manet-routing-safe-steady-state-detector-packets-received-steady-state-times-vs-number-of-values-1000-seconds.png2012-02-03T21:03:12Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-packets-received-data-values-200-seconds.png&diff=6435File:Manet-routing-safe-steady-state-detector-packets-received-data-values-200-seconds.png2012-02-03T21:01:10Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Manet-routing-safe-steady-state-detector-packets-received-data-values-1000-seconds.png&diff=6434File:Manet-routing-safe-steady-state-detector-packets-received-data-values-1000-seconds.png2012-02-03T18:58:03Z<p>Watrous: </p>
<hr />
<div></div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=6193HOWTO get ns-3 to detect steady-state times in your data2011-08-31T19:50:34Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
* Batch Means (BM) Test<br />
** '''White, K., Cobb, M., & Spratt, S. (2000)'''. A Comparison of Five Steady-state Truncation Heuristics for Simulation. ''Proceedings of the 32nd Winter Simulation Conference (WSC '00)'', 755–760.<br />
*** In this reference, they say: "Cash, et al. (1992), Nelson (1992), and Goldsman, et al. (1994), propose a family of related tests for detecting the presence of bias in an output series. These tests generalize and extend the earlier work Schruben (1982) and Schruben, et al. (1983)."<br />
** '''Cash, C.R., Dippold, D.G., Nelson, B.L., Pollard, W.P., and Long, J.M. (1992)'''. Evaluation of Tests for Initial-condition Bias. ''Proceedings of the 1992 Winter Simulation Conference'', 577-585.<br />
** Algorithm: Bias detection test.<br />
** Reason: Steady-state detection method can detect if not in steady-state (unlike MSER-5).<br />
** Reason: Single-replicate algorithm.<br />
** Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has an exponential bias along with the steady-state times that my implementations of the Batch Mean (BM) algorithm and the MSER-5 algrothm are currently returning. The plot also shows vertical lines where the steady-state times are. [[Image:steady-state-detector-test-exponential-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: The BM method is giving a steady-state time after steady-state has been reached, which would cause it to delete more values than the MSER-5 method.<br />
** Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with an exponential bias. [[Image:steady-state-detector-test-exponential-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: The above figure shows a problem with the BM method that data sets with fewer values than are required for steady-state (to the left of the blue line) are being called steady-state by the algorithm and giving steady-state times much less than the true steady-state time (the green line).<br />
** Note: The following plot shows a simulated data set '''(with 2,000 data values)''' that has a linearly increasing bias. My implementation of the Batch Mean (BM) algorithm did not find a steady-state time for this number of data values. [[Image:steady-state-detector-test-linearly-increasing-bias-data-values.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: The following plot shows the steady-state times calculated as a function of the number of data values for the simulated data set with a linearly increasing bias. [[Image:steady-state-detector-test-linearly-increasing-bias-steady-state-times-vs-number-of-values.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: The above figure shows a problem with the BM method that data sets that are not at steady-state, i.e. all of the data values in this data set, are being called steady-state by the algorithm.<br />
** '''Note: Because the Batch Mean (BM) algorithm is detecting steady-state for data sets that are not in steady-state, it is being REJECTED.'''<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-linearly-increasing-bias-data-values.png&diff=6192File:Steady-state-detector-test-linearly-increasing-bias-data-values.png2011-08-31T18:52:33Z<p>Watrous: Data values and steady-state times for data with linearly increasing bias</p>
<hr />
<div>Data values and steady-state times for data with linearly increasing bias</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-linearly-increasing-bias-steady-state-times-vs-number-of-values.png&diff=6191File:Steady-state-detector-test-linearly-increasing-bias-steady-state-times-vs-number-of-values.png2011-08-31T18:51:11Z<p>Watrous: Steady-state times vs number of data values for data with linearly increasing bias</p>
<hr />
<div>Steady-state times vs number of data values for data with linearly increasing bias</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-exponential-bias-steady-state-times-vs-number-of-values.png&diff=6190File:Steady-state-detector-test-exponential-bias-steady-state-times-vs-number-of-values.png2011-08-31T18:49:40Z<p>Watrous: Steady-state times vs number of data values for data with exponential bias</p>
<hr />
<div>Steady-state times vs number of data values for data with exponential bias</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-exponential-bias-data-values.png&diff=6189File:Steady-state-detector-test-exponential-bias-data-values.png2011-08-31T18:48:01Z<p>Watrous: Data values and steady-state times for data with exponential bias</p>
<hr />
<div>Data values and steady-state times for data with exponential bias</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.12&diff=5958Ns-3.122011-08-16T17:12:42Z<p>Watrous: /* Bugs being worked */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.12. Tom Henderson, John Abraham, and Mitch Watrous will be jointly managing the release.<br />
<br />
= Proposed release schedule =<br />
<br />
# May 25 -- ns-3.11 posted, ns-3.12 open phase begins.<br />
# Wed Aug 17 -- code freeze on ns-3-dev, ns-3.12 RC1 posted<br />
# Wed Aug 24 -- ns-3.12 released<br />
<br />
Please see [[Release Process]] concerning general aspects of the release schedule.<br />
<br />
= Proposed supported platforms =<br />
<br />
* Ubuntu 11.04 (32/64 bit)<br />
* Ubuntu 10.04 LTS (64 bit)<br />
* Fedora Core 15 (32/64 bit)<br />
* Mac OS X Lion and Snow Leopard (32 bit)<br />
<br />
= Packaging =<br />
<br />
We will use ns-allinone-3.12.tar.bz2 packaging as before.<br />
<br />
Contributors can also prepare rpms and debs, but these can occur after the release, and there are some lingering issues regarding these packages:<br />
* rpms for Fedora (Vedran Miletić)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009234.html<br />
* debs for Ubuntu/Debian (YunQiang Su)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009222.html<br />
<br />
= Release blockers =<br />
<br />
None at this time.<br />
<br />
= New features being considered for ns-3.12 =<br />
<br />
These should be reviewed quickly to see if they can be integrated to this release.<br />
<br />
* [http://codereview.appspot.com/4823051/ Dynamic Source Routing (DSR)]<br />
* [http://codereview.appspot.com/4747046 TDMA model]<br />
* [http://codereview.appspot.com/4644042/ ObjectMap], to add a map to the attribute system instead of a vector<br />
<br />
<br />
= Bugs being worked =<br />
<br />
We are trying to close these bugs for this release cycle (no guarantees that we will get them all):<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=272 272 documentation] (Tom)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=388 388 documentation] (Tom)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=707 707 Build OpenSolaris]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=756 756 Graceful build failure when libraries broken]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=827 827 Wifi] (Ruben) Update RELEASE NOTES<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=845 845 ConfigStore] (Mathieu; patch is pending)</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=904 904 Packet const methods] Wontfix </s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=928 928 variable length array]</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=929 929 variable length array]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=962 962 path to attributes has bogus entries] May be invalid<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1010 1010 Patch UAN module] (Andrea)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1039 1039 TCP Nagle and RTO] (Adrian) (also bug 1227 is a duplicate)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1043 1043 LTE constructor problem] (Giuseppe, patch pending)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1046 1046 DSDV: NULL callback]<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1047 1047 multicast > 16 interfaces]</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1050 1050 variable length array] duplicate of 928/929?</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1055 1055 possible UAN bug] (Andrea)<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1082 1082 LogComponentEnable() does not provide error handling]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1107 1107 monolithic python bindings] (Gustavo) related to 1240<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1112 1112 TCP deadlock] (Adrian)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1127 1127 Netanim compatibility for xcode 4 and wiki updates] (John, probably wontfix)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1134 1134 Energy attribute spelling error] (Tony)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1163 1163 forgot to increment ephemeral port] (Adrian)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1164 1164 IPV4 TCP Bind an already used port failed but without setting errno]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1165 1165 memset used with constant zero length parameter]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1166 1166 ns-3 TCP DCE problems with RST] (Adrian)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1168 1168 BridgeNetDevice: Default ExpirationTime not realistic] (Gustavo)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 1172 Implement trivial XML output feature for Animation Interface] (John, probably wontfix)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1173 1173 Valgrind detects memory leaks from Click when using ns-3-click] (Ruben and Lalith)<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1176 1176 modules document fails to build]</s><br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1177 1177 The auto generated man pages have some error]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1179 1179 Add LTE trace hooks for supporting visualizer] (Giuseppe)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1180 1180 LTE: Add TX/RX trace support] (Giuseppe)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1186 1186 Ipv4Header lacks DSCP and ECN] Needs patch<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 1192 Some test cases fail to clean up properly (missing DoTeardown)] (Mathieu, Claudio)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1212 1212 attribute path documentation broken] (Mitch)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1216 1216 RvBatteryModel segfalts if the update event happens before load changes] (Andrea)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1217 1217 simple-odfm-wimax-phy creates UniformVariable in a critical path] (Flavio)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1218 1218 Supply a mechanism to build python binding for multi-version of python]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1220 1220 FdReader always stops with NS_FATAL_ERROR] (Tom G.)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1226 1226 Incomplete YansWifiChannel Python bindings]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1227 1227 TCP minRTO too low] (Adrian)<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1228 1228 modular build problem]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1229 1229 Multiplication overflow in WaypointMobilityModel::EndMobility method]<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1237 1237 code cleanups related to include]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1238 1238 documentation]<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1239 1239 Add assertion that reference count never overflows]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1240 1240 1240 remove monolithic python bindings from waf]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1241 1241 1241 documentation]<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=1243 1243 Exiting waf shell should not generate report on what modules are built]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1246 1246 waf --enable-shared-and-static]<br />
<br />
<s>Also, the collection of "set but not used" bugs (such as bugs 1147-1162).</s> Note that the general solution for bug 1170 is still in work.<br />
<br />
Also, the collection of RealtimeSimulatorImpl and TapBridge patches (Tom Goff is looking at):<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=603 Simulator::Next useless] depends on 631<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=631 RealtimeSimulator and Ctrl-C]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=903 Tap Bridge and Emu Net Devices Do Not Shut Down Properly]<br />
* <s>[https://www.nsnam.org/bugzilla/show_bug.cgi?id=975 stopping TapBridge]</s><br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1000 Make realtime simulator last until stop]<br />
<br />
= Active reviews sliding past ns-3.12 =<br />
<br />
These are not going to complete in time for the ns-3.12 release.<br />
<br />
* [http://codereview.appspot.com/4517125/ BRITE Integration]<br />
** under revision<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-June/009121.html LTE code from the Lena project]<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback]<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 Animation interface changes (to support XML output)] <br />
* [http://codereview.appspot.com/2104052/show Virtual Access Point (VAP) for WiFi]<br />
** needs updated to ns-3.11 code; needs review<br />
* [http://codereview.appspot.com/2135046/show GetRelativeVelocity() for MobilityModel]<br />
** needs review<br />
* [http://codereview.appspot.com/2173042/show PhySimWifi]<br />
** needs review<br />
* [http://codereview.appspot.com/2334041/show thread-safe implementation of ScheduleWithContext]<br />
* [http://codereview.appspot.com/1587041/show simple wireless models]<br />
* [http://codereview.appspot.com/1055041/show Jamming model]<br />
* [http://codereview.appspot.com/850045/show MPLS]<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2010-February/007419.html TCP Vegas] (blocked by lack of progress on TCP congestion control architecture)<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-August/009255.html RED for ns-3]<br />
** Needs rebased to ns-3-dev; needs test code<br />
<br />
= ns-3.12 features merged =<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header Class MinMaxAvgTotalCalculator does not implement all the methods in its interface]<br />
* Fragmentation support for IPv4<br />
<br />
= ns-3.12 bugs fixed =<br />
<br />
This list needs updating.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header FragmentOffset field]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1235 invalid tests pass]<br />
<br />
= Feature requests =<br />
<br />
This lists some features that have been requested but for which a patch is needed.<br />
<br />
* make spectrum model compatible with ns-3 WiFi<br />
* ns-2 packet UID feature <br />
* API for TOS bytes (issue 897) ''may wait for netfilter support''<br />
* Chord/DHT (authors are planning to revise code based on comments)<br />
* app store and build system refactoring<br />
* fragmentation for IPv4: being worked by Vedran Miletic and Tommaso Pecorella<br />
* TDMA wireless model<br />
** being worked on by Hemanth Narra<br />
* Netdevice queue state support<br />
** being worked on by Ruben Merz</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.12&diff=5949Ns-3.122011-08-10T15:18:48Z<p>Watrous: /* ns-3.12 bugs fixed */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.12. Tom Henderson, John Abraham, and Mitch Watrous will be jointly managing the release.<br />
<br />
= Proposed release schedule =<br />
<br />
# May 25 -- ns-3.11 posted, ns-3.12 open phase begins.<br />
# Wed Aug 17 -- code freeze on ns-3-dev, ns-3.12 RC1 posted<br />
# Wed Aug 24 -- ns-3.12 released<br />
<br />
Please see [[Release Process]] concerning general aspects of the release schedule.<br />
<br />
= Proposed supported platforms =<br />
<br />
* Ubuntu 11.04 (32/64 bit)<br />
* Ubuntu 10.04 LTS (64 bit)<br />
* Fedora Core 15 (32/64 bit)<br />
* Mac OS X Lion and Snow Leopard (32 bit)<br />
<br />
= Packaging =<br />
<br />
We will use ns-allinone-3.12.tar.bz2 packaging as before.<br />
<br />
Contributors can also prepare rpms and debs, but these can occur after the release, and there are some lingering issues regarding these packages:<br />
* rpms for Fedora (Vedran Miletić)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009234.html<br />
* debs for Ubuntu/Debian (YunQiang Su)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009222.html<br />
<br />
= Release blockers =<br />
<br />
None at this time.<br />
<br />
= New features being considered for ns-3.12 =<br />
<br />
These should be reviewed quickly to see if they can be integrated to this release.<br />
<br />
* [http://codereview.appspot.com/4823051/ Dynamic Source Routing (DSR)]<br />
* [http://codereview.appspot.com/4747046 TDMA model]<br />
* [http://codereview.appspot.com/4644042/ ObjectMap], to add a map to the attribute system instead of a vector<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-August/009255.html RED for ns-3]<br />
<br />
= Bugs being worked =<br />
<br />
We are trying to close these bugs for this release cycle (no guarantees that we will get them all):<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=272 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=388 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=707 Build OpenSolaris]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=827 Wifi] Update RELEASE NOTES<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=845 ConfigStore]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=904 Packet const methods]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=928 variable length array]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=929 variable length array]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=962 path to attributes has bogus entries]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1010 Patch UAN module]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1039 TCP Nagle and RTO] (also bug 1227 is a duplicate)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1043 LTE constructor problem]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1046 DSDV: NULL callback]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1047 multicast > 16 interfaces] this just needs verification that it is closed<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1050 variable length array] duplicate of 928/929?<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1055 possible UAN bug]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1082 LogComponentEnable() does not provide error handling]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1107 monolithic python bindings] related to 1240<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1112 TCP deadlock]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1127 Netanim compatibility for xcode 4 and wiki updates]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1134 Energy attribute spelling error]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1163 forgot to increment ephemeral port]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1164 IPV4 TCP Bind an already used port failed but without setting errno]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1165 memset used with constant zero length parameter]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1166 ns-3 TCP DCE problems with RST]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1168 BridgeNetDevice: Default ExpirationTime not realistic]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 Implement trivial XML output feature for Animation Interface]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1173 Valgrind detects memory leaks from Click when using ns-3-click]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1176 modules document fails to build]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1177 The auto generated man pages have some error]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1179 Add LTE trace hooks for supporting visualizer]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1180 LTE: Add TX/RX trace support]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1186 Ipv4Header lacks DSCP and ECN] Needs patch<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 Some test cases fail to clean up properly (missing DoTeardown)]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1212 attribute path documentation broken]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1216 RvBatteryModel segfalts if the update event happens before load changes]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1217 simple-odfm-wimax-phy creates UniformVariable in a critical path]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1218 Supply a mechanism to build python binding for multi-version of python]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1220 FdReader always stops with NS_FATAL_ERROR]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1226 Incomplete YansWifiChannel Python bindings]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1227 TCP minRTO too low]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1228 modular build problem]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1229 Multiplication overflow in WaypointMobilityModel::EndMobility method]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1237 code cleanups related to include]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1238 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1239 Add assertion that reference count never overflows]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1240 remove monolithic python bindings from waf]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1241 documentation]<br />
<br />
Also, the collection of "set but not used" bugs (such as bugs 1147-1162). Note that the general solution for bug 1170 is still in work.<br />
<br />
Also, the collection of RealtimeSimulatorImpl and TapBridge patches (Tom Goff is looking at):<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=603 Simulator::Next useless] depends on 631<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=631 RealtimeSimulator and Ctrl-C]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=903 Tap Bridge and Emu Net Devices Do Not Shut Down Properly]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=975 stopping TapBridge]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1000 Make realtime simulator last until stop]<br />
<br />
= Active reviews sliding past ns-3.12 =<br />
<br />
These are not going to complete in time for the ns-3.12 release.<br />
<br />
* [http://codereview.appspot.com/4517125/ BRITE Integration]<br />
** under revision<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-June/009121.html LTE code from the Lena project]<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback]<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 Animation interface changes (to support XML output)] <br />
* [http://codereview.appspot.com/2104052/show Virtual Access Point (VAP) for WiFi]<br />
** needs updated to ns-3.11 code; needs review<br />
* [http://codereview.appspot.com/2135046/show GetRelativeVelocity() for MobilityModel]<br />
** needs review<br />
* [http://codereview.appspot.com/2173042/show PhySimWifi]<br />
** needs review<br />
* [http://codereview.appspot.com/2334041/show thread-safe implementation of ScheduleWithContext]<br />
* [http://codereview.appspot.com/1587041/show simple wireless models]<br />
* [http://codereview.appspot.com/1055041/show Jamming model]<br />
* [http://codereview.appspot.com/850045/show MPLS]<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2010-February/007419.html TCP Vegas] (blocked by lack of progress on TCP congestion control architecture)<br />
<br />
= ns-3.12 features merged =<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header Class MinMaxAvgTotalCalculator does not implement all the methods in its interface]<br />
* Fragmentation support for IPv4<br />
<br />
= ns-3.12 bugs fixed =<br />
<br />
This list needs updating.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header FragmentOffset field]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1235 invalid tests pass]<br />
<br />
= Feature requests =<br />
<br />
This lists some features that have been requested but for which a patch is needed.<br />
<br />
* make spectrum model compatible with ns-3 WiFi<br />
* ns-2 packet UID feature <br />
* API for TOS bytes (issue 897) ''may wait for netfilter support''<br />
* Chord/DHT (authors are planning to revise code based on comments)<br />
* app store and build system refactoring<br />
* fragmentation for IPv4: being worked by Vedran Miletic and Tommaso Pecorella<br />
* TDMA wireless model<br />
** being worked on by Hemanth Narra<br />
* Netdevice queue state support<br />
** being worked on by Ruben Merz</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.12&diff=5948Ns-3.122011-08-10T15:10:10Z<p>Watrous: /* Bugs being worked */</p>
<hr />
<div>{{TOC}}<br />
<br />
This page summarizes the ongoing release planning for ns-3.12. Tom Henderson, John Abraham, and Mitch Watrous will be jointly managing the release.<br />
<br />
= Proposed release schedule =<br />
<br />
# May 25 -- ns-3.11 posted, ns-3.12 open phase begins.<br />
# Wed Aug 17 -- code freeze on ns-3-dev, ns-3.12 RC1 posted<br />
# Wed Aug 24 -- ns-3.12 released<br />
<br />
Please see [[Release Process]] concerning general aspects of the release schedule.<br />
<br />
= Proposed supported platforms =<br />
<br />
* Ubuntu 11.04 (32/64 bit)<br />
* Ubuntu 10.04 LTS (64 bit)<br />
* Fedora Core 15 (32/64 bit)<br />
* Mac OS X Lion and Snow Leopard (32 bit)<br />
<br />
= Packaging =<br />
<br />
We will use ns-allinone-3.12.tar.bz2 packaging as before.<br />
<br />
Contributors can also prepare rpms and debs, but these can occur after the release, and there are some lingering issues regarding these packages:<br />
* rpms for Fedora (Vedran Miletić)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009234.html<br />
* debs for Ubuntu/Debian (YunQiang Su)<br />
** discussed here: http://mailman.isi.edu/pipermail/ns-developers/2011-August/009222.html<br />
<br />
= Release blockers =<br />
<br />
None at this time.<br />
<br />
= New features being considered for ns-3.12 =<br />
<br />
These should be reviewed quickly to see if they can be integrated to this release.<br />
<br />
* [http://codereview.appspot.com/4823051/ Dynamic Source Routing (DSR)]<br />
* [http://codereview.appspot.com/4747046 TDMA model]<br />
* [http://codereview.appspot.com/4644042/ ObjectMap], to add a map to the attribute system instead of a vector<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-August/009255.html RED for ns-3]<br />
<br />
= Bugs being worked =<br />
<br />
We are trying to close these bugs for this release cycle (no guarantees that we will get them all):<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=272 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=388 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=707 Build OpenSolaris]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=827 Wifi] Update RELEASE NOTES<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=845 ConfigStore]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=904 Packet const methods]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=928 variable length array]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=929 variable length array]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=962 path to attributes has bogus entries]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1010 Patch UAN module]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1039 TCP Nagle and RTO] (also bug 1227 is a duplicate)<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1043 LTE constructor problem]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1046 DSDV: NULL callback]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1047 multicast > 16 interfaces] this just needs verification that it is closed<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1050 variable length array] duplicate of 928/929?<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1055 possible UAN bug]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1082 LogComponentEnable() does not provide error handling]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1107 monolithic python bindings] related to 1240<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1112 TCP deadlock]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1127 Netanim compatibility for xcode 4 and wiki updates]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1134 Energy attribute spelling error]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1163 forgot to increment ephemeral port]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1164 IPV4 TCP Bind an already used port failed but without setting errno]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1165 memset used with constant zero length parameter]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1166 ns-3 TCP DCE problems with RST]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1168 BridgeNetDevice: Default ExpirationTime not realistic]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 Implement trivial XML output feature for Animation Interface]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1173 Valgrind detects memory leaks from Click when using ns-3-click]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1176 modules document fails to build]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1177 The auto generated man pages have some error]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1179 Add LTE trace hooks for supporting visualizer]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1180 LTE: Add TX/RX trace support]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1186 Ipv4Header lacks DSCP and ECN] Needs patch<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1192 Some test cases fail to clean up properly (missing DoTeardown)]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1212 attribute path documentation broken]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1216 RvBatteryModel segfalts if the update event happens before load changes]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1217 simple-odfm-wimax-phy creates UniformVariable in a critical path]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1218 Supply a mechanism to build python binding for multi-version of python]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1220 FdReader always stops with NS_FATAL_ERROR]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1226 Incomplete YansWifiChannel Python bindings]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1227 TCP minRTO too low]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1228 modular build problem]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1229 Multiplication overflow in WaypointMobilityModel::EndMobility method]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1237 code cleanups related to include]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1238 documentation]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1239 Add assertion that reference count never overflows]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1240 remove monolithic python bindings from waf]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1241 documentation]<br />
<br />
Also, the collection of "set but not used" bugs (such as bugs 1147-1162). Note that the general solution for bug 1170 is still in work.<br />
<br />
Also, the collection of RealtimeSimulatorImpl and TapBridge patches (Tom Goff is looking at):<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=603 Simulator::Next useless] depends on 631<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=631 RealtimeSimulator and Ctrl-C]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=903 Tap Bridge and Emu Net Devices Do Not Shut Down Properly]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=975 stopping TapBridge]<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1000 Make realtime simulator last until stop]<br />
<br />
= Active reviews sliding past ns-3.12 =<br />
<br />
These are not going to complete in time for the ns-3.12 release.<br />
<br />
* [http://codereview.appspot.com/4517125/ BRITE Integration]<br />
** under revision<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-June/009121.html LTE code from the Lena project]<br />
* [http://codereview.appspot.com/4664057/ NetDevice queue feedback]<br />
** explained here: http://mailman.isi.edu/pipermail/ns-developers/2011-July/009170.html<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1172 Animation interface changes (to support XML output)] <br />
* [http://codereview.appspot.com/2104052/show Virtual Access Point (VAP) for WiFi]<br />
** needs updated to ns-3.11 code; needs review<br />
* [http://codereview.appspot.com/2135046/show GetRelativeVelocity() for MobilityModel]<br />
** needs review<br />
* [http://codereview.appspot.com/2173042/show PhySimWifi]<br />
** needs review<br />
* [http://codereview.appspot.com/2334041/show thread-safe implementation of ScheduleWithContext]<br />
* [http://codereview.appspot.com/1587041/show simple wireless models]<br />
* [http://codereview.appspot.com/1055041/show Jamming model]<br />
* [http://codereview.appspot.com/850045/show MPLS]<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2010-February/007419.html TCP Vegas] (blocked by lack of progress on TCP congestion control architecture)<br />
<br />
= ns-3.12 features merged =<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header Class MinMaxAvgTotalCalculator does not implement all the methods in its interface]<br />
* Fragmentation support for IPv4<br />
<br />
= ns-3.12 bugs fixed =<br />
<br />
This list needs updating.<br />
<br />
* [https://www.nsnam.org/bugzilla/show_bug.cgi?id=1102 IPv4 header FragmentOffset field]<br />
<br />
= Feature requests =<br />
<br />
This lists some features that have been requested but for which a patch is needed.<br />
<br />
* make spectrum model compatible with ns-3 WiFi<br />
* ns-2 packet UID feature <br />
* API for TOS bytes (issue 897) ''may wait for netfilter support''<br />
* Chord/DHT (authors are planning to revise code based on comments)<br />
* app store and build system refactoring<br />
* fragmentation for IPv4: being worked by Vedran Miletic and Tommaso Pecorella<br />
* TDMA wireless model<br />
** being worked on by Hemanth Narra<br />
* Netdevice queue state support<br />
** being worked on by Ruben Merz</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5914HOWTO get ns-3 to detect steady-state times in your data2011-07-28T17:02:29Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
* Batch Means Test<br />
** '''White, K., Cobb, M., & Spratt, S. (2000)'''. A Comparison of Five Steady-state Truncation Heuristics for Simulation. ''Proceedings of the 32nd Winter Simulation Conference (WSC '00)'', 755–760.<br />
*** In this reference, they say: "Cash, et al. (1992), Nelson (1992), and Goldsman, et al. (1994), propose a family of related tests for detecting the presence of bias in an output series. These tests generalize and extend the earlier work Schruben (1982) and Schruben, et al. (1983)."<br />
** '''Cash, C.R., Dippold, D.G., Nelson, B.L., Pollard, W.P., and Long, J.M. (1992)'''. Evaluation of Tests for Initial-condition Bias. ''Proceedings of the 1992 Winter Simulation Conference'', 577-585.<br />
** Algorithm: Bias detection test.<br />
** Reason: Steady-state detection method can detect if not in steady-state (unlike MSER-5).<br />
** Reason: Single-replicate algorithm.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5902HOWTO get ns-3 to detect steady-state times in your data2011-07-25T18:17:16Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
* Batch Means Test<br />
** '''White, K., Cobb, M., & Spratt, S. (2000)'''. A Comparison of Five Steady-state Truncation Heuristics for Simulation. ''Proceedings of the 32nd Winter Simulation Conference (WSC '00)'', 755–760.<br />
*** In this reference, they say: "Cash, et al. (1992), Nelson (1992), and Goldsman, et al. (1994), propose a family of related tests for detecting the presence of bias in an output series. These tests generalize and extend the earlier work Schruben (1982) and Schruben, et al. (1983)."<br />
** Algorithm: Bias detection test.<br />
** Reason: Steady-state detection method can detect if not in steady-state (unlike MSER-5).<br />
** Reason: Single-replicate algorithm.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5901HOWTO get ns-3 to detect steady-state times in your data2011-07-25T17:43:03Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
*** In this reference, they call Pawlikowski (1990), the "Goodness-Of-Fit method", and they say: "Both the goodness of fit method and Kimbler’s double exponential smoothing method consistently and severely underestimated the truncation point and were therefore rejected."<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place. [[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
** '''Hoad, K., Robinson, S., & Davies, R. (2008)'''. Automating Warm-Up Length Estimation. ''Proceedings of the 2008 Winter Simulation Conference'', 532–540.<br />
*** In this reference, they say: "For example, the crossing-of-the-mean rule (Fishman 1973, Wilson and Pritsker 1978a, 1978b) was heavily criticised in the literature for being extremely sensitive to the selection of its main parameter, which was system-dependent, and misspecification of which caused significant over or under-estimation of the warm-up length (Pawlikowski 1990)."<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5870HOWTO get ns-3 to detect steady-state times in your data2011-07-19T23:57:07Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
* Schruben's test applied after 25 crossings of the running Mean<br />
** '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
*** In this reference, they say: "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5869HOWTO get ns-3 to detect steady-state times in your data2011-07-19T19:57:48Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Algorithms Rejected without Trying ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms Rejected after Trying ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
** Note: The following plot shows a simulated data set with a linearly increasing bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-linearly-increasing-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** '''Note: The problem with the MSER-5 algorithm is that the data set with a linearly increasing bias in the above plot does NOT have a steady-state time, and so the algorithm should NOT be finding one. So, the MSER-5algorithm is being REJECTED.'''<br />
<br />
== Algorithms Currently Trying ==<br />
<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-linearly-increasing-bias.png&diff=5868File:Steady-state-detector-test-linearly-increasing-bias.png2011-07-19T19:56:16Z<p>Watrous: Linearly increasing bias test.</p>
<hr />
<div>Linearly increasing bias test.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5866HOWTO get ns-3 to detect steady-state times in your data2011-07-18T21:17:43Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms Tried ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showed that the MSER-5 method performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings before the MSER-5 method is tried, which means the system is in steady-state and which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: My code seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5865HOWTO get ns-3 to detect steady-state times in your data2011-07-18T21:06:56Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms Tried ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method applied after 25 crossings of the running Mean<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showd it performed the best of the five methods tried.<br />
** '''McNickle, D., Ewing, G.C., and Pawlikowski, K.. (2007)'''. Transient Deletion And The Quality Of Sequential Steady-State Simulation, ''Proceedings 21st European Conference on Modelling and Simulation''.<br />
*** In this reference, they say that the Akaroa2 simulator requires 25 crossings of the running mean of the values before it performs its Schruben’s test.<br />
** Summary: MSER-5 Tries to reduce the mean squared error.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: The MSER-5 method alone was saying the system was in steady-state with as few as 2 data values.<br />
** Note: My code now requires 25 mean crossings, which means the system is in steady-state before the MSER-5 method is tried, which is the situation the MSER-5 method was intended to work with.<br />
** Note: The following plot shows a simulated data set with an exponential bias along with the steady-state time that my implementation of the MSER-5 algorithm (after 25 mean crossings) is currently returning. The plot also shows vertical lines where the steady-state time is and where the 25th mean crossing takes place.<br />
[[Image:steady-state-detector-test-exponential-bias.png|540px|center|Simulated data set with steady-state time.]]<br />
** Note: It seems to be giving a better steady-state time than just using the MSER-5 for the first 1,000 points, but it did require almost 2,000 points to get 25 mean crossings.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=File:Steady-state-detector-test-exponential-bias.png&diff=5864File:Steady-state-detector-test-exponential-bias.png2011-07-18T21:01:17Z<p>Watrous: Simulated data set with steady-state time.</p>
<hr />
<div>Simulated data set with steady-state time.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5819HOWTO get ns-3 to detect steady-state times in your data2011-06-23T18:45:25Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Pawlikowski, K., McNickle, D., and Ewing, G. (1998)'''. Coverage of Confidence Intervals from Sequential Steady-state Simulation, ''Simulation Practice and Theory'', 6:255-267.<br />
** Algorithm: Confidence intervals.<br />
** Reason: Confidence interval algorithm. <br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showd it performed the best of the five methods tried.<br />
** Summary: Tries to reduce the mean squared error.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5818HOWTO get ns-3 to detect steady-state times in your data2011-06-23T17:55:06Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms to Try (In order that they should be tried) ==<br />
<br />
* Mean Squared Error Reduction (MSER-5) method<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
** '''White, K.P. and Cobb, M.J. (2000)'''. A Comparison of Five Steady-State Truncation Heuristics for Simulation. ''Proceedings of the 2000 Winter Simulation Conference''.<br />
*** This reference showd it performed the best of the five methods tried.<br />
** Summary: Tries to reduce the mean squared error.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Summary: Standardized time series methods for steady-state detection.<br />
** Advantages: Single-replicate algorithm.<br />
** Disadvantages: <br />
** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
** Summary: Groups values into batches and computes mean of each batch.<br />
** Advantages: <br />
** Disadvantages: <br />
** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5817HOWTO get ns-3 to detect steady-state times in your data2011-06-23T01:05:09Z<p>Watrous: </p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
*** Summary: 2 stage method:<br />
**** Stage 1: has multiple replications.<br />
**** Stage 2: has single seed.<br />
*** Advantages: Works with a single seed value for stage 2.<br />
*** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
*** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
*** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
*** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
*** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
*** Note: Has equations for an automated version of Welch's visual method.<br />
*** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
*** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms to Try ==<br />
<br />
* Schruben's Maximum Test <br />
** '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
*** Summary: Standardized time series methods for steady-state detection.<br />
*** Advantages: Single-replicate algorithm.<br />
*** Disadvantages: <br />
*** Note: <br />
<br />
* Schruben's Optimal Test<br />
** '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
*** Summary: Standardized time series methods for steady-state detection.<br />
*** Advantages: Single-replicate algorithm.<br />
*** Disadvantages: <br />
*** Note: <br />
<br />
* Mean Squared Error Reduction (MSER-5) method<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
*** Summary: Tries to reduce the mean squared error.<br />
*** Advantages: <br />
*** Disadvantages: <br />
*** Note: <br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
*** Summary: Groups values into batches and computes mean of each batch.<br />
*** Advantages: <br />
*** Disadvantages: <br />
*** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5816HOWTO get ns-3 to detect steady-state times in your data2011-06-22T21:52:10Z<p>Watrous: /* Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* Statistical Process Control (SPC) Approach<br />
** '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
*** Summary: 2 stage method:<br />
**** Stage 1: has multiple replications.<br />
**** Stage 2: has single seed.<br />
*** Advantages: Works with a single seed value for stage 2.<br />
*** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
*** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
* Multiple (3) steady-state algorithms in an Excel based tool.<br />
** '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
*** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
*** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
*** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
*** Note: Has equations for an automated version of Welch's visual method.<br />
*** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
*** Note: Describes how to automate the MSER-5 method.<br />
<br />
== Algorithms to Try ==<br />
<br />
* Batch means<br />
** '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill. '''(See pp. 528–529.)'''<br />
*** Summary: Groups values into batches and computes mean of each batch.<br />
*** Advantages: <br />
*** Disadvantages: <br />
*** Note: <br />
<br />
* Mean Squared Error Reduction (MSER-5) method<br />
** '''White, K.P. and Robinson, S. (2010)'''. The Problem of the Initial Transient (Again) or Why MSER Works. ''Journal of Simulation'' 4, 268-272.<br />
** '''Franklin, W.W. and White, K.P. (2008)'''. Stationarity Tests and MSER-5: Exploring the Intuition behind Mean-Squared-Error-Reduction in Detecting and Correcting Initialization Bias. ''Proceedings of the 2008 Winter Simulation Conference''. <br />
*** Summary: Tries to reduce the mean squared error.<br />
*** Advantages: <br />
*** Disadvantages: <br />
*** Note:</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5815HOWTO get ns-3 to detect steady-state times in your data2011-06-22T18:25:26Z<p>Watrous: /* SAFE Framework */</p>
<hr />
<div>= Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== References to Get ==<br />
<br />
* '''Law, A. M. (2007)'''. Simulation Modeling and Analysis. 4th ed. New York: McGraw-Hill.<br />
** Algorithm: Batch means for steady-state detection. (See pp. 528–529.)<br />
** Reason: An overview of simulation.<br />
** Reason: An overview of single-replicate techniques.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Anderson, T. W. (1994)'''. The Statistical Analysis of Time Series. New York: Wiley. <br />
** Algorithm: Spectral methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. (1982)'''. Detecting Initialisation Bias in Simulation Output. ''Operations Research'', 30(3), 569-590.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L. W. (1983)'''. Confidence Interval Estimation using Standardized Time Series. ''Operations Research'' 31:1090–1108.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Schruben, L., Singh, H., and Tierney, L. (1983)'''. Optimal Tests for Initialisation Bias in Simulation Output. ''Operations Research'', 31( 6), 1167-1178.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: Single-replicate algorithm.<br />
<br />
* '''Pawlikowski, K, (1990)''', Steady State Simulation of Queueing Processes: a Survey of Problems and Solutions, ''ACM Computing Surveys'', 22, June 1990, 123-170.<br />
** Algorithm: Standardized time series methods for steady-state detection.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: "Details of our sequential implementation of a Schruben-based method can be found in Section 3 of Pawlikowski (1990)."<br />
** Reason: (From: Transient Deletion And The Quality Of Sequential Steady-State Simulation, McNickle, Ewing, and Pawlikowski. ''Proceedings 21st European Conference on Modelling and Simulation.'' 2007.) "Sequential Spectral Analysis, a modification of the method proposed by Heidelberger and Welch (1981) and specified in Pawlikowski (1990), was used to estimate the confidence interval width. We have found that this method gives accurate confidence intervals, especially for highly correlated data, such as waiting times in highly loaded queues (Ewing, McNickle and Pawlikowski 2002, McNickle, Pawlikowski, and Ewing, 2004).<br />
<br />
* '''Nakayama, M. K. (1994)'''. Two-stage Stopping Procedures Based on Standardized Time Series. ''Management Science'' 40:1189–1206.<br />
** Algorithm: Standardized time series methods for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Nakayama presents two-stage procedures for obtaining fixed-width confidence intervals using standardized time series methods.<br />
<br />
* '''Glynn, P. W., and W. Whitt. (1992)'''. The Asymptotic Validity of Sequential Stopping Rules in Stochastic Simulations. ''Annals of Applied Probability'' 2:180–198. <br />
** Algorithm: Sequential procedures for confidence intervals.<br />
** Reason: '''Might be a single-replicate algorithm.'''<br />
** Reason: Confidence interval algorithm. <br />
** Reason: Glynn and Whitt consider several so-called sequential procedures.<br />
<br />
== Rejected Algorithms ==<br />
<br />
* '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms to Try ==<br />
<br />
* '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the batch mean method. See Robinson (2002) in Rejected Algorithms section for equations for batch mean method.<br />
** Note: Describes how to automate the MSER-5 (White, K.P., M.J. Cobb and S.C. Spratt. 2000. A Comparison of Five Steady-State Truncation Heuristics for Simulation. Proceedings of the 2000 Winter Simulation Conference (J.A. Joines, R.R. Barton, K. Kang and P.A. Fishwick, eds.). IEEE, Piscataway, NJ, 755-760.) method.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_get_ns-3_to_detect_steady-state_times_in_your_data&diff=5814HOWTO get ns-3 to detect steady-state times in your data2011-06-20T22:34:55Z<p>Watrous: Created page with "= SAFE Framework = == Steady-state detectors == The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached ..."</p>
<hr />
<div>= SAFE Framework =<br />
<br />
== Steady-state detectors ==<br />
<br />
The architecture of SAFE provides for the use of mechanisms to determine when metrics estimated through simulation have reached steady-state. Although well known, the initialization bias problem hasn't been addressed satisfactorily in the domain of network simulation. Few published research studies have taken steps to exclude samples generated during model transients from their statistical data analysis. Among other researchers who have identified the relevance of initialization bias in network simulation, "Perrone, Yuan, and Nicol [2003]":http://redmine.eg.bucknell.edu/safe/attachments/16/perrone2003.pdf report the significance of avoiding the use of samples from transient in the computation of statistical estimators for metrics of interest.<br />
<br />
SAFE enables one to hook up a source of samples to an analysis module, which determines on its own whether the sample data is "in transient" or in steady-state. <br />
<br />
The development of _steady-state detectors_ for SAFE is looking at a broad range of publications that will culminate in the identification of sound data analysis methodology. SAFE will use the information provided by steady-state detectors to implement data deletion in the samples collected thereby avoiding initialization bias.<br />
<br />
== Current Stage of Development ==<br />
<br />
This work is at an investigation stage. We are currently going through the literature to identify algorithms that will be suitable for use in SAFE steady-state detectors. <br />
<br />
== Projected Milestones ==<br />
<br />
* Literature search and evaluation: June 8-14, 2011<br />
* First implementation: June 15-30, 2011<br />
* Evaluation of implementation against synthetic data generator: July 1-20, 2011 <br />
<br />
== Rejected Algorithms ==<br />
<br />
* '''Robinson, Stewart (2002)'''. A Statistical Process Control Approach for Estimating the Warm-Up Period. ''Proceedings of the 2002 Winter Simulation Conference''.<br />
** Summary: 2 stage method:<br />
*** Stage 1: has multiple replications.<br />
*** Stage 2: has single seed.<br />
** Advantages: Works with a single seed value for stage 2.<br />
** Disadvantages: Requires visual inspection of final results and initial preliminary runs from Stage 1.<br />
** Note: Has equations for batch mean method, which works for single seeds.<br />
<br />
== Algorithms to Try ==<br />
<br />
* '''Robinson, Stewart (2005)'''. Automated Analysis of Simulation Output Data. ''Proceedings of the 2005 Winter Simulation Conference''.<br />
** Summary: Paper describes an Excel based semi-automated tool that calls SIMUL8. <br />
** Advantages: Automated methods. Describes batch method which works for single seed values.<br />
** Disadvantages: Based on Excel and SIMUL8, which means can't be called from ns-3.<br />
** Note: Has equations for an automated version of Welch's visual method.<br />
** Note: Describes how to automate the MSER-5 (White et al, 2000) method.<br />
*** White, K.P., M.J. Cobb and S.C. Spratt. 2000. A Comparison of Five Steady-State Truncation Heuristics for Simulation. Proceedings of the 2000 Winter Simulation Conference (J.A. Joines, R.R. Barton, K. Kang and P.A. Fishwick, eds.). IEEE, Piscataway, NJ, 755-760.<br />
** Note: Describes how to automate the batch mean method.<br />
** Note: See Robinson (2002) in rejected algorithms for equations for batch mean method.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTOs&diff=5813HOWTOs2011-06-20T22:12:07Z<p>Watrous: /* Data Analysis-Related Items */</p>
<hr />
<div>{{TOC}}<br />
== How to do Various Interesting Things with ns-3 ==<br />
<br />
=== System-Related Items ===<br />
<br />
* [[HOWTO make ns-3 interact with the real world]]<br />
* [[HOWTO use VirtualBox to run simulations on Windows machines]]<br />
* [[HOWTO get ns-3 running on Mac OS X (10.5.2 Intel)]]<br />
* [[HOWTO get ns-3 running on Mac OS X (10.6.2 Intel)]]<br />
* [[HOWTO Use Linux Containers to set up virtual networks]]<br />
* [[HOWTO use VMware to set up virtual networks (Windows)]]<br />
* [[HOWTO use ns-3 scripts to drive real hardware (experimental)]]<br />
* [[HOWTO use ns-3 directly on the ORBIT testbed hardware]]<br />
* [[HOWTO use ns-3 in the ORBIT testbed environment]]<br />
* [[HOWTO use ns-3 directly on the CMU wireless emulator]]<br />
* [[HOWTO use ns-3 with other libraries]]<br />
* [http://www.nsnam.org/wiki/index.php/Ns3_on_windows HOWTO use ns-3 on Windows - Microsoft Visual Studio 2010 only]<br />
<br />
=== Programming-Related Items ===<br />
* [[HOWTO make and use a new application]]<br />
* [[HOWTO extract the IP src/dst address from a packet]]<br />
* [[HOWTO create a new type of protocol header or trailer]]<br />
* [[HOWTO use null callbacks]]<br />
* [[HOWTO create a new OSI layer 1 + 2 implementation]]<br />
* [[HOWTO create a traffic generator]]<br />
* [[HOWTO configure eclipse with ns-3]]<br />
* [[HOWTO determine the path of an attribute or trace source]]<br />
* [[HOWTO use the ConfigStore]]<br />
* [[HOWTO determine all the available Values]]<br />
* [http://www.eg.bucknell.edu/~perrone/2010/08/19/looking-at-ns-3-packet-traces/ HOWTO work with ns-3 packet traces]<br />
* [http://www.eg.bucknell.edu/~perrone/2010/08/27/creating-a-new-module-in-ns-3/ HOWTO create a new module in ns-3]<br />
* [[HOWTO enable a subset of ns-3's modules]]<br />
* [[HOWTO enable/disable examples and tests in ns-3]]<br />
<br />
=== Debugging-Related Items ===<br />
* [[HOWTO use NS_LOG to find a problem]]<br />
* [[HOWTO understand and find cause of terminated with signal errors]]<br />
* [[HOWTO debug smart pointer | HOWTO debug smart pointers and packet pointers]]<br />
* [[HOWTO use Valgrind to debug memory problems]]<br />
<br />
=== Mercurial-Related Items ===<br />
* [[HOWTO use Mercurial Queues to manage your ns-3 patches]]<br />
<br />
=== Data Analysis-Related Items ===<br />
* [[HOWTO get ns-3 data into SciPy]]<br />
* [[HOWTO get ns-3 data into ROOT]]<br />
* [[HOWTO get ns-3 data into SciLab]]<br />
* [[HOWTO get ns-3 data into Octave]]<br />
* [[HOWTO get ns-3 to detect steady-state times in your data]]<br />
<br />
== Blogs and Personal Pages about ns-3 ==<br />
* [http://gicl.cs.drexel.edu/people/tjkopena/wiki/pmwiki.php?n=NS3.NS3 Joseph Kopena's ns-3 wiki]<br />
* [http://eugen.dedu.free.fr/ns3-tutorial.html Eugen Dedu's tutorial]<br />
* [http://www.eg.bucknell.edu/~perrone/tag/ns-3/ Felipe Perrone's blog]<br />
<br />
----<br />
[[User:Craigdo|Craigdo]] 16:34, 22 April 2010 (UTC)</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=HOWTO_enable_a_subset_of_ns-3%27s_modules&diff=5655HOWTO enable a subset of ns-3's modules2011-05-10T21:16:51Z<p>Watrous: /* Enable modules using the ns-3 configuration file */</p>
<hr />
<div>{{TOC}}<br />
<br />
As with most software projects, ns-3 is ever growing larger in terms of number of modules, lines of code, and memory footprint. <br />
<br />
Users, however, may only use a few of those modules at a time. Therefore, users may want to explicitly enable only the subset of the possible ns-3 modules that they actually need for their research.<br />
<br />
This HOWTO discusses how to enable only the ns-3 modules that you are intersted in using.<br />
<br />
== HOWTO enable a subset of ns-3's modules ==<br />
<br />
If shared libraries are being built, then enabling a module will cause at least one library to be built: libns3-modulename.so. If the module has a test library and test libraries are being built, then libns3-modulename-test.so will be built, too. Other modules that the module depends on and their test libraries will also be built. <br />
<br />
By default, all modules are built in ns-3. <br />
<br />
There are two ways to enable a subset of ns-3's modules:<br />
# Using waf's --enable-modules option<br />
# Using the ns-3 configuration file<br />
<br />
=== Enable modules using waf's --enable-modules option ===<br />
<br />
To enable only the core module with example and tests, for example,<br />
try these commands:<br />
<br />
./waf clean<br />
./waf configure --enable-examples --enable-tests --enable-modules=core<br />
./waf build<br />
cd build/debug/<br />
ls<br />
<br />
and the following libraries should be present:<br />
<br />
bindings libns3-core.so ns3 scratch utils<br />
examples libns3-core-test.so samples src<br />
<br />
Running test.py will cause only those tests that depend on module core to be run:<br />
<br />
24 of 24 tests passed (24 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors)<br />
<br />
Repeat the above steps for the "network" module instead of the "core" module, and the following will be built, since network depends on core:<br />
<br />
bindings libns3-core.so libns3-network.so ns3 scratch utils<br />
examples libns3-core-test.so libns3-network-test.so samples src<br />
<br />
Running test.py will cause those tests that depend on only the core and network modules to be run:<br />
<br />
31 of 31 tests passed (31 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors)<br />
<br />
=== Enable modules using the ns-3 configuration file ===<br />
<br />
A configuration file, .ns3rc, has been added to ns-3 that allows users to specify which modules are to be included in the build.<br />
<br />
When enabling a subset of ns-3 modules, the precedence rules are as follows:<br />
<br />
# the --enable-modules configure string overrides any .ns3rc file<br />
# the .ns3rc file in the top level ns-3 directory is next consulted, if present<br />
# the system searches for ~/.ns3rc if the above two are unspecified<br />
<br />
If none of the above limits the modules to be built, all modules that waf knows about will be built.<br />
<br />
The maintained version of the .ns3rc file in the ns-3-dev repository resides in the ns-3-dev/utils directory. The reason for this is if it were in the top-level directory, ns-3-dev, it would be prone to accidental checkins from maintainers that enable the modules they want to use. Therefore, users need to manually copy the .ns3rc from the utils directory to their preferred place (top level directory or their home directory) to enable persistent modular build configuration.<br />
<br />
Assuming that you are in the ns-3-dev directory, you can get a copy of the .ns3rc file that is in the utils directory as follows:<br />
<br />
cp utils/.ns3rc .<br />
<br />
The .ns3rc file should now be in your ns-3-dev directory, and it contains the following:<br />
<br />
#! /usr/bin/env python<br />
<br />
# A list of the modules that will be enabled when ns-3 is run.<br />
# Modules that depend on the listed modules will be enabled also.<br />
#<br />
# All modules can be enabled by choosing 'all_modules'.<br />
modules_enabled = ['all_modules']<br />
<br />
# Set this equal to true if you want examples to be run.<br />
examples_enabled = False<br />
<br />
# Set this equal to true if you want tests to be run.<br />
tests_enabled = False<br />
<br />
Use your favorite editor to modify the .ns3rc file to only enable the core module with examples and tests like this:<br />
<br />
#! /usr/bin/env python<br />
<br />
# A list of the modules that will be enabled when ns-3 is run.<br />
# Modules that depend on the listed modules will be enabled also.<br />
#<br />
# All modules can be enabled by choosing 'all_modules'.<br />
modules_enabled = ['core']<br />
<br />
# Set this equal to true if you want examples to be run.<br />
examples_enabled = True<br />
<br />
# Set this equal to true if you want tests to be run.<br />
tests_enabled = True<br />
<br />
Only the core module will be enabled now if you try these commands:<br />
<br />
./waf clean<br />
./waf configure<br />
./waf build<br />
cd build/debug/<br />
ls<br />
<br />
and the following libraries should be present:<br />
<br />
bindings libns3-core.so ns3 scratch utils<br />
examples libns3-core-test.so samples src<br />
<br />
Running test.py will cause only those tests that depend on module core to be run:<br />
<br />
24 of 24 tests passed (24 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors)<br />
<br />
Repeat the above steps for the "network" module instead of the "core" module, and the following will be built, since network depends on core:<br />
<br />
bindings libns3-core.so libns3-network.so ns3 scratch utils<br />
examples libns3-core-test.so libns3-network-test.so samples src<br />
<br />
Running test.py will cause those tests that depend on only the core and network modules to be run:<br />
<br />
31 of 31 tests passed (31 passed, 0 skipped, 0 failed, 0 crashed, 0 valgrind errors)</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.11&diff=5654Ns-3.112011-05-10T21:10:44Z<p>Watrous: /* To Do List */</p>
<hr />
<div>{{TOC}}<br />
This page summarizes the ongoing release planning for ns-3.11. Josh Pelkey <jpelkey@gatech.edu> will manage the release.<br />
<br />
= Proposed Release Schedule =<br />
<br />
# January 5 -- ns-3.10 posted<br />
# January 5 -- ns-3.11 '''''Open phase begins'''''<br />
# March 18 -- Deadline for new feature merge<br />
# March 18 -- Begin the phase of small feature development and bug fixing<br />
# April 6 -- Small feature development and bug fixing ends<br />
# April 6 -- '''''Open phase ends'''''<br />
# April 6 -- '''''Maintenance phase begins'''''<br />
# May 15 -- '''''Maintenance phase ends'''''<br />
# May 15 -- '''''Code freeze phase begins'''''<br />
# May 15 -- ns-3.11-RC1<br />
# May 17 -- ns-3.11-RC2<br />
# May 19 -- ns-3.11-RC3<br />
# May 22 -- ns-3.11 posted<br />
# May 22 -- '''''Code freeze phase ends'''''<br />
# May 22 -- ns-3.12 Open phase begins<br />
<br />
= To Do List =<br />
<br />
'''For first release candidate'''<br />
<br />
* Mitch Watrous is working on the following modular build-related items:<br />
** Update documentation: how to use the template module<br />
** Update documentation: how to create a new module<br />
** Update documentation: how to use the configuration (.ns3rc) file<br />
<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-April/008797.html Documentation updates] are planned:<br />
** Update project documentation for modular build changes (Mitch Watrous)<br />
** Split existing manual to a "developers manual" and a "model library" manual<br />
** Remove "testing" document; move pieces to developers manual and model library manual<br />
** Add python page to manual<br />
** Remove [http://mailman.isi.edu/pipermail/ns-developers/2011-March/008742.html duplicate doxygen documentation]<br />
** Create module template that can be easily cloned<br />
** Create datasheet for ns-3<br />
** Create cheatsheet(s) for ns-3<br />
** Write down duties of release manager somewhere<br />
<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2011-April/008816.html Coding style update] will be applied<br />
<br />
* Move python examples to modular bindings<br />
<br />
<br />
'''For final release'''<br />
<br />
* Fix AODV [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1099 Bug 1099] - AODV performance problems<br />
<br />
* Other possible bugs:<br />
** TCP deadlock<br />
** int 64x64 overflow (needs confirmed)<br />
** 555 DCF (check also whether 1011 is invalid)<br />
** 1047,1054,1056,1058 (Tom H)<br />
** 1087 (MPI finalize?)<br />
** nsc on gcc-4.6.0 (1093)<br />
** rounding errors in mobility model (976)?<br />
<br />
* Review/update tutorial and documentation<br />
* Release new website<br />
<br />
= ns-3.11 Features Merged =<br />
* [http://www.nsnam.org/wiki/index.php/App_Store_Technical_Requirements modular build]<br />
* [http://codereview.appspot.com/3988043/show ns-3 click]<br />
* [http://codereview.appspot.com/4266051/show ns-3 openflow]<br />
<br />
= Bugs fixed for ns-3.11 =<br />
<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=445 445] -- Is the class name Scalar in nstime.h appropriate?<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=699 699] -- TestCase::DoRun probably should not return a bool<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1017 1017] -- node --> internet-stack --> node<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1018 1018] -- mobility --> helper --> mobility circular dependency<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1019 1019] -- common --> node --> common circular dependency<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1038 1038] -- Time::Get*Seconds () return signed integer while actually returning unsigned.<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1040 1040] -- Olsr and multiple interfaces<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1044 1044] -- Seconds (1e-9) creates Time that is not IsPositive ()<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1048 1048] -- suggested MatrixPropagationLossModel::SetLoss() API change<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1049 1049] -- Issues with OLSR HNA messages sent by multiple gateways<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1052 1052] -- olsr::RoutingProtocol::LinkTupleUpdated incorrect detection of asymetric link with multiple interface nodes<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1060 1060] -- Setting NqStaWifiMac Active Probing true, crashes simulation<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1061 1061] -- Title level inconsistent in click.rst<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1062 1062] -- rescale-pdf not working<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1064 1064] -- Correct Friis propagation loss equation in spectrum module<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1066 1066] -- Set Radiotap modulation type flags correctly<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1069 1069] -- ApWifiMac unduly calls RegularWifiMac::Receive and crashes<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1070 1070] -- csma/ipv6 examples duplicate NS_LOG_COMPONENT_DEFINE<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1075 1075] -- Python examples fail when static built is enabled<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1076 1076] -- Waf gives an error if you enable only a single module<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1078 1078] -- PacketTagList::Add performs NS_ASSERT counting one byte too few<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1079 1079] -- MPI code doesn't compile<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1081 1081] -- wutils.py refers to the old directory for visualizer module<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1083 1083] -- scratch doesn't work when only some modules are enabled<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1085 1085] -- Build failed: list index out of range<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1086 1086] -- sanity check for not calling Object::Dispose () twice<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1088 1088] -- Code in scratch directory assumes all modules are enabled<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1089 1089] -- lwip build issue<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1090 1090] -- WifiMacQueue is not accessible through the attribute path<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1092 1092] -- Problem building FreeBSD stack with g++ 4.5.2 (Ubuntu)<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1098 1098] -- when to show users which modules are enabled?<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1105 1105] -- Move topology helpers into separate per-device modules<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1106 1106] -- Remove "CanvasLocation" dependencies in p2p helpers<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1108 1108] -- test.py -e option needs repair<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1110 1110] -- Sort alphabetically and display "Modules built" in columns<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1117 1117] -- bindings failures on selected platforms</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.11&diff=5621Ns-3.112011-04-29T17:10:30Z<p>Watrous: /* To Do List */</p>
<hr />
<div>{{TOC}}<br />
This page summarizes the ongoing release planning for ns-3.11. Josh Pelkey <jpelkey@gatech.edu> will manage the release.<br />
<br />
= Proposed Release Schedule =<br />
<br />
# January 5 -- ns-3.10 posted<br />
# January 5 -- ns-3.11 '''''Open phase begins'''''<br />
# March 18 -- Deadline for new feature merge<br />
# March 18 -- Begin the phase of small feature development and bug fixing<br />
# April 6 -- Small feature development and bug fixing ends<br />
# April 6 -- '''''Open phase ends'''''<br />
# April 6 -- '''''Maintenance phase begins'''''<br />
# April 30 -- '''''Maintenance phase ends'''''<br />
# April 30 -- '''''Code freeze phase begins'''''<br />
# April 30 -- ns-3.11-RC1<br />
# May 3 -- ns-3.11-RC2<br />
# May 6 -- ns-3.11-RC3<br />
# May 8 -- ns-3.11 posted<br />
# May 8 -- '''''Code freeze phase ends'''''<br />
# May 8 -- ns-3.12 Open phase begins<br />
<br />
As described in the [[Roadmap]] there are three broad sections in the release schedule. During the open phase, people wanting to include a new feature in ns-3.11 should contact Josh and arrange to have their features merged into ns-3-dev. You will be expected to provide the following:<br />
<br />
* A mercurial patch, bundle or repo against the current version of ns-3-dev that contains your proposed feature addition. You need to make sure that we can apply this patch and build and run (debug and optimized as appropriate) all unit and regression tests successfully on all of our target machines<br />
* A summary of the additions you are proposing and an explanation of any changes to existing code that had to be done in order to support your feature (this will be used to generate release notes and will be provided to maintainers if a code review is indicated)<br />
* Some kind of unit or system test that can be used to determine if your feature is actually working at each stage of the integration<br />
<br />
One of us will take a quick look at your proposed addition and determine if a code review is required. According to the ''book of instructions'' a code review requiring positive acknowledgment by maintainers is indicated if:<br />
<br />
* Your proposed feature does not work with all models or on all platforms<br />
* Your feature changes pre-existing APIs<br />
* Your feature crosses maintainer boundaries<br />
<br />
Just to be safe, we will probably run a feature submission by at least one maintainer according to the general area of applicability of the feature. For example, if you submit an entirely new device driver model, as a courtesy we will run this submission by the maintainers of the current devices. The maintainers won't have any responsibility to positively ack the submission, but we will take some time to allow a reasonable review.<br />
<br />
The open phase is divided into two parts: new feature merge period and small feature development and bug fixing. <br />
<br />
During the new feature merge period we can:<br />
* Review the code that might be merged<br />
* Clean up the bug tracker: solve as much bugs as possible<br />
* Merge the new features that has +1 and from maintainers<br />
<br />
After February 14th, the new feature merge period ends and small feature development and bug fixes begins. During the latter, no more merges are accepted and we can:<br />
* Accept limited, small, self contained changes/features to ns-3-dev and to merged new features. As specified before, no more merges are accepted<br />
* Review the code to be merged for next releases<br />
* Cleanup the ns-3-dev bug tracker, solve as much bugs as possible<br />
<br />
The end of the small feature development and bug fixes coincides (March 5th) with the beginning of the maintenance phase. No new features may be added, but the maintainers may check in fixes to bugs; and people with new features that have been accepted and previously merged may fix bugs in existing features. Please don't try to sneak in more new features or you may have your whole feature set removed at the release manager's discretion. You can ask if you want to add small, self-contained features, but there are no guarantees that we will okay them.<br />
<br />
On March 30th, 2011, we are going to enter the code freeze phase. This indicates that we are in the final stages of the release and our primary goal is stability. During the code freeze phase, only P1 bugfixes will be allowed to be checked in. Our ''goal'' will be to reduce the number of P1 bugs to zero before the release of ns-3.11.<br />
<br />
'''''We will reserve the right to veto (and remove) any new feature addition if it begins to cause problems and looks like it threatens the stability of the release at any time in the release process.'''''<br />
<br />
= Candidate Contributions for Inclusion in ns-3.11 =<br />
<br />
<br />
* [http://codereview.appspot.com/2104052/show Virtual Access Point (VAP) for WiFi]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2135046/show GetRelativeVelocity() for MobilityModel]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2173042/show PhySimWifi]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2334041/show thread-safe implementation of ScheduleWithContext]<br />
* [http://codereview.appspot.com/1587041/show simple wireless models]<br />
* [http://codereview.appspot.com/1055041/show Jamming model]<br />
* [http://codereview.appspot.com/850045/show MPLS]<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2010-February/007419.html TCP Vegas] (blocked by lack of progress on TCP congestion control architecture)<br />
* [http://codereview.appspot.com/3374041/ NetAnim trace hooks for wireless]<br />
<br />
= ns-3.11 Features Merged =<br />
* [http://codereview.appspot.com/3988043/show ns-3 click]<br />
* [http://codereview.appspot.com/4266051/show ns-3 openflow]<br />
<br />
= The ns-3.11 Bug List =<br />
<br />
'''Highest Priority'''<br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1038 1038] -- Time::Get*Seconds () return signed integer while actually returning unsigned.</s><br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1044 1044] -- Seconds (1e-9) creates Time that is not IsPositive ()</s><br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=445 445] -- Class name rename Scalar->Dimensionless</s><br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=976 976] -- wifi-wired-bridging regression test fails because of rounding errors in mobility model<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1042 1042] -- AODV RERR implosion (missing RERR_RATELIMIT)<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1079 1079] -- MPI code doesn't compile<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1095 1095] -- MPI enable configuration failed in Fedroa 14<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1099 1099] -- AODV performance problems<br />
<br />
<br />
'''High Priority'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1033 1033] -- airtime-metric <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=409 409] -- Routing messages can exceed MTU, and fragmentation not supported <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=631 631] -- RealtimeSimulatorImpl not compatible with python bindings <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=555 555] -- DCF immediate access bug<br />
** patch existing / needs further testing and verification<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=521 521] -- Ipv4 global routing inefficient<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938] -- missing Doxygen <br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1018 1018] -- mobility --> helper --> mobility circular dependency</s><br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1017 1017] -- node --> internet-stack --> node</s><br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=756 756] -- Build should be configurable to avoid using optional components<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=912 912] -- modeling processing delays<br />
<br />
<br />
'''Possibly easy fixes'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=976 976] -- udp tx buffer is not fixed size<br />
** possibly WONTFIX<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1006 1006] -- UDP socket tx buffer back pressure needed<br />
**somewhat related to bug 141<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=996 996] -- TCP FIN-WAIT-2 bug<br />
** pinged Adrian about this one, likely fixed<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=272 272] --InternetStackHelper::Install does not mention the fact that it aggregates PacketSocketFactory<br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1038 1038] -- Time::Get*Seconds () return signed integer while actually returning unsigned</s><br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=798 798] -- In test.py: CRASH: TestSuite ns3-tcp-cwnd CRASH: TestSuite ns3-tcp-interoperability.<br />
** is this still valid?<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=957 957] -- Issue with test.py<br />
** patch exists<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=190 190] -- Reminder: NS_LOG_APPEND_CONTEXT<br />
** remove from tracker?<br />
<br />
<br />
'''For Josh Pelkey -- me'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582] -- tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1039 1039] -- TCP Nagle algorithm and RTO calculation<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=730 730] -- Enabling fragmentation at run-time breaks simulation<br />
<br />
= Feature requests =<br />
<br />
* make spectrum model compatible with ns-3 WiFi<br />
* ns-2 packet UID feature <br />
* API for TOS bytes (issue 897) ''may wait for netfilter support''<br />
* Chord/DHT (authors are planning to revise code based on comments)<br />
* app store and build system refactoring<br />
* fragmentation for IPv4: being worked by Vedran Miletic<br />
* TDMA wireless model<br />
** being worked on by Hemanth Narra<br />
<br />
= To Do List =<br />
<br />
Mitch Watrous is working on the following modular build-related items:<br />
<br />
# Update documentation: fix module paths.<br />
# Update documentation: how to use the configuration (.ns3rc) file<br />
<br />
[http://mailman.isi.edu/pipermail/ns-developers/2011-April/008797.html Documentation updates] are planned:<br />
<br />
# Update project documentation for modular build changes (Mitch Watrous)<br />
# Split existing manual to a "developers manual" and a "model library" manual<br />
# Remove "testing" document; move pieces to developers manual and model library manual<br />
# Add python page to manual<br />
# Remove [http://mailman.isi.edu/pipermail/ns-developers/2011-March/008742.html duplicate doxygen documentation]<br />
# Create module template that can be easily cloned<br />
# Create datasheet for ns-3<br />
# Create cheatsheet(s) for ns-3<br />
# Write down duties of release manager somewhere<br />
<br />
[http://mailman.isi.edu/pipermail/ns-developers/2011-April/008816.html Coding style update] is being considered.</div>Watroushttps://www.nsnam.org/mediawiki/index.php?title=Ns-3.11&diff=5602Ns-3.112011-04-20T18:42:48Z<p>Watrous: /* To Do List */</p>
<hr />
<div>{{TOC}}<br />
This page summarizes the ongoing release planning for ns-3.11. Josh Pelkey <jpelkey@gatech.edu> will manage the release.<br />
<br />
= Proposed Release Schedule =<br />
<br />
# January 5 -- ns-3.10 posted<br />
# January 5 -- ns-3.11 '''''Open phase begins'''''<br />
# March 18 -- Deadline for new feature merge<br />
# March 18 -- Begin the phase of small feature development and bug fixing<br />
# April 6 -- Small feature development and bug fixing ends<br />
# April 6 -- '''''Open phase ends'''''<br />
# April 6 -- '''''Maintenance phase begins'''''<br />
# April 30 -- '''''Maintenance phase ends'''''<br />
# April 30 -- '''''Code freeze phase begins'''''<br />
# April 30 -- ns-3.11-RC1<br />
# May 3 -- ns-3.11-RC2<br />
# May 6 -- ns-3.11-RC3<br />
# May 8 -- ns-3.11 posted<br />
# May 8 -- '''''Code freeze phase ends'''''<br />
# May 8 -- ns-3.12 Open phase begins<br />
<br />
As described in the [[Roadmap]] there are three broad sections in the release schedule. During the open phase, people wanting to include a new feature in ns-3.11 should contact Josh and arrange to have their features merged into ns-3-dev. You will be expected to provide the following:<br />
<br />
* A mercurial patch, bundle or repo against the current version of ns-3-dev that contains your proposed feature addition. You need to make sure that we can apply this patch and build and run (debug and optimized as appropriate) all unit and regression tests successfully on all of our target machines<br />
* A summary of the additions you are proposing and an explanation of any changes to existing code that had to be done in order to support your feature (this will be used to generate release notes and will be provided to maintainers if a code review is indicated)<br />
* Some kind of unit or system test that can be used to determine if your feature is actually working at each stage of the integration<br />
<br />
One of us will take a quick look at your proposed addition and determine if a code review is required. According to the ''book of instructions'' a code review requiring positive acknowledgment by maintainers is indicated if:<br />
<br />
* Your proposed feature does not work with all models or on all platforms<br />
* Your feature changes pre-existing APIs<br />
* Your feature crosses maintainer boundaries<br />
<br />
Just to be safe, we will probably run a feature submission by at least one maintainer according to the general area of applicability of the feature. For example, if you submit an entirely new device driver model, as a courtesy we will run this submission by the maintainers of the current devices. The maintainers won't have any responsibility to positively ack the submission, but we will take some time to allow a reasonable review.<br />
<br />
The open phase is divided into two parts: new feature merge period and small feature development and bug fixing. <br />
<br />
During the new feature merge period we can:<br />
* Review the code that might be merged<br />
* Clean up the bug tracker: solve as much bugs as possible<br />
* Merge the new features that has +1 and from maintainers<br />
<br />
After February 14th, the new feature merge period ends and small feature development and bug fixes begins. During the latter, no more merges are accepted and we can:<br />
* Accept limited, small, self contained changes/features to ns-3-dev and to merged new features. As specified before, no more merges are accepted<br />
* Review the code to be merged for next releases<br />
* Cleanup the ns-3-dev bug tracker, solve as much bugs as possible<br />
<br />
The end of the small feature development and bug fixes coincides (March 5th) with the beginning of the maintenance phase. No new features may be added, but the maintainers may check in fixes to bugs; and people with new features that have been accepted and previously merged may fix bugs in existing features. Please don't try to sneak in more new features or you may have your whole feature set removed at the release manager's discretion. You can ask if you want to add small, self-contained features, but there are no guarantees that we will okay them.<br />
<br />
On March 30th, 2011, we are going to enter the code freeze phase. This indicates that we are in the final stages of the release and our primary goal is stability. During the code freeze phase, only P1 bugfixes will be allowed to be checked in. Our ''goal'' will be to reduce the number of P1 bugs to zero before the release of ns-3.11.<br />
<br />
'''''We will reserve the right to veto (and remove) any new feature addition if it begins to cause problems and looks like it threatens the stability of the release at any time in the release process.'''''<br />
<br />
= Candidate Contributions for Inclusion in ns-3.11 =<br />
<br />
<br />
* [http://codereview.appspot.com/2104052/show Virtual Access Point (VAP) for WiFi]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2135046/show GetRelativeVelocity() for MobilityModel]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2173042/show PhySimWifi]<br />
** insufficient reviews<br />
* [http://codereview.appspot.com/2334041/show thread-safe implementation of ScheduleWithContext]<br />
* [http://codereview.appspot.com/1587041/show simple wireless models]<br />
* [http://codereview.appspot.com/1055041/show Jamming model]<br />
* [http://codereview.appspot.com/850045/show MPLS]<br />
* [http://mailman.isi.edu/pipermail/ns-developers/2010-February/007419.html TCP Vegas] (blocked by lack of progress on TCP congestion control architecture)<br />
* [http://codereview.appspot.com/3374041/ NetAnim trace hooks for wireless]<br />
<br />
= ns-3.11 Features Merged =<br />
* [http://codereview.appspot.com/3988043/show ns-3 click]<br />
* [http://codereview.appspot.com/4266051/show ns-3 openflow]<br />
<br />
= The ns-3.11 Bug List =<br />
<br />
'''Highest Priority'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1038 1038] -- Time::Get*Seconds () return signed integer while actually returning unsigned.<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1044 1044] -- Seconds (1e-9) creates Time that is not IsPositive ()<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=445 445] -- Class name rename Scalar->Dimensionless<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=976 976] -- wifi-wired-bridging regression test fails because of rounding errors in mobility model<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1042 1042] -- AODV RERR implosion (missing RERR_RATELIMIT)<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1079 1079] -- MPI code doesn't compile<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1095 1095] -- MPI enable configuration failed in Fedroa 14<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1099 1099] -- AODV performance problems<br />
<br />
<br />
'''High Priority'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1033 1033] -- airtime-metric <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=409 409] -- Routing messages can exceed MTU, and fragmentation not supported <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=631 631] -- RealtimeSimulatorImpl not compatible with python bindings <br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=555 555] -- DCF immediate access bug<br />
** patch existing / needs further testing and verification<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=521 521] -- Ipv4 global routing inefficient<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=938 938] -- missing Doxygen <br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1018 1018] -- mobility --> helper --> mobility circular dependency</s><br />
* <s>bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1017 1017] -- node --> internet-stack --> node</s><br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=756 756] -- Build should be configurable to avoid using optional components<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=912 912] -- modeling processing delays<br />
<br />
<br />
'''Possibly easy fixes'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=976 976] -- udp tx buffer is not fixed size<br />
** possibly WONTFIX<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1006 1006] -- UDP socket tx buffer back pressure needed<br />
**somewhat related to bug 141<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=996 996] -- TCP FIN-WAIT-2 bug<br />
** pinged Adrian about this one, likely fixed<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=272 272] --InternetStackHelper::Install does not mention the fact that it aggregates PacketSocketFactory<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1038 1038] -- Time::Get*Seconds () return signed integer while actually returning unsigned<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=798 798] -- In test.py: CRASH: TestSuite ns3-tcp-cwnd CRASH: TestSuite ns3-tcp-interoperability.<br />
** is this still valid?<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=957 957] -- Issue with test.py<br />
** patch exists<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=190 190] -- Reminder: NS_LOG_APPEND_CONTEXT<br />
** remove from tracker?<br />
<br />
<br />
'''For Josh Pelkey -- me'''<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=582 582] -- tags are not serialized and deserialized from Packet::Serialize and Packet::Deserialize<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=1039 1039] -- TCP Nagle algorithm and RTO calculation<br />
* bug [http://www.nsnam.org/bugzilla/show_bug.cgi?id=730 730] -- Enabling fragmentation at run-time breaks simulation<br />
<br />
= Feature requests =<br />
<br />
* make spectrum model compatible with ns-3 WiFi<br />
* ns-2 packet UID feature <br />
* API for TOS bytes (issue 897) ''may wait for netfilter support''<br />
* Chord/DHT (authors are planning to revise code based on comments)<br />
* app store and build system refactoring<br />
* fragmentation for IPv4: being worked by Vedran Miletic<br />
* TDMA wireless model<br />
** being worked on by Hemanth Narra<br />
<br />
= To Do List =<br />
<br />
Mitch Watrous is working on the following modular build-related items:<br />
<br />
# Make python examples not be run if they have unbuilt dependencies. (This requires creation of waf build log file.)<br />
# Make C++ examples with unbuilt dependencies not be run. (This also requires creation of waf build log file.)<br />
# Update documentation: fix module paths.<br />
# Update documentation: how to use the configuration (.ns3rc) file<br />
<br />
[http://mailman.isi.edu/pipermail/ns-developers/2011-April/008797.html Documentation updates] are planned:<br />
<br />
# Update project documentation for modular build changes (Mitch Watrous)<br />
# Split existing manual to a "developers manual" and a "model library" manual<br />
# Remove "testing" document; move pieces to developers manual and model library manual<br />
# Add python page to manual<br />
# Remove [http://mailman.isi.edu/pipermail/ns-developers/2011-March/008742.html duplicate doxygen documentation]<br />
# Create module template that can be easily cloned<br />
# Create datasheet for ns-3<br />
# Create cheatsheet(s) for ns-3<br />
# Write down duties of release manager somewhere<br />
<br />
[http://mailman.isi.edu/pipermail/ns-developers/2011-April/008816.html Coding style update] is being considered.</div>Watrous