Bugzilla – Bug 1497
eliminate random variation in trace-based tests
Last modified: 2013-12-14 02:09:44 EST
We have several test suites performing binary file comparison against known-good pcap traces. routing-aodv-regression routing-olsr-regression devices-mesh-dot11s-regression devices-mesh-flame-regression ns3-tcp variants These are a real headache to maintain based on the randomness in different models used by the tests. While the seed and run number can be fixed, and selected RandomVariableStreams have been assigned to set stream numbers in these tests, there are still other (as yet undiscovered) random variables in the system that are using automatic stream assignment. To see the effects of this, define a static RandomVariableStream variable somewhere in the system, and run ./test.py, and pcap will no longer match for some of these tests. Ideally, it would be best to remove these pcap tests by testing instead for specific events (IMO), but failing that, I would like to lock down the randomness such that the pcap output cannot be perturbed by other changes in the system. To close this bug, I would like to see either (or both) of the following changes: 1) definition of a static random variable (which will perturb the stream assignment of all non-static instances) in some file does not change the pcap output e.g. in some implementation file compiled into ns-3, temporarily add static Ptr<NormalRandomVariable> testStaticVariable = CreateObject<NormalRandomVariable> (); and make sure all tests still pass 2) set all streams in use in the test so that none of the streams used in the simulation end with the value of "-1"; config-store output may possibly be used to check this.
> 1) definition of a static random variable (which will perturb the stream > assignment of all non-static instances) in some file does not change the pcap > output > > e.g. in some implementation file compiled into ns-3, temporarily add > > static Ptr<NormalRandomVariable> testStaticVariable = > CreateObject<NormalRandomVariable> (); > > and make sure all tests still pass to clarify a bit further, I am not suggesting to add this variable permanently to ns-3-dev, but rather that the tests are fixed such that temporary insertion of this variable does not perturb traces.
This bug is a little harder than it looks because many temporary random variable stream variables are constructed and destructed with stream = -1 and config-store does not show streams associated with attributes.
(In reply to comment #2) > This bug is a little harder than it looks because many temporary random > variable stream variables are constructed and destructed with stream = -1 and > config-store does not show streams associated with attributes. Does config-store save not show the eventual stream assignment? I would expect it to.
Here are a couple random variable streams from the config-store output file: default ns3::RandomRectanglePositionAllocator::X "ns3::UniformRandomVariable[Min=0.0|Max=1.0]" default ns3::RandomRectanglePositionAllocator::Y "ns3::UniformRandomVariable[Min=0.0|Max=1.0]" No stream is shown.
Created attachment 1446 [details] patch to fix aodv This patch fixes AODV; the random variable was being used before SetStream could be called. While this makes the test traces fixed with respect to other random variables in the system, it does not permit the test cases to be shuffled in order, since new Mac48Address will be allocated (a Mac48Address::Reset() method would help in this regard). The mesh tests still need to be fixed, probably along the same lines (check that random variables are not used before their streams are fixed.