Bugzilla – Bug 791
We need to get rid of waf --regression
Last modified: 2010-10-19 13:09:23 EDT
We are in a state where we are halfway migrated from the old pcap-style regression tests. We need to finish the job. It has been suggested that we hang waf regression off the side of test.py, but I think this is a bad idea since test.py is an integrated framework that lets us get test results out in a consistent way. Just having test.py call out to waf regression will end up with a pile of stuff printed to standard out that will get lost in the nightlies. It is a bag on the side of the framework. Blech! For now, we need to make sure that everyone understands that there is a waf regression that does things that test.py does not. i.e., if you are going to check in a change, you need to run test.py and waf --regression. Then we need to make a plan for retiring the tests in waf regression and allocate warm bodies to make new test.py tests happen or it will never get done.
E-mail from Tom Henderson: We want to remove ./waf --regression, which is the old regression framework. Some of ./waf --regression will be simply deleted, since it is already ported over. Some will If you want to get started (till I have more time for detailed instructions), we need a "csma" test suite, with a test case for each of these tests: ns-regression:~/hg/sep10/ns-3-allinone/ns-3-dev> ./waf --regression Waf: Entering directory `/home/tomh/hg/sep10/ns-3-allinone/ns-3-dev/build' [703/725] regression-test (test-csma-bridge) [704/725] regression-test (test-csma-broadcast) [705/725] regression-test (test-csma-multicast) [706/725] regression-test (test-csma-one-subnet) PASS test-csma-multicast [707/725] regression-test (test-csma-packet-socket) PASS test-csma-packet-socket [708/725] regression-test (test-csma-ping) PASS test-csma-ping [709/725] regression-test (test-csma-raw-ip-socket) PASS test-csma-broadcast [710/725] regression-test (test-csma-star) PASS test-csma-bridge. So, hg clone http://code.nsnam.org/ns-3-allinone cd ns-3-allinone ./download.py ./build.py cd ns-3-dev ./test.py (confirm that it runs OK). copy error-model-test-suite.cc to a file csma-test-suite.cc. Do a regular expression substitution to change out "ErrorModel" for "Csma" in the class and method names. Change "error-model" to "csma" in the constructor. change the global variable errorModelTestSuite to csmaTestSuite. Now, edit wscript to add it to the build, and see if you get the following output from ./test.py: PASS: TestSuite csma I will send some more instructions later but the above will help to get started if you want to look at something today.
Right now, regression tests run using waf are implemented in python code at the upper level and compare output from ns-3 against reference test sets. I understand what you suggested for csma-test-suite.cc and that makes sense for a normal unit test that is self contained and doesn't compare with a reference test set. It would be nice to be able to reuse the python reference differencing code in regression.py and make it fit into the format of the test suites used in test.py. Is there C++ reference differencing code that exists already? If not, it looks to me like we would have to reimplement the reference differencing code in the C++ test suite code that is called by test.py.
E-mail from Tom Henderson: We can talk tomorrow, but I would like to reduce the reference differencing to specifically targeted items that are trying to be validated. There is a reference differencing ability also in the test.py framework. For instance, the test routing-aodv-regression in the src/routing/aodv/test directory contains some reference traces.
Plan: 1. Make the regression tests initially all be C++ TestCase's that just check to see if they ran properly. Start with csma-bridge first. 2. Tom will determine which tests need to be made into full fledged tests that check their results to see that they have worked properly. 3. I will convert those those tests selected by Tom into full fledged test cases.
There are presently 22 tests in waf --regression. I propose that the following be done with them. For each, the aim will be to test that the relevant functionality works (e.g. multicast) in a way that avoids using trace file diff if possible. TestSuite csma of type BVT will pick up these test cases: 1) TestCase CsmaBridge(test-csma-bridge) 2) TestCase CsmaBroadcast (test-csma-broadcast) 3) TestCase CsmaMulticast (test-csma-multicast) 4) TestCase CsmaOneSubnet (test-csma-one-subnet) 5) TestCase CsmaPacketSocket (test-csma-packet-socket) 6) TestCase CsmaPing (test-csma-ping) 7) TestCase CsmaBroadcast (test-csma-broadcast) 8) TestCase CsmaRawIpSocket (test-csma-raw-ip-socket) 9) TestCase CsmaStar (test-csma-star) TestSuite global-routing of type BVT will pick up these test cases: 1) TestCase SimpleGlobalRouting (test-simple-global-routing) 2) TestCase DynamicGlobalRouting (test-dynamic-global-routing) 3) TestCase GlobalRoutingSlash32 (test-global-routing-slash32) TestSuite static-routing of type BVT will pick up these test cases: 1) TestCase StaticRoutingSlash32 (test-static-routing-slash32) TestSuite realtime of type BVT will pick up this test case 1) TestCase RealtimeUdpEcho (test-realtime-udp-echo) These regression tests are not really testing anything not already covered in the test framework, so I propose that they just continue to be tested as Examples. test-object-names test-second test-simple-error-model test-tcp-large-transfer test-tcp-nsc-lfn test-third test-udp-echo test-wifi-wired-bridging
Created attachment 992 [details] Bug 791 patch for ns-3-dev This patch fixes the problem for ns-3-dev.
Created attachment 993 [details] Bug 791 patch for ns-3-allinone This patch fixes the problem for ns-3-allinone.
Created attachment 994 [details] Bug 791 patch for www This patch fixes the problem for www.
Bug closed. ns-3-allinone changeset: 75aed625c976 ns-3-dev changesets: a071889af159 and edc862ef7d63 www changeset: dd6ee263c85b