Bug 791 - We need to get rid of waf --regression
We need to get rid of waf --regression
Product: ns-3
Classification: Unclassified
Component: regression
All All
: P4 normal
Assigned To: Mitch Watrous
Depends on:
  Show dependency treegraph
Reported: 2010-01-12 19:55 EST by Craig Dowell
Modified: 2010-10-19 13:09 EDT (History)
2 users (show)

See Also:

Bug 791 patch for ns-3-dev (102.83 KB, patch)
2010-10-08 19:46 EDT, Mitch Watrous
Details | Diff
Bug 791 patch for ns-3-allinone (7.99 KB, patch)
2010-10-08 19:47 EDT, Mitch Watrous
Details | Diff
Bug 791 patch for www (678 bytes, patch)
2010-10-08 19:49 EDT, Mitch Watrous
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Craig Dowell 2010-01-12 19:55:31 EST
We are in a state where we are halfway migrated from the old pcap-style regression tests.  We need to finish the job.

It has been suggested that we hang waf regression off the side of test.py, but I think this is a bad idea since test.py is an integrated framework that lets us get test results out in a consistent way.  Just having test.py call out to waf regression will end up with a pile of stuff printed to standard out that will get lost in the nightlies.  It is a bag on the side of the framework.  Blech!

For now, we need to make sure that everyone understands that there is a waf regression that does things that test.py does not.  i.e., if you are going to check in a change, you need to run test.py and waf --regression.

Then we need to make a plan for retiring the tests in waf regression and allocate warm bodies to make new test.py tests happen or it will never get done.
Comment 1 Mitch Watrous 2010-09-09 11:42:27 EDT
E-mail from Tom Henderson:

We want to remove ./waf --regression, which is the old regression
framework.  Some of ./waf --regression will be simply deleted, since it is
already ported over.  Some will

If you want to get started (till I have more time for detailed
instructions), we need a "csma" test suite, with a test case for each of
these tests:

ns-regression:~/hg/sep10/ns-3-allinone/ns-3-dev> ./waf --regression
Waf: Entering directory `/home/tomh/hg/sep10/ns-3-allinone/ns-3-dev/build'
[703/725] regression-test (test-csma-bridge)
[704/725] regression-test (test-csma-broadcast)
[705/725] regression-test (test-csma-multicast)
[706/725] regression-test (test-csma-one-subnet)
PASS test-csma-multicast
[707/725] regression-test (test-csma-packet-socket)
PASS test-csma-packet-socket
[708/725] regression-test (test-csma-ping)
PASS test-csma-ping
[709/725] regression-test (test-csma-raw-ip-socket)
PASS test-csma-broadcast
[710/725] regression-test (test-csma-star)
PASS test-csma-bridge.  

hg clone http://code.nsnam.org/ns-3-allinone
cd ns-3-allinone
cd ns-3-dev
./test.py (confirm that it runs OK).

copy error-model-test-suite.cc to a file csma-test-suite.cc.  Do a regular
expression substitution to change out "ErrorModel" for "Csma" in the class
and method names.  Change "error-model" to "csma" in the constructor. 
change the global variable errorModelTestSuite to csmaTestSuite.  Now, edit
wscript to add it to the build, and see if you get the following output
from ./test.py:

PASS:  TestSuite csma

I will send some more instructions later but the above will help to get
started if you want to look at something today.
Comment 2 Mitch Watrous 2010-09-10 12:27:40 EDT
Right now, regression tests run using waf are implemented in python code at the upper level and compare output from ns-3 against reference test sets.

I understand what you suggested for csma-test-suite.cc and that makes sense for a normal unit test that is self contained and doesn't compare with a reference test set.

It would be nice to be able to reuse the python reference differencing code in regression.py and make it fit into the format of the test suites used in test.py.

Is there C++ reference differencing code that exists already?

If not, it looks to me like we would have to reimplement the reference differencing code in the C++ test suite code that is called by test.py.
Comment 3 Mitch Watrous 2010-09-10 12:28:03 EDT
E-mail from Tom Henderson:

We can talk tomorrow, but I would like to reduce the reference differencing to specifically targeted items that are trying to be validated.  There is a reference differencing ability also in the test.py framework.  For instance, the test routing-aodv-regression in the src/routing/aodv/test directory contains some reference traces.
Comment 4 Mitch Watrous 2010-09-10 12:36:00 EDT

1. Make the regression tests initially all be C++ TestCase's that just check to see if they ran properly.  Start with csma-bridge first.

2. Tom will determine which tests need to be made into full fledged tests that check their results to see that they have worked properly.

3. I will convert those those tests selected by Tom into full fledged test cases.
Comment 5 Tom Henderson 2010-09-10 15:30:02 EDT
There are presently 22 tests in waf --regression.  I propose that the following be done with them.  For each, the aim will be to test that the relevant functionality works (e.g. multicast) in a way that avoids using trace file diff if possible.

TestSuite csma of type BVT will pick up these test cases:
1) TestCase CsmaBridge(test-csma-bridge)
2) TestCase CsmaBroadcast (test-csma-broadcast)
3) TestCase CsmaMulticast (test-csma-multicast)
4) TestCase CsmaOneSubnet (test-csma-one-subnet)
5) TestCase CsmaPacketSocket (test-csma-packet-socket)
6) TestCase CsmaPing (test-csma-ping)
7) TestCase CsmaBroadcast (test-csma-broadcast)
8) TestCase CsmaRawIpSocket (test-csma-raw-ip-socket)
9) TestCase CsmaStar (test-csma-star)

TestSuite global-routing of type BVT will pick up these test cases:
1) TestCase SimpleGlobalRouting (test-simple-global-routing)
2) TestCase DynamicGlobalRouting (test-dynamic-global-routing)
3) TestCase GlobalRoutingSlash32 (test-global-routing-slash32)

TestSuite static-routing of type BVT will pick up these test cases:
1) TestCase StaticRoutingSlash32 (test-static-routing-slash32)

TestSuite realtime of type BVT will pick up this test case
1) TestCase RealtimeUdpEcho (test-realtime-udp-echo)

These regression tests are not really testing anything not already covered in the test framework, so I propose that they just continue to be tested as Examples.

Comment 6 Mitch Watrous 2010-10-08 19:46:34 EDT
Created attachment 992 [details]
Bug 791 patch for ns-3-dev

This patch fixes the problem for ns-3-dev.
Comment 7 Mitch Watrous 2010-10-08 19:47:53 EDT
Created attachment 993 [details]
Bug 791 patch for ns-3-allinone

This patch fixes the problem for ns-3-allinone.
Comment 8 Mitch Watrous 2010-10-08 19:49:31 EDT
Created attachment 994 [details]
Bug 791 patch for www

This patch fixes the problem for www.
Comment 9 Mitch Watrous 2010-10-19 13:09:23 EDT
Bug closed.

ns-3-allinone changeset: 75aed625c976

ns-3-dev changesets: a071889af159 and edc862ef7d63

www changeset: dd6ee263c85b