Bugzilla – Bug 999
Some test suites fail when NSC has not been built.
Last modified: 2010-10-08 13:59:07 EDT
The following 3 test suites fail when NSC has not been built: TestSuite nsc-tcp-loss TestSuite ns3-tcp-cwnd TestSuite ns3-tcp-interoperability Sam's comments from Bug 980 are: This seems to be because the tests are not explicitly dependent on the variable ENABLE_NSC in the test.py script. It's not immediately evident how they should be configured there. Somehow we need to make sure these tests are not run when ENABLE_NSC=False; this already works for e.g. tcp/tcp-nsc-lfn but the code there seems specific to tests in the examples/ directory.
A blunt-instrument way to deal with this issue is to take the interesting_config_items in test.py and set corresponding environment variables before starting to execute the test-runners. The test cases could look at the environment variables (e.g. ENABLE_NSC) and report back SKIPPED if they can't run. I think we should report that a test case has been skipped rather than silently not executing it, BTW.
Created attachment 985 [details] Patch for bug 999.
I have attached a patch that fixes this bug. There were 3 problems that this patch fixes: (1.) When the NSC library was missing, the NSC_ENABLED flag wasn't being set to false in src/internet-stack/wscript. (2.) test.py was using the wrong flag (ENABLE_NSC) and wasn't skipping NSC dependent tests when NSC was missing. (3.) src/test/ns3tcp/wscript was not building ns3tcp-socket-test-suite when NSC was missing.
(In reply to comment #3) > I have attached a patch that fixes this bug. > > There were 3 problems that this patch fixes: > > (1.) When the NSC library was missing, the NSC_ENABLED flag wasn't being set to > false in src/internet-stack/wscript. > > (2.) test.py was using the wrong flag (ENABLE_NSC) and wasn't skipping NSC > dependent tests when NSC was missing. > > (3.) src/test/ns3tcp/wscript was not building ns3tcp-socket-test-suite when NSC > was missing. I tested this patch in three ways: 1) normal (with nsc present), valgrind (test.py -g) and no valgrind 2) NSC directory totally absent 3) NSC directory present, but NSC not built successfully I found that I needed to change the third argument below from True to False so that valgrind would skip these examples in case 1) - ("tcp/tcp-nsc-lfn", "ENABLE_NSC == True", "True"), - ("tcp/tcp-nsc-zoo", "ENABLE_NSC == True", "True"), + ("tcp/tcp-nsc-lfn", "NSC_ENABLED == True", "False"), + ("tcp/tcp-nsc-zoo", "NSC_ENABLED == True", "False"), In cases 2 and 3, the NSC examples and tests are silently skipped but otherwise the patch works as expected.
(In reply to comment #1) > A blunt-instrument way to deal with this issue is to take the > interesting_config_items in test.py and set corresponding environment variables > before starting to execute the test-runners. The test cases could look at the > environment variables (e.g. ENABLE_NSC) and report back SKIPPED if they can't > run. > > I think we should report that a test case has been skipped rather than silently > not executing it, BTW. If NSC is not enabled, the NSC test suites will not be compiled and test-runner will not know about them; hence, they are silently skipped. I'm not sure whether you are recommending that they are somehow reported as skipped. The NSC examples are also silently skipped because the second argument here evaluates to False: ("tcp/tcp-nsc-lfn", "NSC_ENABLED == True", "False"), ("tcp/tcp-nsc-zoo", "NSC_ENABLED == True", "False"), If we want to make the examples show as skipped when NSC is not built, it seems like it probably could be most easily done by pulling them out of the list of examples such as the list of python_tests is done (i.e. move those tests to nsc_tests, and add a test such as "if not NSC_ENABLED: job.set_is_skipped(True)") If NSC is not built or present, do people want these examples to still show up as skipped? SKIP: Example tcp/tcp-nsc-lfn SKIP: Example tcp/tcp-nsc-zoo More broadly, I think we should decide what are the criteria for reporting as SKIP, because this question will come up when we modularize ns-3. I might lean towards reporting SKIP for cases in which the test or example has been already built as an executable but cannot be run under the test configuration (e.g. valgrind), but if the user explicitly disables a feature or module at configuration time, or the required module cannot be found, from the perspective of test-runner, those tests (or examples) do not exist and could be silently skipped.
Created attachment 989 [details] Second version of patch for bug 999. I have attached a second version of the patch that fixes the problem with case 1 that was described in comment #4.
Bug closed. ns-3-dev changeset: c6e03f378655