A Discrete-Event Network Simulator
API
Loading...
Searching...
No Matches
ns3::ExampleAsTestSuite Class Reference

Execute an example program as a test suite. More...

#include "example-as-test.h"

+ Inheritance diagram for ns3::ExampleAsTestSuite:
+ Collaboration diagram for ns3::ExampleAsTestSuite:

Public Member Functions

 ExampleAsTestSuite (const std::string name, const std::string program, const std::string dataDir, const std::string args="", const TestDuration duration=QUICK, const bool shouldNotErr=true)
 Constructor.
 
- Public Member Functions inherited from ns3::TestSuite
 TestSuite (std::string name, Type type=UNIT)
 Construct a new test suite.
 
TestSuite::Type GetTestType ()
 get the kind of test this test suite implements
 
- Public Member Functions inherited from ns3::TestCase
 TestCase (const TestCase &)=delete
 
virtual ~TestCase ()
 Destructor.
 
std::string GetName () const
 
TestCaseoperator= (const TestCase &)=delete
 

Additional Inherited Members

- Public Types inherited from ns3::TestSuite
enum  Type {
  ALL = 0 , UNIT , SYSTEM , EXAMPLE ,
  PERFORMANCE
}
 Type of test. More...
 
- Public Types inherited from ns3::TestCase
enum  TestDuration { QUICK = 1 , EXTENSIVE = 2 , TAKES_FOREVER = 3 }
 How long the test takes to execute. More...
 
- Protected Member Functions inherited from ns3::TestCase
 TestCase (std::string name)
 Constructor.
 
void AddTestCase (TestCase *testCase, TestDuration duration=QUICK)
 Add an individual child TestCase to this test suite.
 
TestCaseGetParent () const
 Get the parent of this TestCase.
 
bool IsStatusFailure () const
 Check if any tests failed.
 
bool IsStatusSuccess () const
 Check if all tests passed.
 
void SetDataDir (std::string directory)
 Set the data directory where reference trace files can be found.
 
void ReportTestFailure (std::string cond, std::string actual, std::string limit, std::string message, std::string file, int32_t line)
 Log the failure of this TestCase.
 
bool MustAssertOnFailure () const
 Check if this run should assert on failure.
 
bool MustContinueOnFailure () const
 Check if this run should continue on failure.
 
std::string CreateDataDirFilename (std::string filename)
 Construct the full path to a file in the data directory.
 
std::string CreateTempDirFilename (std::string filename)
 Construct the full path to a file in a temporary directory.
 

Detailed Description

Execute an example program as a test suite.

You can use this TestSuite to add an example to the test suite with checking of the example output (std::out and std::err). This is an alternative to adding an example using the examples-to-run.py file. The key difference between the two methods is what criteria is used to for success. Examples added to examples-to-run.py will be run and the exit status checked (non-zero indicates failure). ExampleAsTestSuite adds checking of output against a specified known "good" reference file.

Warning
If you are thinking about using this class, strongly consider using a standard test instead. The TestSuite class has better checking using the NS_TEST_* macros and in almost all cases is the better approach. If your test can be done with a TestSuite class you will be asked by the reviewers to rewrite the test when you do a pull request.
Test Addition

To use an example program as a test you need to create a test suite file and add it to the appropriate list in your module CMakeLists.txt file. The "good" output reference file needs to be generated for detecting regressions.

Let's assume your module is called mymodule, and the example program is mymodule/examples/mod-example.cc. First you should create a test file mymodule/test/mymodule-examples-test-suite.cc which looks like this:

#include "ns3/example-as-test.h"
static ns3::ExampleAsTestSuite g_modExampleOne("mymodule-example-mod-example-one",
"mod-example", NS_TEST_SOURCEDIR, "--arg-one");
static ns3::ExampleAsTestSuite g_modExampleTwo("mymodule-example-mod-example-two",
"mod-example", NS_TEST_SOURCEDIR, "--arg-two");
Execute an example program as a test suite.

The arguments to the constructor are the name of the test suite, the example to run, the directory that contains the "good" reference file (the macro NS_TEST_SOURCEDIR is normally the correct directory), and command line arguments for the example. In the preceding code the same example is run twice with different arguments.

You then need to add that newly created test suite file to the list of test sources in mymodule/CMakeLists.txt. Building of examples is an option so you need to guard the inclusion of the test suite:

if (bld.env['ENABLE_EXAMPLES']):
module.source.append('model/mymodule-examples-test-suite.cc')

Since you modified a CMakeLists.txt file you need to reconfigure and rebuild everything.

You just added new tests so you will need to generate the "good" output reference files that will be used to verify the example:

./test.py --suite="mymodule-example-*" --update

This will run all tests starting with "mymodule-example-" and save new "good" reference files. Updating the reference file should be done when you create the test and whenever output changes. When updating the reference output you should inspect it to ensure that it is valid. The reference files should be committed with the new test.

Test Verification

You can run the test with the standard test.py script. For example to run the suites you just added:

./test.py --suite="mymodule-example-*"

This will run all mymodule-example-... tests and report whether they produce output matching the reference files.

Writing good examples for testing

When setting up an example for use by this class you should be very careful about what output the example generates. For example, writing output which includes simulation time (especially high resolution time) makes the test sensitive to potentially minor changes in event times. This makes the reference output hard to verify and hard to keep up-to-date. Output as little as needed for the example and include only behavioral state that is important for determining if the example has run correctly.

Definition at line 209 of file example-as-test.h.

Constructor & Destructor Documentation

◆ ExampleAsTestSuite()

ns3::ExampleAsTestSuite::ExampleAsTestSuite ( const std::string  name,
const std::string  program,
const std::string  dataDir,
const std::string  args = "",
const TestDuration  duration = QUICK,
const bool  shouldNotErr = true 
)

Constructor.

Parameters
[in]nameThe test case name, typically the program name and summary of the arguments, such as my-example-foo
[in]programThe actual example program names, such as my-example
[in]dataDirThe location of the reference file. This is normally provided by the symbol NS_TEST_SOURCEDIR in the module-examples-test-suite.cc file. The reference file should be named after the test case name, for example my-example-foo.log. If you use the --update argument to test.py or test-runner the reference file will be created with the correct name.
[in]argsAny additional arguments to the program.
[in]shouldNotErrWhether an error return status should be considered a test failure. This is useful when testing error detection which might return a non-zero status. The output (on std::cout and std::cerr) will be compared to the reference logs as normal.
[in]durationAmount of time this test takes to execute (defaults to QUICK).

The documentation for this class was generated from the following file: