Execute an example program as a test suite. More...
#include "example-as-test.h"
Public Member Functions | |
ExampleAsTestSuite (const std::string name, const std::string program, const std::string dataDir, const std::string args="", const TestDuration duration=QUICK, const bool shouldNotErr=true) | |
Constructor. | |
Public Member Functions inherited from ns3::TestSuite | |
TestSuite (std::string name, Type type=UNIT) | |
Construct a new test suite. | |
TestSuite::Type | GetTestType () |
get the kind of test this test suite implements | |
Public Member Functions inherited from ns3::TestCase | |
TestCase (const TestCase &)=delete | |
virtual | ~TestCase () |
Destructor. | |
std::string | GetName () const |
TestCase & | operator= (const TestCase &)=delete |
Additional Inherited Members | |
Public Types inherited from ns3::TestSuite | |
enum | Type { ALL = 0 , UNIT , SYSTEM , EXAMPLE , PERFORMANCE } |
Type of test. More... | |
Public Types inherited from ns3::TestCase | |
enum | TestDuration { QUICK = 1 , EXTENSIVE = 2 , TAKES_FOREVER = 3 } |
How long the test takes to execute. More... | |
Protected Member Functions inherited from ns3::TestCase | |
TestCase (std::string name) | |
Constructor. | |
void | AddTestCase (TestCase *testCase, TestDuration duration=QUICK) |
Add an individual child TestCase to this test suite. | |
TestCase * | GetParent () const |
Get the parent of this TestCase. | |
bool | IsStatusFailure () const |
Check if any tests failed. | |
bool | IsStatusSuccess () const |
Check if all tests passed. | |
void | SetDataDir (std::string directory) |
Set the data directory where reference trace files can be found. | |
void | ReportTestFailure (std::string cond, std::string actual, std::string limit, std::string message, std::string file, int32_t line) |
Log the failure of this TestCase. | |
bool | MustAssertOnFailure () const |
Check if this run should assert on failure. | |
bool | MustContinueOnFailure () const |
Check if this run should continue on failure. | |
std::string | CreateDataDirFilename (std::string filename) |
Construct the full path to a file in the data directory. | |
std::string | CreateTempDirFilename (std::string filename) |
Construct the full path to a file in a temporary directory. | |
Execute an example program as a test suite.
You can use this TestSuite to add an example to the test suite with checking of the example output (std::out
and std::err
). This is an alternative to adding an example using the examples-to-run.py
file. The key difference between the two methods is what criteria is used to for success. Examples added to examples-to-run.py
will be run and the exit status checked (non-zero indicates failure). ExampleAsTestSuite adds checking of output against a specified known "good" reference file.
To use an example program as a test you need to create a test suite file and add it to the appropriate list in your module CMakeLists.txt file. The "good" output reference file needs to be generated for detecting regressions.
Let's assume your module is called mymodule
, and the example program is mymodule/examples/mod-example.cc
. First you should create a test file mymodule/test/mymodule-examples-test-suite.cc
which looks like this:
The arguments to the constructor are the name of the test suite, the example to run, the directory that contains the "good" reference file (the macro NS_TEST_SOURCEDIR
is normally the correct directory), and command line arguments for the example. In the preceding code the same example is run twice with different arguments.
You then need to add that newly created test suite file to the list of test sources in mymodule/CMakeLists.txt
. Building of examples is an option so you need to guard the inclusion of the test suite:
Since you modified a CMakeLists.txt file you need to reconfigure and rebuild everything.
You just added new tests so you will need to generate the "good" output reference files that will be used to verify the example:
./test.py --suite="mymodule-example-*" --update
This will run all tests starting with "mymodule-example-" and save new "good" reference files. Updating the reference file should be done when you create the test and whenever output changes. When updating the reference output you should inspect it to ensure that it is valid. The reference files should be committed with the new test.
You can run the test with the standard test.py
script. For example to run the suites you just added:
./test.py --suite="mymodule-example-*"
This will run all mymodule-example-...
tests and report whether they produce output matching the reference files.
When setting up an example for use by this class you should be very careful about what output the example generates. For example, writing output which includes simulation time (especially high resolution time) makes the test sensitive to potentially minor changes in event times. This makes the reference output hard to verify and hard to keep up-to-date. Output as little as needed for the example and include only behavioral state that is important for determining if the example has run correctly.
Definition at line 209 of file example-as-test.h.
ns3::ExampleAsTestSuite::ExampleAsTestSuite | ( | const std::string | name, |
const std::string | program, | ||
const std::string | dataDir, | ||
const std::string | args = "" , |
||
const TestDuration | duration = QUICK , |
||
const bool | shouldNotErr = true |
||
) |
Constructor.
[in] | name | The test case name, typically the program name and summary of the arguments, such as my-example-foo |
[in] | program | The actual example program names, such as my-example |
[in] | dataDir | The location of the reference file. This is normally provided by the symbol NS_TEST_SOURCEDIR in the module-examples-test-suite.cc file. The reference file should be named after the test case name, for example my-example-foo.log . If you use the --update argument to test.py or test-runner the reference file will be created with the correct name. |
[in] | args | Any additional arguments to the program. |
[in] | shouldNotErr | Whether an error return status should be considered a test failure. This is useful when testing error detection which might return a non-zero status. The output (on std::cout and std::cerr ) will be compared to the reference logs as normal. |
[in] | duration | Amount of time this test takes to execute (defaults to QUICK). |