A Discrete-Event Network Simulator
API
Loading...
Searching...
No Matches
example-as-test.h
Go to the documentation of this file.
1/*
2 * Copyright (c) 2020 Lawrence Livermore National Laboratory
3 *
4 * SPDX-License-Identifier: GPL-2.0-only
5 *
6 * Author: Peter D. Barnes, Jr. <pdbarnes@llnl.gov>
7 */
8
9#ifndef NS3_EXAMPLE_AS_TEST_SUITE_H
10#define NS3_EXAMPLE_AS_TEST_SUITE_H
11
12#include "test.h"
13
14#include <string>
15
16/**
17 * @file
18 * @ingroup testing
19 * Enable examples to be run as meaningful tests.
20 * Declaration of classes ns3::ExampleAsTestSuite and ns3::ExampleAsTestCase.
21 */
22
23namespace ns3
24{
25
26/**
27 * @ingroup testing
28 * Execute an example program as a test, by comparing the output
29 * to a reference file.
30 *
31 * User can subclass and override the GetCommandTemplate() and
32 * GetPostProcessingCommand() methods if more complex
33 * example invocation patterns are required.
34 *
35 * @see examples-as-tests-test-suite.cc
36 */
38{
39 public:
40 /**
41 * Constructor.
42 * @param [in] name The test case name, typically the program name
43 * and summary of the arguments, such as `my-example-foo`
44 * @param [in] program The actual example program names, such as `my-example`
45 * @param [in] dataDir The location of the reference file.
46 * This is normally provided by the symbol
47 * `NS_TEST_SOURCEDIR` in the `module-examples-test-suite.cc`
48 * file.
49 * The reference file should be named after
50 * the test case name,
51 * for example `my-example-foo.log`. If you use
52 * the `--update` argument to `test.py` or
53 * `test-runner` the reference file will be created
54 * with the correct name.
55 * @param [in] args Any additional arguments to the program.
56 * @param [in] shouldNotErr Whether an error return status should be
57 * considered a test failure. This is useful when testing
58 * error detection which might return a non-zero status.
59 * The output (on `std::cout` and `std::cerr`) will
60 * be compared to the reference logs as normal.
61 */
62 ExampleAsTestCase(const std::string name,
63 const std::string program,
64 const std::string dataDir,
65 const std::string args = "",
66 const bool shouldNotErr = true);
67
68 /** Destructor. */
69 ~ExampleAsTestCase() override;
70
71 /**
72 * Customization point for more complicated patterns
73 * to invoke the example program.
74 *
75 * @returns The string to be given to the `ns3 --command-template=` argument.
76 */
77 virtual std::string GetCommandTemplate() const;
78
79 /**
80 * Customization point for tests requiring post-processing of stdout.
81 *
82 * For example to sort return `"| sort"`
83 *
84 * One common case is to mask memory addresses, which can change
85 * when things are built on different platforms, recompiled locally,
86 * or even from run to run. A simple post-processing filter could be
87 *
88 * `"| sed -E 's/0x[0-9a-fA-F]{8,}/0x-address/g'"`
89 *
90 * Default is `""`, no additional processing.
91 *
92 * @returns The string of post-processing commands
93 */
94 virtual std::string GetPostProcessingCommand() const;
95
96 // Inherited
97 void DoRun() override;
98
99 protected:
100 std::string m_program; /**< The program to run. */
101 std::string m_dataDir; /**< The source directory for the test. */
102 std::string m_args; /**< Any additional arguments to the program. */
103 bool m_shouldNotErr; /**< Whether error return status is a test failure. */
104
105 // end of class ExampleAsTestCase
106};
107
108/**
109 * @ingroup testing
110 * Execute an example program as a test suite.
111 *
112 * You can use this TestSuite to add an example to the test suite with
113 * checking of the example output (`std::out` and `std::err`). This
114 * is an alternative to adding an example using the
115 * `examples-to-run.py` file. The key difference between the two
116 * methods is what criteria is used to for success. Examples added to
117 * `examples-to-run.py` will be run and the exit status checked
118 * (non-zero indicates failure). ExampleAsTestSuite adds checking of
119 * output against a specified known "good" reference file.
120 *
121 * @warning If you are thinking about using this class, strongly
122 * consider using a standard test instead. The TestSuite class has
123 * better checking using the NS_TEST_* macros and in almost all cases
124 * is the better approach. If your test can be done with a TestSuite
125 * class you will be asked by the reviewers to rewrite the test when
126 * you do a pull request.
127 *
128 * @par Test Addition
129 *
130 * To use an example program as a test you need to create a test suite
131 * file and add it to the appropriate list in your module CMakeLists.txt
132 * file. The "good" output reference file needs to be generated for
133 * detecting regressions.
134 *
135 * Let's assume your module is called `mymodule`, and the example
136 * program is `mymodule/examples/mod-example.cc`. First you should
137 * create a test file `mymodule/test/mymodule-examples-test-suite.cc`
138 * which looks like this:
139 *
140 * \code{.cpp}
141 * #include "ns3/example-as-test.h"
142 * static ns3::ExampleAsTestSuite g_modExampleOne("mymodule-example-mod-example-one",
143 * "mod-example", NS_TEST_SOURCEDIR, "--arg-one");
144 * static ns3::ExampleAsTestSuite g_modExampleTwo("mymodule-example-mod-example-two",
145 * "mod-example", NS_TEST_SOURCEDIR, "--arg-two"); \endcode
146 *
147 * The arguments to the constructor are the name of the test suite, the
148 * example to run, the directory that contains the "good" reference file
149 * (the macro `NS_TEST_SOURCEDIR` is normally the correct directory),
150 * and command line arguments for the example. In the preceding code
151 * the same example is run twice with different arguments.
152 *
153 * You then need to add that newly created test suite file to the list
154 * of test sources in `mymodule/CMakeLists.txt`. Building of examples
155 * is an option so you need to guard the inclusion of the test suite:
156 *
157 * \code{.py}
158 * if (bld.env['ENABLE_EXAMPLES']):
159 * module.source.append('model/mymodule-examples-test-suite.cc')
160 * @endcode
161 *
162 * Since you modified a CMakeLists.txt file you need to reconfigure and
163 * rebuild everything.
164 *
165 * You just added new tests so you will need to generate the "good"
166 * output reference files that will be used to verify the example:
167 *
168 * `./test.py --suite="mymodule-example-*" --update`
169 *
170 * This will run all tests starting with "mymodule-example-" and save
171 * new "good" reference files. Updating the reference file should be
172 * done when you create the test and whenever output changes. When
173 * updating the reference output you should inspect it to ensure that
174 * it is valid. The reference files should be committed with the new
175 * test.
176 *
177 * @par Test Verification
178 *
179 * You can run the test with the standard `test.py` script. For
180 * example to run the suites you just added:
181 *
182 * `./test.py --suite="mymodule-example-*"`
183 *
184 * This will run all `mymodule-example-...` tests and report whether they
185 * produce output matching the reference files.
186 *
187 * @par Writing good examples for testing
188 *
189 * When setting up an example for use by this class you should be very
190 * careful about what output the example generates. For example,
191 * writing output which includes simulation time (especially high
192 * resolution time) makes the test sensitive to potentially minor
193 * changes in event times. This makes the reference output hard to
194 * verify and hard to keep up-to-date. Output as little as needed for
195 * the example and include only behavioral state that is important for
196 * determining if the example has run correctly.
197 *
198 */
200{
201 public:
202 /**
203 * @copydoc ExampleAsTestCase::ExampleAsTestCase
204 * @param [in] duration Amount of time this test takes to execute
205 * (defaults to QUICK).
206 */
207 ExampleAsTestSuite(const std::string name,
208 const std::string program,
209 const std::string dataDir,
210 const std::string args = "",
211 const Duration duration = Duration::QUICK,
212 const bool shouldNotErr = true);
213 // end of class ExampleAsTestSuite
214};
215
216} // namespace ns3
217
218#endif /* NS3_EXAMPLE_TEST_SUITE_H */
Execute an example program as a test, by comparing the output to a reference file.
virtual std::string GetCommandTemplate() const
Customization point for more complicated patterns to invoke the example program.
bool m_shouldNotErr
Whether error return status is a test failure.
virtual std::string GetPostProcessingCommand() const
Customization point for tests requiring post-processing of stdout.
void DoRun() override
Implementation to actually run this TestCase.
ExampleAsTestCase(const std::string name, const std::string program, const std::string dataDir, const std::string args="", const bool shouldNotErr=true)
Constructor.
std::string m_args
Any additional arguments to the program.
std::string m_dataDir
The source directory for the test.
std::string m_program
The program to run.
~ExampleAsTestCase() override
Destructor.
Execute an example program as a test suite.
ExampleAsTestSuite(const std::string name, const std::string program, const std::string dataDir, const std::string args="", const Duration duration=Duration::QUICK, const bool shouldNotErr=true)
Constructor.
encapsulates test code
Definition test.h:1050
Duration
How long the test takes to execute.
Definition test.h:1054
A suite of tests to run.
Definition test.h:1267
Every class exported by the ns3 library is enclosed in the ns3 namespace.
ns3::TestCase, ns3::TestSuite, ns3::TestRunner declarations, and NS_TEST_ASSERT macro definitions.