[Capturing the thread]
On May 31, 2015, at 5:53 AM, Tommaso Pecorella <tpecorella@mac.com> wrote:
Hi all, 

I'm re-doing the Energy model tests, and one thing just came to my mind. 

We use examples as tests, and that’s ok, but we actually don’t validate them. I mean, the examples are run, but they are “failing” only if they crash. 

In order to check if an example failed without a crash, we could add an assert at the end, checking for some meaningful thing (e.g., if there was data transmission). 
However, this will cause major issues. I’m 110% sure that some dumb user will modify the example and NOT the assert, and will come up asking “why”. Now, we have already plenty of things that users can misuse and ask “why” (the usual answer is RTFM), we don’t need one more. 

Question: would it be possible to have an assert-like macro that works only if the example is called by test.py ? 
That would solve both problems. We could use examples in a more consistent way, and wouldn’t be misused by the usual idiot. 

Cheers, 

T. 

PS: the root cause is that one of the energy tests could be easily converted into an example, and it would be a nice example too. 
PPS: this could solve the “tests using strange dependencies” issue. Make an example out of it, and live happy.

[Capturing the thread]
On May 31, 2015, at 11:14 AM, Tom Henderson <tomh@tomh.org> wrote:

I have previously given thought to consistently adding a '--test' command line option to many of our examples so that some state could be checked by the test suite.  This might protect against the case you describe; if the user who clones and modifies the examples but does not invoke the '--test' option, he or she won't see a program error.

- Tom

[Capturing the thread]
On May 31, 2015, at 3:55 PM, Tommaso Pecorella <tpecorella@mac.com> wrote:
the —test option could be a good idea, I’ll try it.

The only problem is: what to do if the module fails ? An assert would mean a crash for test.py, and that’s not really informative. Perhaps we can use the TEST assert set… I’ll have to try, perhaps it’s possible. Tomorrow I’ll let you know.

Cheers,

T.

[Capturing the thread]
On May 31, 2015, at 10:11 PM, Tom Henderson <tomh@tomh.org> wrote:

I'd suggest to just return with an exit code 1 (which test.py will report as FAIL, not CRASH), and not use test macros in test.h (which are designed for use with the test framework) but instead use stderr or logging statements to report what went wrong.  These statements won't show up when running test.py, but people can simply rerun the test with waf to see this extra output.

- Tom

[Capturing the thread]
On Jun 1, 2015, at 10:35 AM, Barnes, Peter D. <barnes26@llnl.gov> wrote:
As an alternative, does (or could) test.py set an environment variable?  Then a new NS_EXAMPLE_ASSERT(...) macro could automatically check it, before calling NS_TEST_ASSERT(...)  This would be simpler for users IMHO, and avoid capturing the generic command argument "--test" as having special meaning, de-facto reserved for ns-3 examples.

Peter

[Capturing the thread]
On Jun 2, 2015, at 12:54 AM, Tom Henderson <tomh@tomh.org> wrote:

I'd be fine with this solution also.

- Tom

Created attachment 2119 [details]
Enhanced sample-simulator that reports errors and returns error code.

As usual, Tom's first suggestion was correct:  test.py checks the shell return code provided by the example, so nothing special is absolutely needed for test.py.

What *is* needed are examples that check their results, report if they are unexpected, and return an error code.

I added a few lines to sample-simulator.cc as an example, in the patch.  To force it to (incorrectly) indicate failure, change the value of EXPECTED near the top of the file.

Run this way to see full output:  $ ./waf --run sample-simulator

Run this way to test:  $ ./test.py --example sample-simulator

(In reply to Peter Barnes from comment #7)
> Created attachment 2119 [details]
> Enhanced sample-simulator that reports errors and returns error code.
> 
> As usual, Tom's first suggestion was correct:  test.py checks the shell
> return code provided by the example, so nothing special is absolutely needed
> for test.py.
> 
> What *is* needed are examples that check their results, report if they are
> unexpected, and return an error code.
> 
> I added a few lines to sample-simulator.cc as an example, in the patch.  To
> force it to (incorrectly) indicate failure, change the value of EXPECTED
> near the top of the file.
> 
> Run this way to see full output:  $ ./waf --run sample-simulator
> 
> Run this way to test:  $ ./test.py --example sample-simulator

I know that this works. The pros of having an NS_EXAMPLE_ASSERT macro is that you can not only do a test, but also provide an error message.
It's a matter of code clarity tbh, as these two snippets should be roughly be equivalent:
If (some_condition)
{
  NS_LOG_ERROR ("I can't believe but we found a bug");
  return -1;
}

and
NS_EXAMPLE_ASSERT_MSG (!some_condition, "I can't believe but we found a bug");

I'm totally fine with simply returning an error code (and having an error message). I just think that a macro would be more readable.

(In reply to Tommaso Pecorella from comment #8)

> 
> I'm totally fine with simply returning an error code (and having an error
> message). I just think that a macro would be more readable.


I agree that the macro is more readable, but keep in mind that users typically write their own scripts by copying examples, so the use of this macro will propagate beyond the test suite.

I'm neutral to slightly -1 on the new macro for this reason.  What about using FATAL_ERROR for readability?

(In reply to Tom Henderson from comment #9)
> (In reply to Tommaso Pecorella from comment #8)
> 
> > 
> > I'm totally fine with simply returning an error code (and having an error
> > message). I just think that a macro would be more readable.
> 
> 
> I agree that the macro is more readable, but keep in mind that users
> typically write their own scripts by copying examples, so the use of this
> macro will propagate beyond the test suite.
> 
> I'm neutral to slightly -1 on the new macro for this reason.  What about
> using FATAL_ERROR for readability?

p.s. I realize that you asked for something slightly different (asserts that trigger only in test suite context, to suppress bad copy/paste/edit experiences on the mailing list) but IMO if we put condition checking in the examples with good failure output text, the positive effect of propagating that might outweigh the copy/paste issues on the list.

I think the essence of this has been addressed by the examples-as-tests work recently merged in GitLab !245 commit  5dab9258