Bug 348 - regression tests do not work in private or disconnected networks
regression tests do not work in private or disconnected networks
Status: RESOLVED FIXED
Product: ns-3
Classification: Unclassified
Component: regression
ns-3.2
All All
: P2 normal
Assigned To: Gustavo J. A. M. Carneiro
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2008-09-14 14:18 EDT by Tom Henderson
Modified: 2008-11-19 12:38 EST (History)
2 users (show)

See Also:


Attachments

Note You need to log in before you can comment on or make changes to this bug.
Description Tom Henderson 2008-09-14 14:18:45 EDT
A typical use case is that someone will download the release and use it on a machine where it is not well connected to the Internet, or where the user's http_proxy variable is not set.

A user may first download the release tarball only.  Suppose this tarball is downloaded onto a machine where either it is behind a NAT and http_proxy is not set, or the machine may not even have reachability to the public network.  For instance, a user who wants to try out ns-3 during a train ride.

I think the behavior in this case, regardless of whether hg is installed or not, is reasonable with the current release candidate; a user who types in "./waf --regression" on this released version will get a connect error in a reasonably short time:  "Abort: name or service name not known".  How to remedy this can be documented; waf could even output some suggestions if this socket error occurs.

However, what if this user then tries to separately download the matching regression trace tarball and unpack it on the machine?  He or she will put these regression traces in the ns-3.2/regression trace directory, and try again "./waf --regression" and will get this type of error:

========== Running Regression Tests ==========
Retrieving ns-3.2-RC2-bis-ref-traces.tar.bz2 from web.
Traceback (most recent call last):
  File "./waf", line 141, in <module>
    Scripting.prepare()
  File "/home/tomh/ns-3.2-RC2-bis/.waf-1.4.2-81764acf62a7c7247901b9a99367b3a2/wafadmin/Scripting.py", line 207, in prepare
    main()
  File "/home/tomh/ns-3.2-RC2-bis/.waf-1.4.2-81764acf62a7c7247901b9a99367b3a2/wafadmin/Scripting.py", line 299, in main
    if fun:fun()
  File "/home/tomh/ns-3.2-RC2-bis/wscript", line 451, in shutdown
    run_regression()
  File "/home/tomh/ns-3.2-RC2-bis/wscript", line 934, in run_regression
    urllib.urlretrieve(REGRESSION_TRACES_URL + traceball, traceball)
  File "/usr/lib/python2.5/urllib.py", line 89, in urlretrieve
    return _urlopener.retrieve(url, filename, reporthook, data)
  File "/usr/lib/python2.5/urllib.py", line 222, in retrieve
    fp = self.open(url, data)
  File "/usr/lib/python2.5/urllib.py", line 190, in open
    return getattr(self, name)(url)
  File "/usr/lib/python2.5/urllib.py", line 325, in open_http
    h.endheaders()
  File "/usr/lib/python2.5/httplib.py", line 860, in endheaders
    self._send_output()
  File "/usr/lib/python2.5/httplib.py", line 732, in _send_output
    self.send(msg)
  File "/usr/lib/python2.5/httplib.py", line 699, in send
    self.connect()
  File "/usr/lib/python2.5/httplib.py", line 667, in connect
    socket.SOCK_STREAM):
IOError: [Errno socket error] (-2, 'Name or service not known')

I verified this on a machine behind a NAT, both with and without hg in the path, and got similar behavior.  I wasn't able to drop in the release trace tarball and have it work seamlessly, nor do I know how to make it work.  

I think that the current behavior is that if these traces are placed into the regression/ directory, waf assumes that it must go out and synchronize these traces, and will fail with the above type of error.

So, I think that for user-friendliness, one of these steps needs to be taken:

1) include regression traces in the main tarball, and have the system work self-contained even in the absence of network connectivity
2) document how one can separately download both the release and regression traces, put them together, and make it work in a non-connected environment

This also is problematic on the development tree.  Similarly, I just tried the following on ns-3-dev:

hg pull
hg update
./waf --regression
(disable network access)
./waf --regression
========== Running Regression Tests ==========
Synchronizing reference traces using Mercurial.
Pulling http://code.nsnam.org/ns-3-dev-ref-traces from repo.
abort: error: No address associated with nodename
Synchronizing reference traces using Mercurial failed. 


I really ought to be able to tell the system "Just use whatever regression traces you have downloaded in the past-- do not try to synchronize them"  Can I do that somehow?  ./waf configure --offline  or ./waf configure --disable-updates?
Comment 1 Mathieu Lacage 2008-09-14 14:40:31 EDT
changeset fc078b692b68
Comment 2 Tom Henderson 2008-09-14 16:37:49 EDT
Does not work for me (OS X, ppc); I still cannot run regression when disconnected from the network.  I still get:

powerbook:/scratch/hg/ns-3-dev% ./waf --regression
Entering directory `/scratch/hg/ns-3-dev/build'
Compilation finished successfully 
========== Running Regression Tests ==========
Synchronizing reference traces using Mercurial.
Pulling http://code.nsnam.org/ns-3.2-RC2-bis-ref-traces from repo.
abort: error: No address associated with nodename
Synchronizing reference traces using Mercurial failed. 

I do not see from the changeset how waf is detecting that I am offline.

Also, I don't understand why you changed the VERSION flag on ns-3-dev repo.
Comment 3 Mathieu Lacage 2008-09-14 20:57:47 EDT
(In reply to comment #2)
> Does not work for me (OS X, ppc); I still cannot run regression when
> disconnected from the network.  I still get:
> 
> powerbook:/scratch/hg/ns-3-dev% ./waf --regression
> Entering directory `/scratch/hg/ns-3-dev/build'
> Compilation finished successfully 
> ========== Running Regression Tests ==========
> Synchronizing reference traces using Mercurial.
> Pulling http://code.nsnam.org/ns-3.2-RC2-bis-ref-traces from repo.
> abort: error: No address associated with nodename
> Synchronizing reference traces using Mercurial failed. 
> 
> I do not see from the changeset how waf is detecting that I am offline.

It does not but, if you uncompress the regression tarballs, it won't try to fetch them again now which I consider is a fix for your problem good enough for the release. Please mark bug fixed if this is ok. If this is not ok, you will have to fix it yourself: it's just a python script after all.

> 
> Also, I don't understand why you changed the VERSION flag on ns-3-dev repo.

ha, I screwed up that part. reverted. sorry about that.

> 

Comment 4 Mathieu Lacage 2008-09-14 21:01:11 EDT
I should point out that I disagree that the solution is to add yet another flag to our waf scripts. The real solution is to clealy split all this crap network access from the main waf scripts but you already refused to do this so I can't do much more.
Comment 5 Craig Dowell 2008-09-14 21:42:12 EDT
We've been pulled in a lot of different directions regarding various options, platforms, regression tests, networks, no networks, etc.

I think we need to take some time after ns-3.2 and figure out what ecactly we want this stuff to do and actually design a solution.  I saw an email about ns-3.3 priorities fly by.  I think that a build system evaluation and cleanup is in order for that release.
Comment 6 Tom Henderson 2008-09-15 00:27:48 EDT
> > 
> > I do not see from the changeset how waf is detecting that I am offline.
> 
> It does not but, if you uncompress the regression tarballs, it won't try to
> fetch them again now which I consider is a fix for your problem good enough for
> the release. Please mark bug fixed if this is ok. If this is not ok, you will
> have to fix it yourself: it's just a python script after all.

Not working for me still.  We can chat about it tomorrow AM.
Comment 7 Tom Henderson 2008-09-17 00:57:02 EDT
> 
> Not working for me still.  We can chat about it tomorrow AM.
> 

Deferred until post-3.2 release.
Comment 8 Gustavo J. A. M. Carneiro 2008-11-18 13:09:34 EST
In the NS-3.3 plan I see:
  #  Build System Refactoring -- Dropped from ns-3.3 

However, isn't this part of the build system refactoring?
Comment 9 Craig Dowell 2008-11-18 13:54:16 EST
Yes, this was originally part of the build system refactoring work; but it bothers one of us sufficiently that he wants it broken out and implemented separately -- as a blocker for ns-3.3.
Comment 10 Gustavo J. A. M. Carneiro 2008-11-18 14:00:23 EST
Could the person that is most bothered care to explain with more clarity what he wants to see implemented?  I must say I that I am confused after reading this bug report and could use a concise and clear summary of what we want to accomplish.

In parallel I am trying now to give a go at a 'download.py' script for the build system refactoring.  I probably won't finish build system refactoring in time for 3.3, but at least it's a start.
Comment 11 Craig Dowell 2008-11-18 14:38:23 EST
Well, I'm not "the guy," but we've talked about it sufficiently that I think I can at least try and speak for him.

The essential use case is a developer using a laptop with intermittent network connectivity.  The user wants to be able download our code and the regression traces while a network connection is available -- either by tarball or by Mercurial.  He then wants to take his laptop to an area in which networking is not available and do development locally.  He wants to be able to run the regression tests without having to return to a net connection.

The simplest thing seems to be to add an option to run regression without checking for or downloading new trace files; something along the lines of,

  ./waf --regression-static    # run tests without checking for new bits

Then, the user in question, who's name rhymes remarkably well with Henderson, could download the ns-3 bits and run the regression tests once while connected.  He could then run the above command whenever he is not connected.

Did this cover the issue, Tom?
Comment 12 Tom Henderson 2008-11-19 01:09:17 EST
(In reply to comment #11)
> Well, I'm not "the guy," but we've talked about it sufficiently that I think I
> can at least try and speak for him.
> 
> The essential use case is a developer using a laptop with intermittent network
> connectivity.  The user wants to be able download our code and the regression
> traces while a network connection is available -- either by tarball or by
> Mercurial.  He then wants to take his laptop to an area in which networking is
> not available and do development locally.  He wants to be able to run the
> regression tests without having to return to a net connection.
> 
> The simplest thing seems to be to add an option to run regression without
> checking for or downloading new trace files; something along the lines of,
> 
>   ./waf --regression-static    # run tests without checking for new bits
> 
> Then, the user in question, who's name rhymes remarkably well with Henderson,
> could download the ns-3 bits and run the regression tests once while connected.
>  He could then run the above command whenever he is not connected.
> 
> Did this cover the issue, Tom?
> 

Yes.  I suggest that we just add a workaround for ns-3.3 while the build system refactoring is being worked on.

Here are the two problems I'd like to solve ASAP:

1) I cannot run ./waf --regression when I am offline, even if I have traces in the directory
2) I may want to suppress the downloading of regression traces even when on line.  For instance, when I am working in a private repo that was forked from ns-3-dev, and I'd like to keep the regression traces frozen to avoid having to merge with ns-3-dev at inopportune times.

./waf --regression-static as Craig suggested would seem to handle both problems.  Or else if we could leave some lint file in the directory that would suppress downloads (e.g. a ".regression-static" file).
Comment 13 Gustavo J. A. M. Carneiro 2008-11-19 12:38:50 EST
Since I have to think of the future 'allinone' build system some people wanted, and to avoid explosion of options, instead I added just a --with-reference-traces option, e.g.:

./waf configure --with-reference-traces=regression/ns-3-dev-my-traces

This has two effects folded into one: 1. allows using custom traces, possibly from outside the ns-3 tree, 2. when this option is given, network access is disabled in the future for updating traces.