Bug 1926

Summary: bake stops downloading if source directory already found, regardless of version
Product: bake Reporter: Hans van den Bogert <hansbogert>
Component: bakeAssignee: Daniel Camara <daniel.camara>
Status: CONFIRMED ---    
Severity: enhancement CC: ns-bugs, tomh
Priority: P5    
Version: unspecified   
Hardware: PC   
OS: Mac OS   
Attachments: adds "-dev" for the pybindgen location
set revision for pybindgen

Description Hans van den Bogert 2014-06-12 07:51:20 EDT
When using Ns-3-dev waf cannot find pybindgen, this is because the wscript which looks for ../pybindgen should be looking for ../pybindgen-dev.

Suggested patch should work
Comment 1 Hans van den Bogert 2014-06-12 07:52:22 EDT
Created attachment 1847 [details]
adds "-dev" for the pybindgen location
Comment 2 Tom Henderson 2014-06-13 19:17:23 EDT
I'm guessing that this issue was encountered when using the bake tool.  Can you confirm?

I pushed a change in this patchset which will show up in ns-3.20 release.  The change is different from your proposed patch (to enable the search for pybindgen-dev by waf).  Instead, I eliminated the modulename 'pybindgen-dev' in favor of 'pybindgen'.

here is the changeset:
http://code.nsnam.org/bake/rev/8d78e295f29f

I would like to leave this issue open, or open a new one, to clean up the module naming in bake (reducing the use of "-dev" module names when the upstream package such as nsc does not use this convention).

Does the current patch resolve your immediate problem,though?
Comment 3 Hans van den Bogert 2014-06-15 07:55:11 EDT
(In reply to Tom Henderson from comment #2)
> I'm guessing that this issue was encountered when using the bake tool.  Can
> you confirm?
Yes, encountered using bake

[...]

> Does the current patch resolve your immediate problem,though?

No, the location is correct now, however there is a version mismatch for pybindgen when the "./waf configure" command is run in ns-3-dev (that problem was already there though.)

The bakeconf.xml does not set a required version/revision for pybindgen, so the checkout is the latest. (872 as of writing), whereas the required version during the configure phase is 868. Manually checking out an older revision in an already downloaded pybindgen repo does not work, because the configure script does not look at the tree revno, but the repository revno ("bzr revno --tree" vs. "bzr revno"). My workaround was, deleting the whole pybindgen directory and set the supplied patch.
Comment 4 Hans van den Bogert 2014-06-15 07:55:57 EDT
Created attachment 1848 [details]
set revision for pybindgen
Comment 5 Tom Henderson 2014-06-15 11:49:30 EDT
(In reply to Hans van den Bogert from comment #4)
> Created attachment 1848 [details]
> set revision for pybindgen

I'm going to handle this in another way.  We're on the verge of making an ns-3 release and I haven't tested the diff between r868 and r872, so I plan to update the "required version" in ns-3-dev/bindings/python/wscript once we make the release (in a day or two).  Then I plan to start testing this daily so we don't fall out of sync.  (i.e. add bakeconf.xml's 'ns-3-allinone' module target to our regression scripts).

The fact that ns-3-dev is behind pybindgen doesn't cause a problem with the download.py script in the ns-3-allinone directory since it downloads ns-3-dev first and then checks the wscript to find the right revno for pybindgen, but bake does not work in this way.

If you have any other feedback on bake, comments would be appreciated.
Comment 6 Hans van den Bogert 2014-06-15 14:59:51 EDT
(In reply to Tom Henderson from comment #5)
> ...
> The fact that ns-3-dev is behind pybindgen doesn't cause a problem with the
> download.py script in the ns-3-allinone directory since it downloads
> ns-3-dev first and then checks the wscript to find the right revno for
> pybindgen, but bake does not work in this way.
> 
> If you have any other feedback on bake, comments would be appreciated.

Wouldn't it be better to keep these dependencies in a central place. It seems bake continuously needs to be updated with information which apparently is already entered somewhere else? 
Furthermore,  the issue I had with pybindgen also stems from the fact that bake simply stops downloading a package if it finds the directory is already there, but it isn't checked for the right version. If a dependency package, like pybindgen, is updated since the last configuration, bake does not checkout the newer version. Checking out older revisions is also not done by bake if you do a 're'-configure from say dce-ns3-dev to ns3-dev. The only options are to do a 1) './waf fullclean [-o module]' 2) remove the source folders manually 3) checkout the right version.

Maybe this is intended behaviour, i.e. bake should only be used for downloading and building ns-3 for the first time, and afterwards an user should git/bzr/hg update sources himself?
Comment 7 Hans van den Bogert 2014-06-15 15:01:52 EDT
please disregard, because that example does not apply:
> Checking out older revisions is also not done by bake if you do a 
> 're'-configure from say dce-ns3-dev to ns3-dev.
Comment 8 Tom Henderson 2014-06-15 20:04:51 EDT
(In reply to Hans van den Bogert from comment #6)
> (In reply to Tom Henderson from comment #5)
> > ...
> > The fact that ns-3-dev is behind pybindgen doesn't cause a problem with the
> > download.py script in the ns-3-allinone directory since it downloads
> > ns-3-dev first and then checks the wscript to find the right revno for
> > pybindgen, but bake does not work in this way.
> > 
> > If you have any other feedback on bake, comments would be appreciated.
> 
> Wouldn't it be better to keep these dependencies in a central place. It
> seems bake continuously needs to be updated with information which
> apparently is already entered somewhere else? 

We should keep them in the bakeconf.xml.  The fact that it presently doesn't work for ns-3-dev is an oversight on our part (we missed updating ns-3-dev when Gustavo updated pybindgen) but we can fix it soon.

> Furthermore,  the issue I had with pybindgen also stems from the fact that
> bake simply stops downloading a package if it finds the directory is already
> there, but it isn't checked for the right version. 

I agree that this is a current problem.  I believe that it always used to overwrite existing directory, and then Daniel changed (based on a comment of mine from a while back that bake seemed to be inefficiently always downloading anew each package) to download if not found, but something smarter is needed. 

If a dependency package,
> like pybindgen, is updated since the last configuration, bake does not
> checkout the newer version. Checking out older revisions is also not done by
> bake if you do a 're'-configure from say dce-ns3-dev to ns3-dev. The only
> options are to do a 1) './waf fullclean [-o module]' 2) remove the source
> folders manually 3) checkout the right version.
> 
> Maybe this is intended behaviour, i.e. bake should only be used for
> downloading and building ns-3 for the first time, and afterwards an user
> should git/bzr/hg update sources himself?

I think that the goal from a developer's point of view would be to be able to freshen the modules continually.  It may be that a distinction between download and update is needed.
Comment 9 Tom Henderson 2014-06-23 13:51:57 EDT
./bake.py configure -e ns-3-allinone now works since I updated ns-3-dev to require the tip of the pybindgen repository.

There is a lingering issue about how bake handles download (for the first time) vs update/freshen; leaving this open for that.