Created an attachment (id=103) [details]
proposed patch

It turns out that TraceResolver::CollectSources works differently from what I
expected; it seems to collect all sources below a certain path, while I want
all sources that _match_ a certain path.  I committed a new API, on top of the
attached patch, TraceResolver::CollectSourcesMatchingPath, and implemented
python bindings for Object::TraceConnect, as proof of concept.

http://code.nsnam.org/gjc/ns-3-pybindgen/rev/e0e287057339

This is still an issue AFAIK.  Not P1 because Python bindings will be merged
after ns 3.1.

(In reply to comment #3)
> This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> after ns 3.1.
> 

Gustavo, can you clarify the status of this?  My understanding is that:

1) if a callback is exported explicitly at the API, it is possible to register
a python method with it, thereby enabling people to write python code for ns-3

2) but due to this bug, we just can't hook them through the trace hooks
obtained through the trace path

If I am correct, are there any examples (even a small sample program for
samples/ directory) of how to do this?  Someone here at UW would like to try
writing some Python ns-3 code.

(In reply to comment #4)
> (In reply to comment #3)
> > This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> > after ns 3.1.
> > 
> 
> Gustavo, can you clarify the status of this?  My understanding is that:
> 
> 1) if a callback is exported explicitly at the API, it is possible to register
> a python method with it, thereby enabling people to write python code for ns-3
> 
> 2) but due to this bug, we just can't hook them through the trace hooks
> obtained through the trace path
> 
> If I am correct, are there any examples (even a small sample program for
> samples/ directory) of how to do this?  Someone here at UW would like to try
> writing some Python ns-3 code.
> 

The problem is that 1) in Python we cannot introspect the signature of a
"callable" (i.e. function or method, or class instance with a __call__ method),
and 2) NS-3 also does not provide an API that says what is the expected
callback type for a certain trace path, but it eventually checks in runtime
whether supplied callbacks are of the correct type.  Although the
Config::Connect API accepts a generic callback type (CallbackBase) as
parameter, it later matches it against one of the concrete callback types
(Callback<R,T1,T2,...>).  So the Python bindings have to supply a concrete
callback wrapper based on a Python callable for which I cannot obtain signature
information.

I can think of two possible and contrasting ways to move forward:

 1- Add a new NS-3 API to give back callback type(s) regarding trace path.  The
Python bindings could work things out and the Python API would turn out to be
simple:


   def my_callback(packet):
      print packet

   ns3.Config.Connect("/NodeList/*/....", my_callback)


 2- Force the Python programmer to tell the NS-3 Python bindings what is the
callback signature via a decorator design pattern, e.g.:

   def my_callback(packet):
      print packet

   ns3.Config.Connect("/NodeList/*/....",
ns3.Callback__lt__Ptr__lt__Packet__gt____gt__(my_callback))


Obviously I would prefer option 1 because it is much simpler to the NS-3
programmer.  I wonder if there isn't some Match API already in the works that
could be used as basis for this?...

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> > > after ns 3.1.
> > > 
> > 
> > Gustavo, can you clarify the status of this?  My understanding is that:
> > 
> > 1) if a callback is exported explicitly at the API, it is possible to register
> > a python method with it, thereby enabling people to write python code for ns-3

Is the above true?  Are there any examples of it presently?

> > 
> > 2) but due to this bug, we just can't hook them through the trace hooks
> > obtained through the trace path
> > 
> > If I am correct, are there any examples (even a small sample program for
> > samples/ directory) of how to do this?  Someone here at UW would like to try
> > writing some Python ns-3 code.
> > 
> 
> The problem is that 1) in Python we cannot introspect the signature of a
> "callable" (i.e. function or method, or class instance with a __call__ method),
> and 2) NS-3 also does not provide an API that says what is the expected
> callback type for a certain trace path, but it eventually checks in runtime
> whether supplied callbacks are of the correct type.  Although the
> Config::Connect API accepts a generic callback type (CallbackBase) as
> parameter, it later matches it against one of the concrete callback types
> (Callback<R,T1,T2,...>).  So the Python bindings have to supply a concrete
> callback wrapper based on a Python callable for which I cannot obtain signature
> information.
> 
> I can think of two possible and contrasting ways to move forward:
> 
>  1- Add a new NS-3 API to give back callback type(s) regarding trace path.  The
> Python bindings could work things out and the Python API would turn out to be
> simple:
> 
> 
>    def my_callback(packet):
>       print packet
> 
>    ns3.Config.Connect("/NodeList/*/....", my_callback)
> 

clearly that would be nicer on users but I don't know what is involved in
supporting that.

(In reply to comment #6)
> > > Gustavo, can you clarify the status of this?  My understanding is that:
> > > 
> > > 1) if a callback is exported explicitly at the API, it is possible to register
> > > a python method with it, thereby enabling people to write python code for ns-3
> 
> Is the above true?  Are there any examples of it presently?

Sorry, forgot to answer that.

No, I'm afraid it is _not_ true.  It is currently impossible to connect
callbacks to trace events in Python.

What is possible is to pass Python callable objects into NS-3 APIs that expect
_concrete callbacks_.  Most notable example of this is sockets.  There is even
a unit test connecting Python callbacks to NS-3 sockets (see
utils/python-unit-tests.py).

This bug report is about dealing with APIs that take a callback object of
unspecified type (parameters of type const CallbackBase &).  The type is know
somewhere deep in the guts of the NS-3 runtime, and is checked for, but it is
impossible to query NS-3 what is the callback type before connecting to it
using e.g. Config::Connect.

I am kind of reluctant to work on the NS-3 side of this because last time I
provided a patch it was left to bitrot and is now completely obsolete.  I don't
blame Mathieu, I know he was/is very busy, but I found it very difficult to
produce the patch the first time around, and I would imagine similar difficulty
again.

(In reply to comment #7)

> I am kind of reluctant to work on the NS-3 side of this because last time I
> provided a patch it was left to bitrot and is now completely obsolete.  I don't

The reason for that is that the whole tracing system was reworked at that time.
It seems unlikely that this is going to happen in the near future again so, if
you can come up with a patch, I don't see any reason not to merge it asap.

The last time this issue came up, the patch to the core ns-3 code was fairly
minimal and mainly involved getting a typeid string from the CallbackBase base
class. The main issue seemed to be to write code in the python binding to make
use of that. I suspect that what you had in mind was something along the lines
of:

1) take the user-provided path, use it to find all matching trace sources
2) for each matching trace source, obtain the typeid, and find a matching
autogenerated C++ trace sink, instanciate that trace sink, connect it to the
trace source, and make it call python with the right arguments.

So, it seems that the main challenge is generating all the possible C++ trace
sinks, right ?

Anyway, if the above description of what needs to be done is correct, a more
robust API would be to not export a typeid string but to export a typeid
directly. i.e., CallbackBase would get a std::typeid GetTypeId (void) const;
method and you would store in your binding code a table of pairs of
std::typeid+callback constructor instead of dealing with a typeid string.

(In reply to comment #8)
> (In reply to comment #7)
> 
> > I am kind of reluctant to work on the NS-3 side of this because last time I
> > provided a patch it was left to bitrot and is now completely obsolete.  I don't
> 
> The reason for that is that the whole tracing system was reworked at that time.
> It seems unlikely that this is going to happen in the near future again so, if
> you can come up with a patch, I don't see any reason not to merge it asap.
> 
> The last time this issue came up, the patch to the core ns-3 code was fairly
> minimal and mainly involved getting a typeid string from the CallbackBase base
> class. The main issue seemed to be to write code in the python binding to make
> use of that. I suspect that what you had in mind was something along the lines
> of:
> 
> 1) take the user-provided path, use it to find all matching trace sources
> 2) for each matching trace source, obtain the typeid, and find a matching
> autogenerated C++ trace sink, instanciate that trace sink, connect it to the
> trace source, and make it call python with the right arguments.
> 
> So, it seems that the main challenge is generating all the possible C++ trace
> sinks, right ?

Right.  This makes sense to me.

In NS-3, tracing is like a regular expression.  We should have an API that
returns all "regexp matches", sort of speak.  For each match, an object that
represents the match could include some properties, like config path for the
match, type of match (attribute or traced callback), and callback type in case
of traced callback matches.

> 
> Anyway, if the above description of what needs to be done is correct, a more
> robust API would be to not export a typeid string but to export a typeid
> directly. i.e., CallbackBase would get a std::typeid GetTypeId (void) const;
> method and you would store in your binding code a table of pairs of
> std::typeid+callback constructor instead of dealing with a typeid string.

Right.  Except that it's not std::typeid, more like:

  const std::type_info & GetTypeInfo (void) const;

(you cannot copy type_info objects, you always have to pass around const
references).

The std::type_info reference would be obtained via (e.g.) typeid(Callback<void,
Ptr<const Packet> >).