Bug 127

Created attachment 103 [details]
proposed patch

It turns out that TraceResolver::CollectSources works differently from what I expected; it seems to collect all sources below a certain path, while I want all sources that _match_ a certain path.  I committed a new API, on top of the attached patch, TraceResolver::CollectSourcesMatchingPath, and implemented python bindings for Object::TraceConnect, as proof of concept.

http://code.nsnam.org/gjc/ns-3-pybindgen/rev/e0e287057339

This is still an issue AFAIK.  Not P1 because Python bindings will be merged after ns 3.1.

(In reply to comment #3)
> This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> after ns 3.1.
> 

Gustavo, can you clarify the status of this?  My understanding is that:

1) if a callback is exported explicitly at the API, it is possible to register a python method with it, thereby enabling people to write python code for ns-3

2) but due to this bug, we just can't hook them through the trace hooks obtained through the trace path

If I am correct, are there any examples (even a small sample program for samples/ directory) of how to do this?  Someone here at UW would like to try writing some Python ns-3 code.

(In reply to comment #4)
> (In reply to comment #3)
> > This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> > after ns 3.1.
> > 
> 
> Gustavo, can you clarify the status of this?  My understanding is that:
> 
> 1) if a callback is exported explicitly at the API, it is possible to register
> a python method with it, thereby enabling people to write python code for ns-3
> 
> 2) but due to this bug, we just can't hook them through the trace hooks
> obtained through the trace path
> 
> If I am correct, are there any examples (even a small sample program for
> samples/ directory) of how to do this?  Someone here at UW would like to try
> writing some Python ns-3 code.
> 

The problem is that 1) in Python we cannot introspect the signature of a "callable" (i.e. function or method, or class instance with a __call__ method), and 2) NS-3 also does not provide an API that says what is the expected callback type for a certain trace path, but it eventually checks in runtime whether supplied callbacks are of the correct type.  Although the Config::Connect API accepts a generic callback type (CallbackBase) as parameter, it later matches it against one of the concrete callback types (Callback<R,T1,T2,...>).  So the Python bindings have to supply a concrete callback wrapper based on a Python callable for which I cannot obtain signature information.

I can think of two possible and contrasting ways to move forward:

 1- Add a new NS-3 API to give back callback type(s) regarding trace path.  The Python bindings could work things out and the Python API would turn out to be simple:


   def my_callback(packet):
      print packet

   ns3.Config.Connect("/NodeList/*/....", my_callback)


 2- Force the Python programmer to tell the NS-3 Python bindings what is the callback signature via a decorator design pattern, e.g.:

   def my_callback(packet):
      print packet

   ns3.Config.Connect("/NodeList/*/....", ns3.Callback__lt__Ptr__lt__Packet__gt____gt__(my_callback))


Obviously I would prefer option 1 because it is much simpler to the NS-3 programmer.  I wonder if there isn't some Match API already in the works that could be used as basis for this?...

(In reply to comment #5)
> (In reply to comment #4)
> > (In reply to comment #3)
> > > This is still an issue AFAIK.  Not P1 because Python bindings will be merged
> > > after ns 3.1.
> > > 
> > 
> > Gustavo, can you clarify the status of this?  My understanding is that:
> > 
> > 1) if a callback is exported explicitly at the API, it is possible to register
> > a python method with it, thereby enabling people to write python code for ns-3

Is the above true?  Are there any examples of it presently?

> > 
> > 2) but due to this bug, we just can't hook them through the trace hooks
> > obtained through the trace path
> > 
> > If I am correct, are there any examples (even a small sample program for
> > samples/ directory) of how to do this?  Someone here at UW would like to try
> > writing some Python ns-3 code.
> > 
> 
> The problem is that 1) in Python we cannot introspect the signature of a
> "callable" (i.e. function or method, or class instance with a __call__ method),
> and 2) NS-3 also does not provide an API that says what is the expected
> callback type for a certain trace path, but it eventually checks in runtime
> whether supplied callbacks are of the correct type.  Although the
> Config::Connect API accepts a generic callback type (CallbackBase) as
> parameter, it later matches it against one of the concrete callback types
> (Callback<R,T1,T2,...>).  So the Python bindings have to supply a concrete
> callback wrapper based on a Python callable for which I cannot obtain signature
> information.
> 
> I can think of two possible and contrasting ways to move forward:
> 
>  1- Add a new NS-3 API to give back callback type(s) regarding trace path.  The
> Python bindings could work things out and the Python API would turn out to be
> simple:
> 
> 
>    def my_callback(packet):
>       print packet
> 
>    ns3.Config.Connect("/NodeList/*/....", my_callback)
> 

clearly that would be nicer on users but I don't know what is involved in supporting that.

(In reply to comment #6)
> > > Gustavo, can you clarify the status of this?  My understanding is that:
> > > 
> > > 1) if a callback is exported explicitly at the API, it is possible to register
> > > a python method with it, thereby enabling people to write python code for ns-3
> 
> Is the above true?  Are there any examples of it presently?

Sorry, forgot to answer that.

No, I'm afraid it is _not_ true.  It is currently impossible to connect callbacks to trace events in Python.

What is possible is to pass Python callable objects into NS-3 APIs that expect _concrete callbacks_.  Most notable example of this is sockets.  There is even a unit test connecting Python callbacks to NS-3 sockets (see utils/python-unit-tests.py).

This bug report is about dealing with APIs that take a callback object of unspecified type (parameters of type const CallbackBase &).  The type is know somewhere deep in the guts of the NS-3 runtime, and is checked for, but it is impossible to query NS-3 what is the callback type before connecting to it using e.g. Config::Connect.

I am kind of reluctant to work on the NS-3 side of this because last time I provided a patch it was left to bitrot and is now completely obsolete.  I don't blame Mathieu, I know he was/is very busy, but I found it very difficult to produce the patch the first time around, and I would imagine similar difficulty again.

(In reply to comment #7)

> I am kind of reluctant to work on the NS-3 side of this because last time I
> provided a patch it was left to bitrot and is now completely obsolete.  I don't

The reason for that is that the whole tracing system was reworked at that time. It seems unlikely that this is going to happen in the near future again so, if you can come up with a patch, I don't see any reason not to merge it asap.

The last time this issue came up, the patch to the core ns-3 code was fairly minimal and mainly involved getting a typeid string from the CallbackBase base class. The main issue seemed to be to write code in the python binding to make use of that. I suspect that what you had in mind was something along the lines of:

1) take the user-provided path, use it to find all matching trace sources
2) for each matching trace source, obtain the typeid, and find a matching autogenerated C++ trace sink, instanciate that trace sink, connect it to the trace source, and make it call python with the right arguments.

So, it seems that the main challenge is generating all the possible C++ trace sinks, right ?

Anyway, if the above description of what needs to be done is correct, a more robust API would be to not export a typeid string but to export a typeid directly. i.e., CallbackBase would get a std::typeid GetTypeId (void) const; method and you would store in your binding code a table of pairs of std::typeid+callback constructor instead of dealing with a typeid string.

(In reply to comment #8)
> (In reply to comment #7)
> 
> > I am kind of reluctant to work on the NS-3 side of this because last time I
> > provided a patch it was left to bitrot and is now completely obsolete.  I don't
> 
> The reason for that is that the whole tracing system was reworked at that time.
> It seems unlikely that this is going to happen in the near future again so, if
> you can come up with a patch, I don't see any reason not to merge it asap.
> 
> The last time this issue came up, the patch to the core ns-3 code was fairly
> minimal and mainly involved getting a typeid string from the CallbackBase base
> class. The main issue seemed to be to write code in the python binding to make
> use of that. I suspect that what you had in mind was something along the lines
> of:
> 
> 1) take the user-provided path, use it to find all matching trace sources
> 2) for each matching trace source, obtain the typeid, and find a matching
> autogenerated C++ trace sink, instanciate that trace sink, connect it to the
> trace source, and make it call python with the right arguments.
> 
> So, it seems that the main challenge is generating all the possible C++ trace
> sinks, right ?

Right.  This makes sense to me.

In NS-3, tracing is like a regular expression.  We should have an API that returns all "regexp matches", sort of speak.  For each match, an object that represents the match could include some properties, like config path for the match, type of match (attribute or traced callback), and callback type in case of traced callback matches.

> 
> Anyway, if the above description of what needs to be done is correct, a more
> robust API would be to not export a typeid string but to export a typeid
> directly. i.e., CallbackBase would get a std::typeid GetTypeId (void) const;
> method and you would store in your binding code a table of pairs of
> std::typeid+callback constructor instead of dealing with a typeid string.

Right.  Except that it's not std::typeid, more like:

  const std::type_info & GetTypeInfo (void) const;

(you cannot copy type_info objects, you always have to pass around const references).

The std::type_info reference would be obtained via (e.g.) typeid(Callback<void, Ptr<const Packet> >).