Bugzilla – Bug 708
SendCallback does not work with NSC
Last modified: 2009-12-22 14:37:44 EST
Trying to modify the tcp-large-transfer to used NSC I realized that the SendCallback does not work when using NSC, in the sense that the function registered as a callback is never called.
Looking at TcpSocketImpl I've noticed that the function NotifySend (which calls the callback function) is called several times. While in NscTcpSocketImpl this function is never called.
Alberto, I know that you have done some work to deal with these issues. Could you attach a patch to this bug report ?
Mathieu, actually the only work I've done on this specific problem is to implement the modification you suggested. In NscTcpSocketImpl::SendPendingData I have replaced the call to NotifyDataSent with a call to NotifySend as follows:
Simulator::ScheduleNow(&NscTcpSocketImpl::NotifySend, this, GetTxAvailable () );
I'm not sure the value returned by GetTxAvailable is the right number in this case but it seems to work. I'm not 100% sure about the consequences of these changes and I doubt they should be widely adopted.
As you suggested the "right" solution would be to have nsc inform ns when buffer space (in nsc) is available. I was hoping it could be possible to re-use the mechanisms used for the select/poll system calls but, at least for the moment, I wasn't able to understand how these are implemented. I was at least able to find where Linux does free its buffers when ACK are received. It shouldn't be too complicated to add a function call in the Linux code to notify ns of this event but this would require explicitly adding some lines in tcp_input.c as well as adding a new function to the interface between nsc and ns. The part I'm more concerned about is the need to modify the Linux code, modifications that would be needed every time a new version is "ported" to nsc.
This bug was discussed on email some more. Here is the last email from the mail thread, by Florian. Hopefully this describes the status of this:
Florian Westphal to Alberto, me, Mathieu
Alberto Blanc wrote:
- Hide quoted text -
> On Wed, Sep 30, 2009 at 1:12 PM, Florian Westphal wrote:
> > NscTcpL4Protocol::wakeup() sort of does what you want do to:
> > it walks the list of sockets on the node, calls into
> > nsc-tcp-socket-impl.cc, and nsc-tcp-socket-impl.cc will schedule
> > calls to send_data etc.
> Following Florian's suggestion I modified
> NscTcpL4Protocol::SoftInterrupt so that it walks the list of sockets
> on the node (like wakeup does) updating the value of the cwnd for each
> one (in order to do this I added an UpdateCwnd method to
> NscTcpSocketImpl that actually updates the value of the window,
> provided the socket is in the connected state).
> Trying to test this solution I ran into another problem: I need to
> send large amounts of data using TCP-NSC in order to trace the
> evolution of the window. As far as I can tell the way of doing this
> with ns-3 is to use the SendCallback method of the socket interface.
> This is explained in the ns-3 manual and the tcp-large-transfer
> examples uses it. The idea is that the user supplies a SendCallback
> function that will be called whenever there is available space in the
> send buffer, i.e. new data can be sent because old one has been
> Unfortunately it looks like this method does not work when using NSC:
> the ns TCP implementation calls Socket::NotifySend, which calls the
> user supplied SendCallback; while, with NSC, this function is never
> called. Mathieu suggested a quick and dirty fix that seems to work:
> in NscTcpSocketImpl::SendPendingData replace the call to
> NotifyDataSent with a call to NotifySend. So I changed that line to:
> Simulator::ScheduleNow(&NscTcpSocketImpl::NotifySend, this,
> GetTxAvailable () );
> I'm not sure the value returned by GetTxAvailable is the right number
> in this case but it seems to work. I've tried understanding how the
> whole sending process but I wasn't very successful. In particular I'm
> somewhat confused by the use of m_sndBufSize and m_txBufferSize, a
> comment in the .h file seems to indicate that
> m_txBuffer is used to queue data while in the SYN_SEN state but then I got lost.
The problem is that ns-3 TCP allows you to send data right after the SYN
was sent. Data is then queued; once the 3W HS has completed, this data will
be ent out (i.e., written to the nsc socket (= The socket fd)).
In nsc, we can only write to a socket once its in connected state.
Thus packets are being put onto the txBuffer until the socket moves to
The major problem is that tcp-socket-impl.cc has a lot of TCP implementation
details (including tcp header flags...), whereas nsc-tcp-socket just
"wraps" the nsc stacks using the BSD socket API.
This means for instance that nsc cannot ever know how many bytes of the
tcp send buffer are currently used (because the socket API doesn't
provide a function to do that).
This is also the reason why you ran into the NotifyDataSent() vs.
NotifySend issue: NotifySend is called when outstanding data
gets ACKed by the peer, which is something that the nsc-wrapper doesn't
know (because it does not deal with TCP internals).
> Do you have any suggestion(s) on what would be the best way for NSC to
> notify the sender that there is space available in the (NSC) buffer?
> Looking at the sim_interface file I didn't see any function that could
> offer this information.
right -- there is no such information. The sim_interface exposes the
(most important) bsd socket system calls.
Right now, SendPendingData() schedules NotifyDataSent() once data was
written to the nsc socket fd. Maybe it should call NotifySend(1), too?
(The "1" is -- of course -- a lie, but i doubt we can do better at this
time 8-/ )
MAYBE we can keep track of an "estimate" of the nsc tx/rx buffers and
use that, problem is that the buffer sizes can change due to