Bug 742 - Implementation of SO_BINDTODEVICE
Implementation of SO_BINDTODEVICE
Status: RESOLVED FIXED
Product: ns-3
Classification: Unclassified
Component: internet
pre-release
All All
: P5 normal
Assigned To: Tom Henderson
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-11-17 10:08 EST by Antti Mäkelä
Modified: 2009-12-15 23:43 EST (History)
2 users (show)

See Also:


Attachments
Patch to allow bindsockopts (48.93 KB, patch)
2009-11-17 10:08 EST, Antti Mäkelä
Details | Diff
Simple example on using device-bound sockets. (7.41 KB, text/plain)
2009-11-17 10:08 EST, Antti Mäkelä
Details
Patch to allow SO_BINDTODEVICE, now works for both directions with UDP (54.50 KB, patch)
2009-11-18 06:08 EST, Antti Mäkelä
Details | Diff
Updated example, demonstrates both directions (8.42 KB, text/plain)
2009-11-18 06:09 EST, Antti Mäkelä
Details
SO_BINDTODEVICE socket option, now supports UDP and TCP in Ipv4 mode (62.55 KB, patch)
2009-11-18 06:37 EST, Antti Mäkelä
Details | Diff
Fixes TCP issue (62.21 KB, patch)
2009-11-24 06:31 EST, Antti Mäkelä
Details | Diff
Example with SO_BINDTODEVICE, uses TCP. (9.54 KB, text/plain)
2009-11-24 06:35 EST, Antti Mäkelä
Details
updated patch for latest ns-3-dev (116.85 KB, patch)
2009-12-10 02:04 EST, Tom Henderson
Details | Diff

Note You need to log in before you can comment on or make changes to this bug.
Description Antti Mäkelä 2009-11-17 10:08:03 EST
Created attachment 657 [details]
Patch to allow bindsockopts

As per discussion at 

http://groups.google.com/group/ns-3-users/browse_thread/thread/013eb49c1ad77436

Here's first draft version of getting SO_BINDTODEVICE to work. It currently works only on UDP sockets, and only on sending side. 

TODO: Receiving side

TCP sockets
RAW sockets

(?).

However, while patch is not that complicated, it affects lots of files, as it changes the so far pretty unused uint32_t oif in RouteOutput to Ptr<NetDevice> instead - and as a consequence, every inherited routing protocol and all methods that have called RouteOutput in it's previous format. 

For now, Static and Global Routing protocols actually utilize the new oif information. Thus, the provided example works, but right now socket doesn't check if arrived packet came in on right interface. As it stands, the patch shouldn't BREAK anything, it's just that the added functionality isn't yet quite complete...

Since there's a lot of changes throughout the tree (node, internet-stack and routing), I'm hoping that this can be merged relatively quickly so I don't have to keep recreating the diffs against newer snapshots of ns-3-dev...

The beef is in socket.cc where two methods have been added for binding a device and checking the value. You can release the binding by binding to zero. Example illustrates the workings quite well.

CCing Tom due to him participating in the thread on mailing list.
Comment 1 Antti Mäkelä 2009-11-17 10:08:45 EST
Created attachment 658 [details]
Simple example on using device-bound sockets.
Comment 2 Tom Henderson 2009-11-18 01:51:56 EST
(In reply to comment #0)
> Created an attachment (id=657) [details]
> Patch to allow bindsockopts
> 
> As per discussion at 
> 
> http://groups.google.com/group/ns-3-users/browse_thread/thread/013eb49c1ad77436
> 
> Here's first draft version of getting SO_BINDTODEVICE to work. It currently
> works only on UDP sockets, and only on sending side. 
> 
> TODO: Receiving side

do you have any sense of whether the receiving side will be easily achieved (since input processing may not have visibility of this socket option at the IP layer, or the routing protocol processing "RouteInput" may not know)?

> 
> TCP sockets
> RAW sockets
> 
> (?).
> 
> However, while patch is not that complicated, it affects lots of files, as it
> changes the so far pretty unused uint32_t oif in RouteOutput to Ptr<NetDevice>
> instead - and as a consequence, every inherited routing protocol and all
> methods that have called RouteOutput in it's previous format. 

understood

> 
> For now, Static and Global Routing protocols actually utilize the new oif
> information. Thus, the provided example works, but right now socket doesn't
> check if arrived packet came in on right interface. As it stands, the patch
> shouldn't BREAK anything, it's just that the added functionality isn't yet
> quite complete...
> 
> Since there's a lot of changes throughout the tree (node, internet-stack and
> routing), I'm hoping that this can be merged relatively quickly so I don't have
> to keep recreating the diffs against newer snapshots of ns-3-dev...
> 
> The beef is in socket.cc where two methods have been added for binding a device
> and checking the value. You can release the binding by binding to zero. Example
> illustrates the workings quite well.
> 
> CCing Tom due to him participating in the thread on mailing list.

I support adding the function and the new socket API, and would support merging what you have now while leaving the bug open to complete the functionality.

You hit all of the points I was looking for in the review (v6 support, doxygen-- although note that you could add the point about releasing the binding to the doxygen).  I do have the concern about how to support on the inbound direction.

An alternative to changing RouteOutput would be to start assigning ip interface indexes from 1 instead of zero, but I think it is probably better to change RouteOutput to take a NetDevice pointer since protocols in general will need to add support for this (most have ignored this parameter), and it will not cause soft failures for users.
Comment 3 Antti Mäkelä 2009-11-18 06:07:56 EST
Actually, I have...Changes are that ipv4-demuxer has now additional methods, and I inherited the bindtonetif method from socket as a virtual function all the way down to udpsocketimpl (so that the bound if can be put into the endpoint as well). 

So, the patch now uploaded works for UDP in both directions. I also fixed a bug I introduced (NetDevice -> GetIfIndex does NOT correspond to Ipv4/Ipv6interfaceindex). 

TODO: TCP and RAW (and ICMP?).
Comment 4 Antti Mäkelä 2009-11-18 06:08:40 EST
Created attachment 661 [details]
Patch to allow SO_BINDTODEVICE, now works for both directions with UDP
Comment 5 Antti Mäkelä 2009-11-18 06:09:16 EST
Created attachment 662 [details]
Updated example, demonstrates both directions
Comment 6 Antti Mäkelä 2009-11-18 06:11:13 EST
Oh, I should mention - it works for UDP Ipv4-only (!) in both directions. I didn't yet update Ipv6demuxer.
Comment 7 Antti Mäkelä 2009-11-18 06:12:29 EST
(In reply to comment #6)
> Oh, I should mention - it works for UDP Ipv4-only (!) in both directions. I
> didn't yet update Ipv6demuxer.

...Although I guess there's not much to update since one doesn't EXIST...:)
Comment 8 Antti Mäkelä 2009-11-18 06:37:25 EST
Created attachment 663 [details]
SO_BINDTODEVICE socket option, now supports UDP and TCP in Ipv4 mode

Here's TCP functionality included - however, it's completely UNTESTED as of now. Since the socket-impl itself only checks routes during connect(), I had to add support to tcp-l4-protocol's Send() and SendPacket() functions by including an additional Ptr<NetDevice> parameter which I'm defaulting to zero in case something else uses this. 

Also, I haven't tested this yet, but right now it SHOULD mean that you can change your bound interface on the fly with TCP too. 

RAW sockets don't have concept of "endpoints" so not sure how to go there - I simply changed the call in SendTo to RouteOutput reflecting the current status so sending works, but I'm not actually sure how receiving even works with Raw sockets - doesn't seem to go through demuxer and endpoint processing at least.
Comment 9 Antti Mäkelä 2009-11-24 05:49:48 EST
Ok, I have reworked the example to TCP, apparently TCP support doesn't work yet properly - continuing work.
Comment 10 Antti Mäkelä 2009-11-24 06:31:04 EST
Created attachment 671 [details]
Fixes TCP issue

Fixes TCP issue - had a leftover in SendSinglePacket, which screwed up Connect(). Tested with attached example.
Comment 11 Antti Mäkelä 2009-11-24 06:35:08 EST
Created attachment 673 [details]
Example with SO_BINDTODEVICE, uses TCP.

Example of SO_BINDTODEVICE that uses TCP. Not completely analogous to the UDP example (UDP one *always* sends replies to interface 1, TCP naturally goes with the flow).

In theory, you *could* change binding in the middle of a flow. However, I doubt it can have any meaningful application. Same thing is pretty much warned against in Linux sockets documentation, too.

One thing that should be done is that at least packet-sink application helper should be updated to support this option.
Comment 12 Antti Mäkelä 2009-12-02 11:41:47 EST
(In reply to comment #2)
> I support adding the function and the new socket API, and would support merging
> what you have now while leaving the bug open to complete the functionality.

  Any idea when this can be moved forward?
Comment 13 Tom Henderson 2009-12-02 12:49:10 EST
(In reply to comment #12)
> (In reply to comment #2)
> > I support adding the function and the new socket API, and would support merging
> > what you have now while leaving the bug open to complete the functionality.
> 
>   Any idea when this can be moved forward?

Antti, I was letting it sit for now since you were still improving it, and to see whether there are comments on the proposed API change.  I can plan to test and merge what you have by the end of the week if you think it is ready.
Comment 14 Antti Mäkelä 2009-12-02 16:33:32 EST
(In reply to comment #13)
> Antti, I was letting it sit for now since you were still improving it, and to
> see whether there are comments on the proposed API change.  I can plan to test
> and merge what you have by the end of the week if you think it is ready.

  I think it's quite ready - it works with (IPv4) TCP and UDP both sending and receiving, and since there is no IPv6 TCP/UDP sockets currently even present, not much to work with that. On raw sockets it works on sending side, and I'm not even sure what would be the "receiving" in case of raw sockets since there's no endpoint-multiplexing anyway - the validation of incoming if should be done at application level.
Comment 15 Tom Henderson 2009-12-10 02:04:50 EST
Created attachment 695 [details]
updated patch for latest ns-3-dev

fixed a few style nits too
Comment 16 Tom Henderson 2009-12-10 02:24:33 EST
Antti, I planned to check this in, but hesitated mainly because I started poking around on the web and think that it may not be quite correct.

According to:
http://linux.die.net/man/7/socket

SO_BINDTODEVICE applies to the receiving direction only.  It seems to me to have been introduced because Linux is liberal by default in accepting datagrams from any interface to any of its addresses.  However, the patch uses m_boundnetdevice to pass to the routing system for the sending direction, in RouteOutput().  Won't this override Socket::Bind()?  Should it be instead restricted to receive side?

There remains the issue that existing ns-3 doesn't respect bind() in the outgoing direction, and none of our dynamic routing protocols appear to support "oif" parameter in the existing RouteOutput() that takes an integer instead of a Ptr<NetDevice>:
- Ipv4NixVectorRouting
- Aodv
- Olsr

but I wonder whether RouteOutput should take (optionally) the address passed to Bind, instead of a device interface index or pointer.  We should also try to make sure that both Tcp and NscTcp support whatever choice is decided.

Finally, it seems that you have provided examples that are intended as tests-- do you intend that they would be included in src/test or src/examples?  I'd prefer the former if they could be reworked to check for packet outcomes instead of using trace files.

Anyway, the next steps might be:
1) (if you agree with limiting scope of SO_BINDTODEVICE to receive direction) prepare a patch + test for that socket change only
2) Add support (as you suggested) to PacketSinkHelper
3) decide on what should be the RouteOutput behavior with respect to Socket::Bind(), and work towards fixing all of those routing protocols and transport implementations, and add tests similar to your current examples
Comment 17 Antti Mäkelä 2009-12-10 02:39:35 EST
(In reply to comment #16)
> Antti, I planned to check this in, but hesitated mainly because I started
> poking around on the web and think that it may not be quite correct.
> 
> According to:
> http://linux.die.net/man/7/socket
> 
> SO_BINDTODEVICE applies to the receiving direction only.  It seems to me to
> have been introduced because Linux is liberal by default in accepting datagrams
> from any interface to any of its addresses.  However, the patch uses
> m_boundnetdevice to pass to the routing system for the sending direction, in
> RouteOutput().  Won't this override Socket::Bind()?  Should it be instead
> restricted to receive side?

  It applies to both directions. Ref e.g. http://codingrelic.geekhold.com/2009/10/code-snippet-sobindtodevice.html

"SO_BINDTODEVICE forces packets on the socket to only egress the bound interface, regardless of what the IP routing table would normally choose. Similarly only packets which ingress the bound interface will be received on the socket, packets from other interfaces will not be delivered to the socket."

> Finally, it seems that you have provided examples that are intended as tests--
> do you intend that they would be included in src/test or src/examples?  I'd
> prefer the former if they could be reworked to check for packet outcomes
> instead of using trace files.

  They can be used as both. Up to you - I'm not certain about the tautology of tests vs. examples.

> Anyway, the next steps might be:
> 1) (if you agree with limiting scope of SO_BINDTODEVICE to receive direction)
> prepare a patch + test for that socket change only

  I don't. 

> 2) Add support (as you suggested) to PacketSinkHelper

  It requires an option, but should be easy.

> 3) decide on what should be the RouteOutput behavior with respect to
> Socket::Bind(), and work towards fixing all of those routing protocols and
> transport implementations, and add tests similar to your current examples

  Now, this probably can be done by having an old-fashioned RouteOutput (where oif is an integer) simply finding out the NetDevice and calling the new one. Right?
Comment 18 Tom Henderson 2009-12-10 13:54:36 EST
(In reply to comment #17)
> (In reply to comment #16)
> > Antti, I planned to check this in, but hesitated mainly because I started
> > poking around on the web and think that it may not be quite correct.
> > 
> > According to:
> > http://linux.die.net/man/7/socket
> > 
> > SO_BINDTODEVICE applies to the receiving direction only.  It seems to me to
> > have been introduced because Linux is liberal by default in accepting datagrams
> > from any interface to any of its addresses.  However, the patch uses
> > m_boundnetdevice to pass to the routing system for the sending direction, in
> > RouteOutput().  Won't this override Socket::Bind()?  Should it be instead
> > restricted to receive side?
> 
>   It applies to both directions. Ref e.g.
> http://codingrelic.geekhold.com/2009/10/code-snippet-sobindtodevice.html

OK, you convinced me.  The main remaining question that I have is how it should interact with bind().  That webpage says:

"There is no particular interaction between bind() and SO_BINDTODEVICE. It is certainly possible to bind to the IP address of the interface to which one will also SO_BINDTODEVICE, as this will ensure that the packets carry the desired source IP address. It is also permissible, albeit weird, to bind to the IP address of one interface but SO_BINDTODEVICE a different interface. It is unlikely that any ingress packets will carry the proper combination of destination IP address and ingress interface, but for very special use cases it could be done."

> > 3) decide on what should be the RouteOutput behavior with respect to
> > Socket::Bind(), and work towards fixing all of those routing protocols and
> > transport implementations, and add tests similar to your current examples
> 
>   Now, this probably can be done by having an old-fashioned RouteOutput (where
> oif is an integer) simply finding out the NetDevice and calling the new one.
> Right?

Am not sure what you are recommending, but I think you are suggesting that the oif version can be used to support bind(), and it calls into the new one by either using m_boundnetdevice if non-zero or otherwise using the index to fetch the right NetDevice?  It doesn't seem to me that it would cover the case where a specific address is bound, or when both bind() and BINDTODEVICE are selected.

Looking at the Linux solution, it seems that a flowspec is passed to route_output; maybe we need to pass some kind of a new flowspec struct/class to cover all of the cases.
Comment 19 Antti Mäkelä 2009-12-11 09:43:59 EST
(In reply to comment #18)
> Am not sure what you are recommending, but I think you are suggesting that the
> oif version can be used to support bind(), and it calls into the new one by
> either using m_boundnetdevice if non-zero or otherwise using the index to fetch
> the right NetDevice?  It doesn't seem to me that it would cover the case where
> a specific address is bound, or when both bind() and BINDTODEVICE are selected.

  I'm still not sure what the issue with bind is. Like it says, BINDTODEVICE has no relation to bind(). I think it should work as-is.

  Bind() can optionally bind to a specific IP address, which is then used as source (or destination, in listener sockets) for packets. With default of 0.0.0.0, the source address is decided based on routing table when sending, and on listener it means that socket should accepts packets on any IP address that the device has.

  If node has two interfaces, with IP addresses 1.1.1.1 and 2.2.2.2, and routing tables where network 10.10.10.0 has better metric via if 1 and 20.20.20.0 has better metric via if 2:

  If I bind() without any options, and send packet to 10.10.10.1, source address will be 1.1.1.1, outbound if will be 1.
  If I bind() to 2.2.2.2, send packet to 10.10.10.1, source addr will be 2.2.2.2, outbound if will be 1.
  If I bind() without any options, and bind to device 2, and send packet to 10.10.10.1, source address SHOULD be 2.2.2.2 and outbound if 2.
  If I bind() with addr 1.1.1.1 and bind to device 2, and send packet to 10.10.10.1, source address SHOULD be 1.1.1.1 and outbound if 2.

  In the last case, if a response packet comes in via if 1 (where other nodes routing table has it) the packet will not be accepted.

  Now, I haven't tested the last two SHOULD-cases. However, can you point out what exactly is the problem and what causes it? Do the dynamic routing protocols actually use oif information?
Comment 20 Tom Henderson 2009-12-12 03:04:01 EST
OK, I agree with your explanation.  So, I would suggest to 
1) check in the latest patch I attached which is basically your last patch on this (maybe adding a bit more doxygen to include the gist of the last few tracker comments)
2) file a bug for each of these dynamic routing protocols needing support for this option
- Ipv4NixVectorRouting
- Aodv
- Olsr
3) file a separate bug to extend PacketSinkHelper to use this, so not to forget about this point
4) convert the examples into tests

am I missing anything in the above?  If not, I will plan to do the above.
Comment 21 Antti Mäkelä 2009-12-12 03:26:54 EST
(In reply to comment #20)
> OK, I agree with your explanation.  So, I would suggest to 
> 1) check in the latest patch I attached which is basically your last patch on
> this (maybe adding a bit more doxygen to include the gist of the last few
> tracker comments)
> 2) file a bug for each of these dynamic routing protocols needing support for
> this option
> - Ipv4NixVectorRouting
> - Aodv
> - Olsr
> 3) file a separate bug to extend PacketSinkHelper to use this, so not to forget
> about this point
> 4) convert the examples into tests
> 
> am I missing anything in the above?  If not, I will plan to do the above.

  I think this covers it all.
Comment 22 Tom Henderson 2009-12-15 23:43:40 EST
changeset:  7fd20c798a7d