Bugzilla – Bug 1202
Bug in IPv4 L3 Protocol for broadcast
Last modified: 2011-10-21 10:00:48 EDT
I notice there might be a bug in the implementation of ipv4-l3- protocol. The problem surfaces when doing a subnet directed broadcast over multiple interfaces which are assigned with IPv4 addresses under the same subnet. It seems the 3.10 code only finds the first interface with the same subnet, even the socket is bound to a different netdevice and the packet is sent using a different source address. This will cause the broadcast traffic being sent out to wrong network interfaces. The code snippet handling this part in 3.10 is as follows, it seems in 3.11 the bug is still there. Refer to http://groups.google.com/group/ns-3-users/browse_thread/thread/40eee1e7512dbebb?pli=1 for the discussion. src/internet-stack/ipv4-l3-protocol.cc // 2) check: packet is destined to a subnet-directed broadcast address uint32_t ifaceIndex = 0; for (Ipv4InterfaceList::iterator ifaceIter = m_interfaces.begin (); ifaceIter != m_interfaces.end (); ifaceIter++, ifaceIndex++) { Ptr<Ipv4Interface> outInterface = *ifaceIter; for (uint32_t j = 0; j < GetNAddresses (ifaceIndex); j++) { Ipv4InterfaceAddress ifAddr = GetAddress (ifaceIndex, j); NS_LOG_LOGIC ("Testing address " << ifAddr.GetLocal () << " with mask " << ifAddr.GetMask ()); if (destination.IsSubnetDirectedBroadcast (ifAddr.GetMask ()) && destination.CombineMask (ifAddr.GetMask ()) == ifAddr.GetLocal ().CombineMask (ifAddr.GetMask ()) ) { NS_LOG_LOGIC ("Ipv4L3Protocol::Send case 2: subnet directed bcast to " << ifAddr.GetLocal ()); ipHeader = BuildHeader (source, destination, protocol, packet->GetSize (), ttl, mayFragment); Ptr<Packet> packetCopy = packet->Copy (); m_sendOutgoingTrace (ipHeader, packetCopy, ifaceIndex); packetCopy->AddHeader (ipHeader); m_txTrace (packetCopy, m_node->GetObject<Ipv4> (), ifaceIndex); outInterface->Send (packetCopy, destination); return; } } }
I agree with the bug report. Do you think you could provide a patch to fix?
Created attachment 1260 [details] program using subnet directed broadcast 1. To my knowledge , ns-3 does not support multiple interfaces belonging to different subnets attached to the same node. In the real-world such a concept would require l3 loadbalancing to choose the outgoing interface. 2. The attached program mysecond.cc shows the usage of a subnet directed broadcast to send one packet to all hosts in a CSMA LAN (except self).This is working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) 3. I am not sure about the problem with bounded netdevices,,, what is the right behavior? What does Linux/Windows/OSX do? Should we try to find the correct outgoing interface or simply drop the packet if the bounded interface is not what the routing layer think is the correct outgoing interface. It will help if you could attach your test-case to reproduce the issue.
(In reply to comment #2) > Created attachment 1260 [details] > program using subnet directed broadcast > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > different subnets attached to the same node. In the real-world such a concept > would require l3 loadbalancing to choose the outgoing interface. > 2. The attached program mysecond.cc shows the usage of a subnet directed > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > 3. I am not sure about the problem with bounded netdevices,,, what is the right > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > outgoing interface or simply drop the packet if the bounded interface is not > what the routing layer think is the correct outgoing interface. > It will help if you could attach your test-case to reproduce the issue. meant to say multiple interfaces belonging to the same subnet attached to the same node
(In reply to comment #3) > (In reply to comment #2) > > Created attachment 1260 [details] > > program using subnet directed broadcast > > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > > different subnets attached to the same node. In the real-world such a concept > > would require l3 loadbalancing to choose the outgoing interface. > > 2. The attached program mysecond.cc shows the usage of a subnet directed > > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > > 3. I am not sure about the problem with bounded netdevices,,, what is the right > > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > > outgoing interface or simply drop the packet if the bounded interface is not > > what the routing layer think is the correct outgoing interface. > > It will help if you could attach your test-case to reproduce the issue. > > > > meant to say multiple interfaces belonging to the same subnet attached to the > same node Hi, sorry I was quite busy with my work recently and not able to respond. Come back to the bug, having multiple interfaces using the same subnet is common in mobile ad hoc networks where every node participates in the routing. The network is often a flat setup with every network interface falls into the same subnet. Selecting the correct interface is the job of the routing protocol. For sockets bounded to certain address, I expect the packets are ONLY sent out via the netdevice with the same address that the socket is bounded to. I assume such behavior is reasonable although I didn't write code to verify it. I don't have any test case. My team found this bug when we trying to implement a neighbor discovery module for delay tolerant network. When a beacon is broadcasted, it was sent to the wrong link, which was the first link with a matching subnet.
ok, if Node A, has 2 interfaces interface x belonging to subnet S interface y belonging to subnet S 1. A wants to send a packet to B which interface does it use, unless there is a loadbalancing scheme in place, if both have equal cost? 2. Can you point out which mobile adhoc protocol relies on multiple interfaces on the same subnet. So that we know whether we are dealing with a specific or general case. (In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > Created attachment 1260 [details] > > > program using subnet directed broadcast > > > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > > > different subnets attached to the same node. In the real-world such a concept > > > would require l3 loadbalancing to choose the outgoing interface. > > > 2. The attached program mysecond.cc shows the usage of a subnet directed > > > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > > > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > > > 3. I am not sure about the problem with bounded netdevices,,, what is the right > > > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > > > outgoing interface or simply drop the packet if the bounded interface is not > > > what the routing layer think is the correct outgoing interface. > > > It will help if you could attach your test-case to reproduce the issue. > > > > > > > > meant to say multiple interfaces belonging to the same subnet attached to the > > same node > > Hi, sorry I was quite busy with my work recently and not able to respond. > > Come back to the bug, having multiple interfaces using the same subnet is > common in mobile ad hoc networks where every node participates in the routing. > The network is often a flat setup with every network interface falls into the > same subnet. Selecting the correct interface is the job of the routing > protocol. > > For sockets bounded to certain address, I expect the packets are ONLY sent out > via the netdevice with the same address that the socket is bounded to. I assume > such behavior is reasonable although I didn't write code to verify it. > > I don't have any test case. My team found this bug when we trying to implement > a neighbor discovery module for delay tolerant network. When a beacon is > broadcasted, it was sent to the wrong link, which was the first link with a > matching subnet.
Created attachment 1261 [details] Program using socket bind to netdevice Program that binds the socket to a netdevice that does not have a route to the destination. As expected the packets get dropped
Hi, I wanted to summarize 1. mysecond.cc illustrates that subnet directed broadcast is indeed working as expected, when the node has one interface 2. myfifth.cc illustrates that if a socket is bound to a netdevice for which the routing layer does not consider it an outgoing interface for a given destination, the packet is indeed dropped 3. For a subnet directed broadcast ns-3 is indeed picking up only the first interface matching the subnet 4. For the cases where there are multiple interfaces on the same node, connected to the same subnet, the behavior in my opinion is open-ended. For instance Cisco IOS does not permit a subnet from appearing on more than one active interface at any time http://www.cisco.com/en/US/docs/ios/12_2/ip/configuration/guide/1cfipadr.html#wp1001012 Here are some google searches of cases where people have struggled to associate multiple interfaces to the same subnet. In all cases I have seen there is no standardized way of how it should work. Most of it are just hacks. http://www.linuxquestions.org/questions/linux-networking-3/multiple-interfaces-all-traffic-flows-through-just-one-538701/ http://support.apple.com/kb/TS3679?viewlocale=en_US 5. If hypothetically we do allow a subnet directed bcast packet to appear out of 2 interfaces belonging to the same subnet on the same node, that particular node, has in a way contributed two almost identical packets onto the same subnet. Thus unless we have l2 isolation such as VLANs or Wifi channels, we have created a sort of packet duplication. Again, my inclination is that, there is no accepted standard on how to handle this. ns-3 does the obvious choice of sending it out through the first interface that matches the subnet as it is not expected for this node to have multiple interfaces connected to the same subnet 6. If you still consider it to be normal practice for mobile nodes to have multiple interfaces for one subnet, please refer us to the use-case or a popular/standardized routing protocol that relies on this concept. IMHO, a routing protocol , once the FIB is populated, is aimed at choosing exactly one interface to send out a packet, unless it is a limited broadcast (not the same as a subnet directed broadcast).
(In reply to comment #4) > (In reply to comment #3) > > (In reply to comment #2) > > > Created attachment 1260 [details] > > > program using subnet directed broadcast > > > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > > > different subnets attached to the same node. In the real-world such a concept > > > would require l3 loadbalancing to choose the outgoing interface. > > > 2. The attached program mysecond.cc shows the usage of a subnet directed > > > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > > > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > > > 3. I am not sure about the problem with bounded netdevices,,, what is the right > > > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > > > outgoing interface or simply drop the packet if the bounded interface is not > > > what the routing layer think is the correct outgoing interface. > > > It will help if you could attach your test-case to reproduce the issue. > > > > > > > > meant to say multiple interfaces belonging to the same subnet attached to the > > same node > > Hi, sorry I was quite busy with my work recently and not able to respond. > > Come back to the bug, having multiple interfaces using the same subnet is > common in mobile ad hoc networks where every node participates in the routing. > The network is often a flat setup with every network interface falls into the > same subnet. Selecting the correct interface is the job of the routing > protocol. I looked into this further. Please read section 7.1 of the below. http://tools.ietf.org/html/draft-clausen-manet-linktype-00#section-7 This draft is suggesting that the configuration you have is invalid. This is also my experience with MANET (that /32 and /128 addresses should be used on MANET interfaces or else problems can arise). So, I think my original endorsement of this bug use case may have been incorrect: 1) if this is a shared full-mesh link like CSMA, sending on only one of possibly multiple interfaces should still reach all nodes on the link 2) if this is a MANET link, no two interfaces should be configured on the same subnet
(In reply to comment #8) > (In reply to comment #4) > > (In reply to comment #3) > > > (In reply to comment #2) > > > > Created attachment 1260 [details] > > > > program using subnet directed broadcast > > > > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > > > > different subnets attached to the same node. In the real-world such a concept > > > > would require l3 loadbalancing to choose the outgoing interface. > > > > 2. The attached program mysecond.cc shows the usage of a subnet directed > > > > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > > > > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > > > > 3. I am not sure about the problem with bounded netdevices,,, what is the right > > > > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > > > > outgoing interface or simply drop the packet if the bounded interface is not > > > > what the routing layer think is the correct outgoing interface. > > > > It will help if you could attach your test-case to reproduce the issue. > > > > > > > > > > > > meant to say multiple interfaces belonging to the same subnet attached to the > > > same node > > > > Hi, sorry I was quite busy with my work recently and not able to respond. > > > > Come back to the bug, having multiple interfaces using the same subnet is > > common in mobile ad hoc networks where every node participates in the routing. > > The network is often a flat setup with every network interface falls into the > > same subnet. Selecting the correct interface is the job of the routing > > protocol. > > I looked into this further. Please read section 7.1 of the below. > http://tools.ietf.org/html/draft-clausen-manet-linktype-00#section-7 > This draft is suggesting that the configuration you have is invalid. This is > also my experience with MANET (that /32 and /128 addresses should be used on > MANET interfaces or else problems can arise). > > So, I think my original endorsement of this bug use case may have been > incorrect: > 1) if this is a shared full-mesh link like CSMA, sending on only one of > possibly multiple interfaces should still reach all nodes on the link > 2) if this is a MANET link, no two interfaces should be configured on the same > subnet Let me first highlight the user case. I agree with you that in a wired network, one copy will be enough. However, I think this issue mainly originates from the context of wireless networks. In MANET, when nodes have multiple interfaces, using them simultaneously will improve the connectivity and increase the throughput when they are configured not to interfere each other, e.g. using different channel or using directional antenna. Note that in MANET, wireless links are dynamically established depending on nodes relative position. If these interfaces are configured to be of different subnets, it is likely to have two nodes happen to be in range of each other but the interfaces are in different subnet. Then IP routing will not be possible unless a routing entry is added manually which is again not quite possible considering the dynamic nature of MANET. Thus, in MANET, we usually use a flat addressing scheme and make interfaces using the same subnet. If we look at how bind() works, according to "Unix Network Programming vol. 1", performing a bind() on a client socket basically sets the source address for the outgoing packets. I think this implies these packets will leave the host via the interface with the bounded IP
(In reply to comment #8) > (In reply to comment #4) > > (In reply to comment #3) > > > (In reply to comment #2) > > > > Created attachment 1260 [details] > > > > program using subnet directed broadcast > > > > 1. To my knowledge , ns-3 does not support multiple interfaces belonging to > > > > different subnets attached to the same node. In the real-world such a concept > > > > would require l3 loadbalancing to choose the outgoing interface. > > > > 2. The attached program mysecond.cc shows the usage of a subnet directed > > > > broadcast to send one packet to all hosts in a CSMA LAN (except self).This is > > > > working.Node 1 and Node 2 receive the packet (use export NS_LOG=PacketSink) > > > > 3. I am not sure about the problem with bounded netdevices,,, what is the right > > > > behavior? What does Linux/Windows/OSX do? Should we try to find the correct > > > > outgoing interface or simply drop the packet if the bounded interface is not > > > > what the routing layer think is the correct outgoing interface. > > > > It will help if you could attach your test-case to reproduce the issue. > > > > > > > > > > > > meant to say multiple interfaces belonging to the same subnet attached to the > > > same node > > > > Hi, sorry I was quite busy with my work recently and not able to respond. > > > > Come back to the bug, having multiple interfaces using the same subnet is > > common in mobile ad hoc networks where every node participates in the routing. > > The network is often a flat setup with every network interface falls into the > > same subnet. Selecting the correct interface is the job of the routing > > protocol. > > I looked into this further. Please read section 7.1 of the below. > http://tools.ietf.org/html/draft-clausen-manet-linktype-00#section-7 > This draft is suggesting that the configuration you have is invalid. This is > also my experience with MANET (that /32 and /128 addresses should be used on > MANET interfaces or else problems can arise). > > So, I think my original endorsement of this bug use case may have been > incorrect: > 1) if this is a shared full-mesh link like CSMA, sending on only one of > possibly multiple interfaces should still reach all nodes on the link > 2) if this is a MANET link, no two interfaces should be configured on the same > subnet Sorry, please ignore the previous reply as it is incomplete. Let me first highlight the user case. I agree with you that in a wired network, one copy will be enough. However, I think this issue mainly originates from the context of wireless networks. 1. Why we want to use the same subnet? In MANET, when nodes have multiple interfaces, using them simultaneously will improve the connectivity and increase the throughput when they are configured not to interfere each other, e.g. using different channel or using directional antenna. Note that in MANET, wireless links are dynamically established depending on nodes relative position. If these interfaces are configured to be of different subnets, it is likely to have two nodes happen to be in range of each other but the interfaces are in different subnet. Then IP routing will not be possible unless a routing entry is added manually which is again not quite possible considering the dynamic nature of MANET. Thus, in MANET, we usually use a flat addressing scheme and make interfaces using the same subnet. 2. Why we need to send broadcast from two different interfaces? In MANET, we broadcast HELLO packets to show the node's presence. Broadcasting from two different interfaces indicating there are two interfaces on this node participating the routing. OLSR supports routing over multiple interfaces, quoted from http://www.olsr.org/docs/report_html/node34.html, "It is very important to note that HELLO messages are generated on a per interface basis. This is because HELLO messages are used for link sensing, which requires the use of possible non-main-addresses. " We also send out broadcast messages for route query, e.g AODV. AODV supports multiple interfaces too, http://www.ietf.org/rfc/rfc3561.txt, "When multiple interfaces are available, a node retransmitting a RREQ message rebroadcasts that message on all interfaces that have been configured...." 3. Why use bind() to specify netdevice to use? Now if we examine what bind() does, according to "Unix Network Programming vol. 1", performing a bind() on a client socket basically sets the source address for the outgoing packets from this socket. This implies these packets will leave the host via the interface with the specified IP address. I believe this should be the same for unicast and broadcast. Therefore, even there are multiple interface with the same subnet, the binding has already specified which interface to use. I believe the load balancing scheme only kicks in when client sockets are not bound to any address, which is common when sending normal unicast traffic. Naturally, I will expect similar result if I bind a socket to a netdevice. Basically, I think NS3 should give user a choice to select a particular netdevice to use for a certain socket. I think the natural way to do it is to bind it to the desired interface.
I made a wrong statement in my previous reply about why use the same subnet. It is actually routing protocol's job to add the route to the neighbors. So it should work if interfaces are assigned with /32 addresses. I recalled the reason to use the same subnet in our setup is because we were developing DTN module. When two nodes establish a link between them, DTN bundles are supposed to be transfered over using TCP convergence layer using IP routing. When the interfaces are assigned with different subnet, the default route doesn't work. Because we are not using DTN as overlay over IP network, running a MANET routing protocol below DTN is a overkill. Therefore, we decided to make the interfaces in the same subnet and specify the outgoing interface by binding socket to the desired netdevice. And then we find that we cannot define the outgoing interface by just binding it. So based on the Internet draft you referenced, I think you are right, this bug is not valid. In our setup, we should use /32 addresses and let the DTN module add an IP route whenever there is a new IP neighbor discovered. > > I looked into this further. Please read section 7.1 of the below. > http://tools.ietf.org/html/draft-clausen-manet-linktype-00#section-7 > This draft is suggesting that the configuration you have is invalid. This is > also my experience with MANET (that /32 and /128 addresses should be used on > MANET interfaces or else problems can arise). > > So, I think my original endorsement of this bug use case may have been > incorrect: > 1) if this is a shared full-mesh link like CSMA, sending on only one of > possibly multiple interfaces should still reach all nodes on the link > 2) if this is a MANET link, no two interfaces should be configured on the same > subnet