Bugzilla – Bug 2966
TCP seems memory thirsty
Last modified: 2018-08-20 16:17:43 EDT
I have some simulations to test performance of a clusters of nodes (<280 nodes).
My TcpTxBuffer & TcpRxBuffer are sized at 4MB each. The TCP RX buffer will always be cleaned up by my applications whenever the TCP notifies my application. My applications only send when TCP tells the application that it has enough TX buffer space. The total buffer in the WHOLE system simulated is ~1.6GB (size for packet sizes accounting). So there are theoretically at most ~1M packets (1.5KB each) in the system simultaneously.
As an example, in a simulation of ~180M packets over ~1.5seconds simulation time, the LSF summary said the peak memory usage is ~200GB. Since there are theoretically at most ~1M packets in the system simultaneously, I felt the memory consumption is way too much.
Please help in relieving this huge memory consumption problem.
Some packets may be dropped in the system due to congestion, and cause re-transmission.
The total TCP byte-in-flight (returned by TcpSocket ByteInFlight) at peak time is ~0.9GB for the WHOLE system.
Is there a smaller scale test case program that you can share (post to this bug report) that demonstrates the problem?
Without measurements, it is impossible to say that the cause is TCP. It may be a tag (packet or byte tag).
Anyway, for your use case, the use of ns-3 seems out of scope. The network stack is mainly for education purpose. You may want to consider the use of a real stack (DCE with Linux could be an option).
Status changes as a feature request.
For each packet, 1 packet tag & 1 byte tag is added.
Would that be the problem?
The Packet Tag will be lost at the first step (in the TCP code), but its metadata will remain in the packet, carrying in all the path a ghost tag.
The byte tag will be duplicated exponentially each time the TCP performs a split (imagine when you have a smaller segment size than the application packet) or a join (retransmitted packets, or merge cases).
Try to see what is the difference with and without using the tags (Packet and Byte) and report the result: that is an important benchmark.
Packet tag clarification: the packet tag is added under TCP layer. So the TCP TX end point won't see it at all.
Byte tag: my application layer needs the byte tag to work.
But I did see the TCP splitting codes replicating the ByteTagList, etc. So I did experiment on the memory allocated for PacketTagList & ByteTagList (& Buffer & meta data). The "net" memory allocated are less then 1GB throughout the simulation. However, I do not know whether the memory returned by delete/free is properly managed. (The total amount of memory EVER allocated is hundreds of GB.)
(In reply to Chih-Yuan Chang from comment #7)
> Packet tag clarification: the packet tag is added under TCP layer. So the
> TCP TX end point won't see it at all.
OK, but the cost is still there for each packet (and the Metadata is propagated even when the tag is removed)
> Byte tag: my application layer needs the byte tag to work.
So it's badly crafted. You should try to avoid byte tags,put the information as payload under the firm of an header
> But I did see the TCP splitting codes replicating the ByteTagList, etc. So I
> did experiment on the memory allocated for PacketTagList & ByteTagList (&
> Buffer & meta data). The "net" memory allocated are less then 1GB throughout
> the simulation. However, I do not know whether the memory returned by
> delete/free is properly managed. (The total amount of memory EVER allocated
> is hundreds of GB.)
Are you saying that without tags the memory consumption is inline to what theoretically we could expect?
Anyway, many tests on packets seems to confirm that there aren't leaks, and tcp does use the packet api to perform operations on the packets,nothing custom.