Networking Libraries Benchmark

Users » emilmont » Notebook » Networking Libraries Benchmark

Networking Libraries Benchmark

Page last updated 02 Aug 2012, by Emilio Monti. 11 replies benchmark, Networking Libraries

You may have noticed that recently we have been busy working on a new official networking library:

Unlike the previous library, the new library does not require polling and provides the standard Berkeley sockets API.

The performance improvement with the old library is huge and we provide you with all the sources to repeat the benchmarks by yourself.

As a simple benchmark we are sending and receiving back, through a TCP Echo Server, 1,000 packets of 127 bytes each and we measure how many bits per second we transmit and receive.

Here are the results:

Legacy "EthernetNetIf" library: (0.006) Mbits/s
New mbed official networking library: (5.852) Mbits/s

That's an improvement of: +97433% !

Python benchmark:

import socket
import random, string
from time import time

ECHO_SERVER_ADDRESS = "192.168.0.50"
ECHO_SERVER_PORT = 7

N_PACKETS = 1000
LEN_PACKET = 127
TOT_BITS = (LEN_PACKET * N_PACKETS * 8) * 2. # TX bits + RX bits
PACKET = ''.join(random.choice(string.ascii_uppercase+string.digits) for _ in range(LEN_PACKET))
MEGA = 1024 * 1024.
UPDATE_STEP = (N_PACKETS/10)

s = socket.socket(socket.AF_INET, socket.SOCK_STREAM)
s.connect((ECHO_SERVER_ADDRESS, ECHO_SERVER_PORT))

start = time()
for i in range(N_PACKETS):
    if (i % UPDATE_STEP) == 0: print "%.2f%%" % (float(i)/float(N_PACKETS) * 100.)
    s.sendall(PACKET)
    data = s.recv(LEN_PACKET)
    assert len(data) == LEN_PACKET
t = time() - start

s.close()

print "TX+RX Throughput: (%.3f)Mbits/s" % ((TOT_BITS / t) / MEGA)

Echo Sever based on the legacy libraries:

Import program EchoServer

Echo Server based on the legacy EthernetNetIf libraries used for a performance comparison with the new networking libraries

Echo Server based on the new mbed official libraries:

11 comments on Networking Libraries Benchmark:

little llumpu

# 02 Aug 2012

Hello Emilio, this is huge increase of speed. Thank You mbed team for official mbed networking stack.

I would also like to ask if there are any plans for official mbed USB host/device stack working with CMSIS RTOS? USB device library actually works with CMSIS RTOS but USB host stack not.

Emilio Monti

# 03 Aug 2012

little llumpu wrote:

are any plans for official mbed USB host/device stack working with CMSIS RTOS?

Yes, we will release an official USB Host stack.

Cheers, Emilio

steve childress

# 04 Aug 2012

That's ICMP 1,000 packets of 127 bytes.

I wonder what the Mbps yield would be with TCP and MTU sized packets. The ICMP case, above, has a very small payload which often means the data rate suffers due to the overhead of the ratio of the header sizes and the payload size. Big payload, good.

Emilio Monti

# 04 Aug 2012

Hi Steve,

steve childress wrote:

That's ICMP 1,000 packets of 127 bytes.

I wonder what the Mbps yield would be with TCP and MTU sized packets. The ICMP case, above, has a very small payload which often means the data rate suffers due to the overhead of the ratio of the header sizes and the payload size. Big payload, good.

That's the beauty of a completely open source TCP/IP stack and benchmark.

Everyone can pick them up and provide improvements, or further analysis.

I'd love to see your benchmark/analysis.

Keep us posted, Emilio

Michael Geissmann

# 29 Apr 2013

Hey everyone. Is there an easy way to switch out the the Ethernetinterfaces. I have a Http Server with the old(slow) EthernetIf and want to replace it with the faster one, but it doesn't look so simple as it sounds..

pablo gindel

# 05 Aug 2013

Hi Michael, I made a HTTP server with the new library, but unfortunately it's not faster but slower than that which exists with the old NetServices library. Here it is:

Import program HTTP_server

HTTP Server upon new mbed Ethernet Interface. Based on original code by Henry Leinen.

I did the test shown here, with that python program, and I got a score of 3.648 Mbit/s. Then, from python side, I set LEN_PACKET = 256, and then I got a score of 7.353 Mbit/s. Wow, that's impressive.

I came up then, to do the test in one direction only, from mbed to PC. In the python program, I commented out this line:

# s.sendall (bytes (PACKET, 'UTF-8'))

(which I had previously modified for python 3).

From the mbed side, the loop turned out as follows:

while (true) {
    // int n = client.receive (buffer, sizeof (buffer));
    // if (n <= 0) break;
    int n = client.send_all (buffer, 256);
    if (n <= 0) break;
}

Also, I changed the method for calculating TOTBITS, in python:

TOT_BITS = (LEN_PACKET * N_PACKETS * 8) * 1. # Only RX bits

Then I performed the test, and the result was: 0.008 Mbit/s... which is consistent with the problem that I have in my HTTP server, which for this reason is slower than NetServices.

What am I doing wrong?

pablo gindel

# 06 Aug 2013

I forget to mention: my PC running Windows XP SP3 (32 bit).

I think that this fault is due the so-called "nagle" algorithm, in combination with a "delayed ack" from windows.

But, in the same scenario, the old netServices library don't have this problem.

Adam Green

# 07 Aug 2013

I hit something similar when I wrote a HTTP server a few years ago. It was actually "slow start" along with "delayed ack" that caused me problems. Once you send enough segments through the socket successfully, "slow start" backs off and uses up more of the TCP send window.

For my performance measurement purposes, I just sent more data in my tests to average away this slow start condition. In the real world, I guess implementing "Keep-Alive" in your HTTP server might help. However, the old NetServices implementation of the HTTP server doesn't appear to have supported "Keep-Alive" either so that isn't the major thing that helped you with that version. I wonder if the difference is due to some difference in the lwipopts.h configuration? I do see one thing a bit odd in that file for the new stack. If you look at lines 111-112 and lines 129-130 of http://mbed.org/users/mbed_official/code/lwip/file/42a34d63b218/lwipopts.h you will see that TCP_SND_BUF and TCP_WND are given two different set of values. The first ones will be discarded but are closer to the recommended values for lwIP based on TCP_MSS but the latter ones would be used. You could try commenting out the first set to see if they help. Warning: The code might fail to link if the resulting RAM usage is too high.

Liam Power

# 08 Nov 2013

Hi Pablo,

Did you ever get this issue sorted? I am seeing the same thing here. LwiP is only sending a single segment out onto the network and is awaiting an ACK before sending naymore. With the lwipopts.h settings I have it should output multiple messages onto the entwork in advance of an ACK from the receiver.

Daniele Lacamera

# 08 Nov 2013

@Liam this is definitely a problem of this very benchmark. If you look at the python code on the host:

s.sendall(PACKET) data = s.recv(LEN_PACKET)

It really waits for the packet to come back (blocking inside the app) before sending a new one.

The test published here is NOT a throughput test, but rather a latency test, so it has no scientific value when measuring TCP/IP performance.

For a real throughput measurement you may want to modify the host code so it looks more like the one here:

https://mbed.org/users/tass/code/PicoTCP_Official_TCP_Full_Benchmark/wiki/Homepage

Igino Chaves

# 19 Nov 2014

Hello Emilio,

Does this library use DMA to communicate with the ethernet device?

Please log in to post comments.