Icon Rufen Sie uns an
+49 441.309197-69 +49 441.309197-69
 
EN

OpenBSD Network Performance Tests on Soekris and Liantec Hardware

Posted by Markus Müller on Monday, August 04, 2008

Abstract

Finally, after waiting for more than a year, we got couple of Liantec EMB-5740 prototypes to play with. Since we’ve been selling Soekris systems for years now and lots of customers ask us about performance data, sometimes in comparison to other, similar, hardware solutions, we sat down and compared the Liantec systems to the Soekris 5501-70. This ended as an IPsec performance comparison between the both.

Description

In this test we wanted to determine how much IPSec traffic the Soekris and the Liantec can handle. For comparison, we also made the tests without IPSec.

Test case

The test setup consisted of two workstations (about 1.5Ghz with 512MB memory each) connected via two routers. The first test was done with two Soekris 5501-70, then with the two Liantec boxes. The workstations were running OpenBSD 4.3 default.

To test the throughput of the boxes, we used a tool called iperf. We could have used tcpbench, but that is only available in OpenBSD current (becomes 4.4) and we also wanted to compare our results to 4.3. On server side the iperf was running with “iperf -s -B 10.1.2.2” and on the client “iperf -c 10.1.2.2 -s1 -t100”. “-s1” makes iperf output every second and “-t100” limits the test to 100 seconds. For IPSec we copied the keys from /etc/isakmpd to the other machine.

Technical comparison

The next table is a small comparison of the technical specification of both boxes:

  Soekris Liantec
CPU 500Mhz (AMD Geode LX single chip processor with CS5536 companion chip) 1000Mhz (VIA Embedded C7-Eden Platform with VIA PadLock Security Engine)
Ram 512MB 512MB
Usb 2 4
VGA 0 1
Ethernet 4×100Mbit 4×1Gbit
RS232 2 2
Heat cold, even under load After some hours very hot
Form factor Slim and small, 19" cases available from Kerberos Fat and heavy; better do not stack because of the heat
Stackable yes preferably not (see above)
Power consumption 8-12 W 25 W (with 2 network cables)
Price 266 EUR (33 EUR for case and power supply) 398 EUR

Dmesg for the Liantec EMB-5740 and Soekris 5501-70

Kernels

All tests were made with three different kernels. Two were default 4.3 and current (4.4) kernels, the other one was a current kernel without pf. We also did a fourth test with some changed buffer sizes, which we used a current kernel for.

Buffers

Following a FAQ advice, we also testet some different buffer sizes to increase network performance, i.e. we set sendspace and recvspace to higher values.

  • net.inet.tcp.recvspace=1638465536
  • net.inet.tcp.sendspace=1638465536

Results

Throughput without IPSec

The following chart is just for comparison, to see what throughput is possible without IPSec. There is just one current (4.4) test as it was not possible to put the Liantecs at their limit with our test-setup, so tuning things would have been useless.

As you can see, in this test case the Liantec box is at least twice as fast as the Soekris. But the results are not really repesentative, because the Soekris was at its limit whereas the Liantec was hanging at about 40% of load. That comes from the fact that the two (older)
workstations were not fast enough to fill a Gbit link with packages to put the Liantec at its limit.

Throughput with IPSec

Quite obvious in this chart is, that the Liantec is about twice as fast as the Soekris and that there is just a litte difference between the kernels (see also the next chart). What troubled us is, that while the Soekris has become faster from 4.3 to current, the Liantec box
slowed down a bit.

Throughput increase (with IPSec)

This chart compares the increase of throughput from the Liantec box to the Soekris. It uses the same data as the graph before, just a different layout. As the chart shows, there is just a small increase in throughput by disabling features like pf and tuning some buffers.

Load

While testing we also compared the load of the two boxes, with the following results (everything with IPSec):

4.3:

  Soekris Liantec
System 96% 97%
Interrupt 2.5% 2%
Idle 1.0% 0.7%
Crypto 96% 96%
Throughput (IPSec) 9.57Mbit 20.1Mbit

current (4.4):

  Soekris Liantec
System 91% 92%
Interrupt 6% 4.5%
Idle 1.5% 2.5%
Crypto 96% 93.5%
Throughput (IPSec) 9.57Mbit 20.4Mbit

The test results are based on the meridian of 5 top(1) graphs.

What is interesting about these comparions is the fact, that the interrupt load has increased from 4.3 to current while system and crypto load decreased. Another remarkable thing is, that there is no scope in throughput with the Soekris, so it is at its limits. On the other
hand it seems to be possible to squeeze some mbits out of the Liantec with some smart tuning.

Heat

As mentioned before, the Liantec box became really hot while testing.

$ sysctl -a | grep hw.sensors.lm1.temp
hw.sensors.lm1.temp0=63.00 degC
hw.sensors.lm1.temp1=62.50 degC

Compared to a Soekris:

test@soekris1# sysctl -a | grep hw.sensors.nsclpcsio0.temp
hw.sensors.nsclpcsio0.temp0=124.00 degC (Remote)
hw.sensors.nsclpcsio0.temp1=127.00 degC (Remote)
hw.sensors.nsclpcsio0.temp2=56.00 degC (Local)

test@soekris2$ sysctl -a | grep hw.sensors.nsclpcsio0.temp
hw.sensors.nsclpcsio0.temp0=113.00 degC (Remote)
hw.sensors.nsclpcsio0.temp1=127.00 degC (Remote)
hw.sensors.nsclpcsio0.temp2=51.00 degC (Local)

It is obvious that the results of the Soekris are not correct, so either the Soekris sensor is broken or the OpenBSD driver has bugs.

Problems occured

While testing the Soekris and Liantec boxes some problems occured. First of all, broken Soekris boards suck. One of the test boards was not able to warm-reboot, which can be really annoying while rebooting different kernels.
A problem with the Liantec boxes is, that they are way too hot. You have to put them in a cold place and may not stack them, otherwise they will probably die because of their heat. Additionally putting a cf-card into the Liantec is really annoying. You have to disassemble the whole box, which means you have to screw off the entire case, the cpu heat sink and the mainboard. When you are finished, you can easily put the cf-card into the slot on the bottom of the mainboard and assemble the whole box again. Besides that, the test case could have
been better. For IPSec and Soekris testing the workstations are fast enough, but for Gbit throughput tests they are not.

What to do next

It would be interesting to test some different buffers and sizes (not just net.inet.tcp.{send,recv}space=65536), e.g. net.inet.tcp.mssdflt or net.inet.tcp.sendpsace=32768. Another interesting test would be to further customise the kernel, e.g. throw IPSec out and see how fast it will be. When there is time it also would be nice to test seperate crypto acceleators like the vpn1401 from Soekris engineering, but from what we have heard, they are worthless.

Summary

Coming to the conclusion of our tests, what do we gain? The tests showed that it is a better choice to use a generic kernel with default settings instead of a crippled one, because this way you only have a performance lost by a few mbit/s but much less maintainance work.
As mentioned earlier, the Liantec box is at least about twice as fast as the Soekris, but also more expensive. Which one you will buy depends on your needs, both have major advances and disadvances.