General Questions

Considering the number of questions we see on the forum and the list, I think it would be handy to look here and see that Jon Då got 30 mbps on his pII with 3com nics and Phil Dexter got wire speed on his Athlon 64  with Intel Pro 1000s using device polling and so forth.

When I considered building my first m0n0wall my biggest unanswered questions were whether it could support the number of users/amount of traffic I needed and how much hardware I would require. The FAQ at the end of the manual is helpful, but most of us know that your mileage will vary and a benchmarks forum would be a great way to compare scenarios.

So I propose the following criteria be included, where available:

CPU name and speed
RAM amount
NIC names, drivers and link speeds
Interface configuration, ie, LAN=NAT, OPT1=bridged, etc.
Polling mode on or off
Implementation, ie, "in the wild" versus "on the bench" (iperf/wget/etc.)
Max transfer speed between two interfaces, simplex/duplex
CPU usage during max transfer speed
Relevant user comments/observations ("14 ipsec tunnels active", "lots of p2p traffic", etc.)

db

edited to add various test configurations and for brevity

I'll go first.

All tests

m0n0 version 1.3b3
CPU -- Sempron 2800+ 1.6 GHz
RAM -- 2*256 MB dual channel
Polling mode off
iperf client and server direct connected to LAN and OPT1 respectively (LAN and WAN in test config 4), on separate subnets
Test -- "iperf -c 192.168.3.109 -d -i 30 -t 300" (full duplex tcp test for 300 seconds between LAN and OPT1)

test config 1

LAN = Intel Pro 1000 GT pci
OPT1 = Intel Pro 1000 GT pci
Sustained speed -- combined 370 Mbps
CPU load -- ~45%

The hard limit of approximately 370 Mbps is presumably due to limitations on the pci bus, as both interfaces share it. cpu usage on iperf client and server was well below 100%.

test config 2

LAN = Intel Pro 1000 GT pci
OPT1 = nvidia nforce 430B MCP onboard GBE (ASUS M2NBP-VM CSM motherboard)
Sustained speed -- combined 840 Mbps
CPU load -- ~91%

As expected, the speed more than doubled in this config using a pci-onboard connection versus the pci-pci connection above.

test config 3

LAN = Intel Pro 1000 PT pci-express
OPT1 = nvidia nforce 430B MCP onboard GBE
Sustained speed -- combined ~1100 Mbps, less stable than before
CPU load -- ~97%, also less stable

Again, by eliminating the pci bus altogether we see more throughput. The monowall's cpu usage rarely dropped below 90% on this test and it was obviously operating near capacity. It's notable though that iperf was using 90%+ cpu on one of the test machines according to top. I don't know if this was a limitation or not, as it is a dual-core and top doesn't tell me if it's using both cores. If so, then the theoretical cpu limit is 200% rather than 100%. If on the other hand iperf is only using a single core, then this limitation could explain the fluctuations seen in throughput and mono's cpu usage during this test.

test config 4

LAN = nvidia nforce 430B MCP onboard GBE
WAN = Intel Pro 1000 PT pci-express
Sustained speed -- combined 1080 Mbps
CPU lad -- 95 - 100%

Introducing NAT into the test showed little to no difference from the otherwise identical test config 3. That surprised me, but then what do I know?

I hope the results to these somewhat arbitrary tests will provide some insight for prospective monowall builders as to approximate hardware sizing requirements.

So let's see some of y'all's figures.

db

I'll go second...

Soekris net5501-70 500Mhz 512M RAM
vr0 - LAN - untagged (VIA VT6105M 10/100)
vr1 - WAN - untagged (VIA VT6105M 10/100)
em0 - multiple subnets - tagged (Intel Pro 1000 GT PCI)

m0n0 version 1.3B4
polling mode off
tested with iperf 1.7.0

For all tests, WAN connected, but unused
vr0 connected to HP Procure 1800-8G switch "A"
em0 connected to another 1800-8G switch "B" configured with VLANs

Test 1:
Mac Pro, OS X, Gig-Ethernet connected to switch "A"
Mac PowerBook, OS X, Gig-Ethernet connected to switch "B"

Traffic Shaping Off:
Sustained speed: 84 Mbps for each interface
CPU load: 75%

Traffic Shaping On: WAN rules only
Sustained speed: 84 Mbps for each interface
CPU load: 90%

Test 2:
Mac Pro, OS X, Gig-Ethernet connected to switch "B", different vlan/subnet
Mac PowerBook, OS X, Gig-Ethernet connected to switch "B", different vlan/subnet

Traffic Shaping Off:
Sustained speed: 242 Mbps for each interface
CPU load: 100%

Traffic Shaping On: WAN rules only
Sustained speed: 173 Mbps for each interface
CPU load: 100%

Through all tests, m0n0wall remained solid, no interface errors.

Note: m0n0wall 1.231 had read errors on vr0 when stressed, (possibly USB IRQ conflict with vr0, sorry I didn't have a chance to investigate more).

Lonnie

It's a good idea in theory, but in practice, varying testing environments and methodologies mean you can't necessarily compare one person's results to another.

But with that said, I still think it'd be good to see. Ideally, a standard means of testing should be defined and followed for all posted numbers.

NAT is indeed no slower than routing with filtering, which is a bit strange since there is a bit more overhead involved in NAT. This is something specific to ipf/ipnat, pf does slow down a bit with NAT, though it's minimal.

I have a ton of performance data from numerous tests that I don't have time to clean up and post today, but I'll post back here later.

m0n0 version 1.3b4
CPU -- Pentium II / 350 MHz
RAM -- 64 MB
Polling mode off
LAN -- fxp0 compaq-branded Intel 100 pci
OPT1 -- dc0 smc-branded Linksys Network Everywhere 100 pci
WAN disconnected
iperf client and server direct connected to LAN and OPT1 respectively, bridged

Test -- "iperf -c 192.168.3.109 -d -i 30 -t 150" (full duplex tcp test for 150 seconds between LAN and OPT1)
iperf server -- Athlon X2 4200+ nvidia GBE on m0n0's OPT1 (dc0)
iperf client -- Sempron 3000+ mobile Realtek 100 on m0n0's LAN (fxp0)

speed LAN -> OPT1: 93.8 mbps steady
           OPT1 -> LAN: 6 - 10 mbps, varying between trials
CPU usage: 38% average not exceeding 50%

I'm impressed with the speed of mono on this particular hardware setup. I chose it for its quiet and presumably low-power operation (no cpu fan). I'm a little confused about the huge imbalance in transfer speed on the full duplex test though.

db

m0n0wall version 1.23 on CF
running on Intel board with 900MHz Celeron and 256MB RAM
(3)Intel PRO/1000 XT server NIC's

Client and server machines are both dual-core Wintel, also with PRO/1000 XT NIC's.

iperf -c <server_ip> -t 60 -l 24M -F VTS_01_1.VOB -f m

The above test gives 295 to 300 Mbits/s with m0n0wall's CPU running at around 80%. Transfers a 1GB VOB file in about 28 seconds.

Client and server running both CPU's at about 25-30%.

Would I see a performance increase if I put a dual-head NIC (I've got one ready) in the m0n0wall to handle the LAN and DMZ interfaces?

I've tried setting up the client and server NIC's to use Jumbo Frames, but it makes VNC crash so I turned them off. Anybody know of a workaround for getting VNC to work with jumbo frames (I use v. 1.2.9)?

Glad to be able to share this geekery with all of you:-)

Update 11/8/2007:
I turned on device polling and the speed results are the same but monowall's cpu usage went down to next to nothing. Seems like the way to go.

Would I see a performance increase if I put a dual-head NIC (I've got one ready) in the m0n0wall to handle the LAN and DMZ interfaces?

I can't see this making a difference in your case. As far as I can tell (intel.com is a little hard to decipher sometimes) the 1000 XT is not a pci card. Any dual-head NIC I've seen uses pci-express, thus eliminating the pci bottleneck, but you've probably already done that and performance on non-pci intel 1000 cards would be similar, I expect. You could always let us know if you discover otherwise.

db