Hardware

Guys,

I wanted to see if other people have seen this.  I know that everybody says an 866mhz P3 should be able to do "wire speed" 100Mbps, but I think that is misleading and it heavily depends on your application.

We're doing internet crawling and literally have thousands and thousands of open NAT connections pulling in http data at any given second (via our 6000kbps/768kbps DSL line). 

We're also running a webserver on port 80 that gets a fair amount of upstream traffic.

Now we're also shaping the data via monowall to give horrible priorities to all crawling data, and good priorities to our web server traffic.

And here's what we see .... the monowall "not keeping up".  By that i mean, when i'm running a big internet crawl (obviously taking up all data on the WAN link), and then i try and copy data between two other subnets (all local between our 100Mbps intel cards) I get about 1/2 the speed of traffic I would get when the crawl is NOT running.

So normally, i will get about 11 megabytes/sec on local transfers, but when the crawls are running, that drops down to 7 or so, almost half.  There are other repercussions we see as well, but in general, they all seem to be related to **when the big internet crawl is running**.

So it seems to me that when monowall is both traffic shaping for thousands of simultaneous connections, that even on a "slow" DSL line, it can bog it down. 

Or am I crazy?  Other option is that something is mis-configured but i'm not seeing what it could possibly be.  My solution in the meantime is to throw hardware at this problem and see what happens.

w

Edit - changed original title

Oops, the title should read 6000kbps down, not 600!  That would've deserved a few more exclamation points

w

normally, i will get about 11 megabytes/sec on local transfers, but when the crawls are running, that drops down to 7 or so, almost half.

Have you checked mono's cpu usage at this point? If your cpu is running 100% while "bogged", then you'll know that's your bottleneck, otherwise you'll have to look at pci bus saturation or some other culprit.

db

what m0n0wall version are you using?

Is CPU pegged at 100% when you're seeing this?

Can you get actual measured results?

Thanks for the comments guys -

We run monowall v1.22

CPU is at about 8-10%, but don't get excited.  Because the server clearly slows down.  I mean really .... intranet transfers (again, that aren't competing for WAN bandwdith) still slow significantly.  But i'm too lazy to get the stats on that again right now (4AM).

But, i do have some stats on another slowdown in monowall that is killing me.  In doing this testing, i've seen the ability of monowall to service DNS lookups drop through the floor.  So when no www crawling is going on (see first post, as in, thousands and thousands of hosts that we crawl) LOCAL dns lookups (as in, dns entries that are entered into monowall, NOT domains that monowall has to query a real DNS to service) take about .040 seconds - aka, 40ms - pretty quick.

Now when the crawl is running, those same DNS lookups take .7 seconds, aka 700ms.  Anybody seen this?  We realized that was a huge part of our problem in servicing *incoming* www server requests in a timely fashion .... because our backend servers do "intra server" communication, each one of those doing a dns lookup.  And when those lookups take .7 seconds, we are hosed, big time.

I've now gone ahead and hardcoded the IP's for most of these intra server backend requests, so users don't have to wait on that lag, but the moral of the story is: why does Monowall's DNS server grind to a halt at 10% cpu for local DNS lookups??

w