Firewall/NAT

I have a Soekris 4801 running M0n0wall 1.33 built March 16th, 2011.  I have a small web/email hosting operation here and my primary customer traffic is on a different connection and does not run through this box.  I have a DSL line connected to the Soekris with PPPoE running in M0n0wall and using NAT to forward TCP/UDP port 53 to our third DNS server.

Under normal traffic load, it runs fine with no problems and little bandwidth used by DNS.  About 2 weeks ago, I have come under a DoS attack where someone is pushing hundreds(possibly thousands) of packets per minute with a single IP as the source(of course, possibly forged) to UDP port 53.  The NAT statetable seems to fill up and overflow. 

During the first wave, they were using two source ip address. This was causing m0n0wall to reboot.  After finding one of the ip addresses, I blocked it in the firewall and then the traffic from the second ip was enought to cause the PPPoE session to fall over and restart.  My upstream provider was kind enough to block those two source ip addresses and all was well.

Since then we have seen other ip addresses(one at a time) do the same thing in waves lasting 2 to 12 hrs.  M0n0wall becomes sluggish and at times, the PPPoE session falls over until I look in the state table, find the offending source address, block it and all returns to normal even while the attempts continue.

One suggestion by my upstream provider was to see if I can fine tune m0n0wall to lower the timeout for idle UDP sessions to limit the number of statetable entries in place at any one time.  I don't see anything like that in the GUI, so I assume we would have to do something else to make a change like that.

Any suggestions as to what I can do the mitigate these attacks?

BTW, the dns server is not a public resovler, but only an authoritative server for the domains I host here.  So if their intent is an amplification attack, my dns server is not participating.  I don't see any errors in the logs for BIND.  But it's difficult at best to capture packets here so I don't know what the payload of the packets are.

Thanks,
Lyle Giese
LCR Computer Services, Inc.

Under Advanced menu there is an option "TCP idle timeout" here You could limit the TTL for idle TCP connections, however, this is for TCP.

Anyhow, any significant DoS or DDoS attacks that chokes Your link no matter what kind of equipment You have, should be resolved on higher network node level, before the choking has begun - in this case Your ISP.

There are more expensive ISP level systems that could automatically prevent this, however all of that makes You pay money, thus You have monetary damage made successfully by the attacker.

You could go with an expensive solution for a robust DNS system from some of the popular DNS hosting companies, or depending on Your skills You could build one, i'll give You a hint - it's all about math.

It's not choking the link.  Once I block the traffic from NAT, all traffic returns to normal.  I run MRTG and you can not even see the traffic on it's graphs. 

We use this DSL link for normal workstation traffic outbound(www, ssh, remote server connections) and while this traffic is hitting the NAT portion of m0n0wall, web traffic crawls to a halt.  Block in the firewall before it hits NAT, you don't even know it's happening.  You have to look at the logs in our syslog server to see if it stopped or not.

Lyle

Okay, then You could try limiting the queries allowed per second by limiting the rate per DNS client.

First You need go to the traffic shaper menu and to create a pipe like in the pipe.png image, and then you need to add rule with Your NAT-ed server IP like in the rule.png image and then enable traffic shaper.

By limiting the bandwidth rate per client it limits the query rate and thus it limits the state table size.
For changing the UDP TTL values on the state table i still cannot help You.

However You could try this exactly as it is in the images and if it doesn't work or impacts performance too much You could tweak it or disable the traffic shaper and revert back to the previous state.

thanks!  And I even understand most of it.

One question, how do I know that dns traffic is or is not effected?

I know my goal is to stop choking the NAT engine in m0n0wall, but how do I know if this rule kicks in and is doing what I want it to do?  Or if I fat fingered something and it's effecting nothing? (besides waiting for the next wave and find out if it chokes)

Thanks,
Lyle

Let me explain:
(assuming that there are no other traffic shaper rules)
- add pipe with name Pipe1, speed 1Kbit/s and mask "source" (leave the other variables as they are)
- add rule with name Rule1 with target Pipe1, interface WAN, protocol UDP, source any, destination single host - your NATed internal server IP, destination port range DNS and direction any (leave the other variables as they are)
- enable traffic shaper

This rule in general says that any single client could be able to exchange 1024bits, that's 128Bytes, which are two 64Byte DNS packets per one second.

So, if the single client asks more than the allowed DNS queries per second, it must wait for the second to pass, thus limiting the data rate, thus limiting the packet rate, thus limiting the query rate, thus limiting the growing state size table per second.
This should allow normal state flushing with no overflows.

I've tried this and it works okay. for me.

A problem might occur if more than 2 DNS packets per second are required by a legitimate user, so the user would get slightly slower response (keep in mind that recursive server caching and higher DNS zone record TTL are helpful in this situation), however if this is the case raising the limit to for example 2Kbit/s would ease the situation. Keep in mind that if attacked i suggest 1Kbit/s as in the previous example.
Anyhow, if bigger DNS packets are used, a value of 2Kbit/s or 4Kbit/s could be used, but still this leads to quite higher states table overflow possibility.

Note: This is not bulletproof, and then again nothing is, if the attacker obtains large amount of client servers with different IPs (larger DDoS attack) the attacker might still be able to overflow the states table or even worse choke the link. Still not worthless this strategy makes the attacker spend more resources and knowledge to attack, which makes a larger scale attack less feasible for the attacker.

If this solution does not suit Your needs, at any time You could turn off the traffic shaper or/and erase the added rules without causing any damage to Your system.

They were throwing about 40 packets per second at my servers while this was in progress.  I have since found that they targeted the 5 domains that have dnssec enabled(no big surprise there).  They wanted the bigger packets an ANY query would generate against those domains.

I understand the theory behind all of this, but short of a confirmed attack, how do I confirm this is working and limiting traffic as expected?  That's the piece I am missing.

Just like whenever I change firewall rules, I want to see syslog entries that prove I wrote the rules to do what I wanted.  That's the single piece I am missing yet.  I don't see that it is possible either.

With firewall rules, you can send rule hits to your syslog server, therefore reading those entries, I can verify that the rules are doing what I wanted them to do and not blocking legit traffic.  And the answer might be that it just is not possible in m0n0wall.

Thanks again for your help!
Lyle

I've tested it and i think it's okay, You could test it too by test quering the DNS server from outside Your LAN network.

Yes You can log packets by going to the Firewall>Rules>Interface(tab)>Edit Rule>log packets that are handled by this rule (option). [log.png image]

Then when this data is sent to Your log server You could see if someone is constantly trying to query Your server.
Also You could check Your state table size at the time to see the impact on it.