Well, perhaps last week I could have. It turns out that it was quite happily performing as an amplifier. And there was me thinking I understood this stuff. Another Decidedly Negative Scenario in terms of my network administrative abilities. But at least I've learnt some more things I didn't know that I didn't know. What follows is a gentle stroll through the intricacies of DNS and firewall management I encountered.
It seems that DNS recursive amplification attacks are back again (if they ever went away). I first noticed the constant blinking of the router lights last week and decided, even though everything seemed to be working fine, to investigate. I checked that I had all the latest updates and patches installed, so that's not the problem. And the Web server (a Windows Server 2008 Hyper-V virtual machine) has Windows Advanced Firewall and IP filtering enabled to allow inbound packets only to the Web server and the DNS server. After a little digging, by selectively disabling the firewall "allow" rules and watching the network connection status display, I discovered it was regular long bursts of requests over UDP on port 53 to the DNS server on my public Web/DNS server machine. But why? Who could be that interested in my DNS entries?
It didn't take long to realize that they were requests for a list of name servers configured in the DNS. This is, of course, the whole point of the attack. The attacker sends a small payload and gets back a ton of data. And the clever bit in terms of the attack is that the return address for the DNS lookup is spoofed, so the response from my DNS server is not sent back to the attacker, but is instead sent to someone else's machine as my free contribution to a DDoS attack on that sever. So what to do about it? My colleague and I run public DNS servers to host the entries for about twenty local Web site and blog domains that we manage (acting as primary and secondary for each other's domains). So I can't just disable DNS.
I'd assumed that just blocking packets inbound from the supposed sender would made no difference - the packets are not actually coming from that address. In fact, they're probably coming from lots of different addesses in some botnet. And, in addition, the attacked addresses are likely to change regularly. But the firewall also sees the spoofed address, and so blocking these addresses does work. While this is not the ideal way to solve the problem, at least it stops some of the flow over the Internet and removes me from the list of innocent attackers. You can add multiple remote IP addresses to a blocking rule, which makes managing the changing list of attack addresses easier. Perhaps, in real life, I should spend a few thousand dollars on a hardware firewall that detects and automatically blocks these kinds of attacks. It could soak up some more expensive electricity and further raise the ambient temperature inside my server cabinet.
I threw together a simple Windows Forms application that monitors the DNS log for recursive name server queries and notifies me so I can quickly detect new ones. You can download it here.
Another core issue is that a public DNS server should not allow public recursive address lookups. It should only resolve public IP address lookups for domains for which it is authoritative (in other words, public domain names that you manage), and should not be referenced from any machines on your internal network. They should have their own DNS server that does recursive lookups to your ISP, or they should send requests directly to your ISP if you don't have an internal DNS server. Even the DNS server machine itself should use the ISPs DNS server to do its own lookups. If you do have recursion enabled (it seems that 70% of DNS servers out there do), your DNS server will even go off and look up the IP addresses of all the name servers and return them - further adding to the load on your connection and the Internet as a whole. So, open the Advanced tab of the Properties dialog for the DNS server itself (in DNS Manager) and make sure that the Disable recursion setting is ticked. This also disables any forwarders defined in the Forwarders tab.
If yours wasn't ticked before, you'll probably now find that a browser and other applications on the server can't get to any external sites. This is because you accepted the defaults when you originally set up the server (i.e. no DNS server address assigned in the network properties dialog), which causes Windows to use the local DNS server (127.0.0.1). Without recursion and forwarders, your DNS server can only do lookups for the domains it hosts. You'll need to replace the 127.0.0.1 address in the Advanced | DNS tab of the Network Properties dialog for the IPv4 and IPv6 protocols with your ISP's DNS server addresses. Applications and services running on the Web/DNS server will now query the DNS servers that are configured in the network properties dialog, and will not use the local DNS server.
After you get everything working again, you'll probably see that an NSLOOKUP from another network into the DNS server returns a list of root server addresses. These are hints to the requesting server on where to go to find the address it is looking for (because your DNS server can no longer do a recursive lookup). And if you look in the DNS log at the contents of the packets returned by the DNS server, you'll see that they're still quite large. So I started wondering - do I actually need root hints in the DNS server if all it's ever going to do is respond to requests for authoritative domains that it hosts? It's never going to need to know where www.someothersite.com actually is, or tell someone else how to find it.
So I decided to experiment with root hints. If I delete or loose them all, I know I can get them back by copying the cache.dns file installed in the %System Files%\dns\backup folder. So I deleted them and then used the Copy from Server button in the Root Hints tab to load them from one of my ISPs servers. And they were completely different from my original set. So I wandered across to IANA to get the latest list - which is remarkably different from that in my ISP's DNS! And, digging deeper, they seem to have different lists in each of their DNS servers. No wonder DNS lookups are slow sometimes...
Anyway, getting back to the issue in hand, do I really need root hints? As a test, I removed all of them from my public DNS server, and removed the forwarders as well. Now a request for one of the authoritative domains still works fine, as do transfers to the secondary DNS server, but recursive requests prompt a very small return payload containing a "Server Fail" code. It's hard to tell from all the blogs and guidance I've read on DNS good practice if this is an approved approach but, to be honest, nobody should be querying my DNS server for non-authoritative domains anyway. So it's their own fault. Meanwhile my contribution to the DDoS attacks is very significantly reduced. Though now I get a Warning entry in Windows Event Log telling me that there are no forwarders or root hints each time the DNS service starts ... but I'm ignoring these.
Having now got my hands dirty, I thought it would be a good idea to see if I could harden the server a little more. Such as applying outbound filtering in the firewall. I do it in ISA Server on my private internal network, but its easy there because there are lots of appropriately pre-configured rules you can apply. Windows Firewall with Advanced Security (WFWAS) seems to have lots of predefined rules, but not for useful things like allowing Web browsing and sending email. Yet I must have read fifty articles on the Web looking for details of appropriate rule properties, and all I could find were articles and blog posts that said things like "Select the predefined SMTP Server rule ". Err, where is that? Only in ISA Server I suspect.
Mind you, it's not hard to use the New Outbound Rule Wizard create a Custom rule that allows Web browsing and services on the machine (such as Windows Update) to work. Basically it's any service/user using TCP through any outbound port to any remote server on ports 80 and 443 (and maybe 8080 or similar as well if you need to access sites you know are running on this or a non-default port. Visual Studio Team Foundation Server is a typical example). Make sure you create it in the Outbound Rules page, and be sure to specify the Public Profile, not Private or Domain. If you look at the Monitoring page in the WFWAS console, you should see that the Public Profile is active. If not, immediately dive into Network and Sharing Center, click "Customize", and set the connection type to Public!
To be more strict with Web access, you could configure the rule to allow only specific services or applications - but that's much more complicated. For example, what about your "alternative" Web browser, or Java Update Service, or the updates service for Adobe Reader? And remember Windows Update... As a mitigating factor, you should be running your browser in Enhanced Security mode on the server anyway, and only using it when absolutely necessary.
Allowing DNS queries from your server, and responses to other servers from your DNS server, out through the firewall is easier because there are preconfigured rules for these. I found that I needed to enable "All Outgoing (TCP)" and "All Outgoing (UDP)" in the "DNS Service" group, and "Core Networking - DNS (UDP Out)". Again, make sure you select the ones for the Public Profile, not Private or Domain. Finally, if you run an SMTP email server, you need to allow packets from this to escape out onto the Internet. I use the IIS 6.0 SMTP Service that is part of Windows Server 2008, carefully configured to prevent relaying. And as it does not receive email (the Reply To for all messages is my usual Web site administrator email address), it cannot be accessed from the Internet anyway because port 25 inbound is closed.
So, a simple Custom rule that allows only the SMTP server application (inetinfo.exe) to go out using TCP through any local port to only port 25 on remote servers should do it. Then flip over to the Properties dialog for the server itself (the root entry) in WFWAS Manager and set the Outbound connections drop-down to Block (the default is Allow) so that the only permitted outbound traffic is that defined in your enabled outbound rules.
Or so I thought. Being not a little naïve, I expected the SMTP service to use the local DNS Client (not the local DNS Server) to do the lookups required for delivering mail. I mean, everything else seems to use this - but not the SMTP Service. It obviously does its own lookups. Maybe this is something to do with last year's patch that changed the behavior of the SMTP Service and stopped the instance on my internal network from relaying essential email status messages - when it was quite happily doing so before. So, anyway, what you need is another rule that allows inetinfo.exe to go out using UDP through any local port to only port 53 on remote servers. And now (at least temporarily) everything started working properly again.
Of course, a couple of hours later I discovered the error messages in Windows Event Log telling me that the Time Service was broken. Ahh.. forgot that one, so add another Custom outbound rule to allow just the Windows Time Service (click the Customize button next to Services in the wizard) to use UDP on any local port to connect to any remote server on port 123. Want to check that you can ping the time servers you use? That's when you'll discover that PING doesn't work any more. And neither does NSLOOKUP or TRACERT. To allow pings out of your server, you can simply enable the preconfigured rule "File and Print Sharing (Echo Request ICMPv4 Out)" - a nice snappy name, though enabling file and print sharing sounds scary. But if you examine the rule (click Customize in the Protocols and Ports tab), you'll see it only allows ICMP Echo requests to escape.
Meanwhile, for TRACERT and NSLOOKUP, you need another Custom outbound rule that allows any service\user to use UDP through any local port to port 53 on any remote server. I imagined that the predefined rule "Core Networking - DNS (UDP Out)" would allow this, but it is limited to the svchost.exe program and so is no help for the other stuff like DOS utilities. If you want to use TRACERT and NSLOOKUP, and you create the rule for them, you can disable the "Core Networking - DNS (UDP Out)" rule. You can also remove the custom rule you created to allow DNS lookups for the SMTP service, as it also uses UDP to remote port 53. However, it's a good idea to leave it in place so the service will still work if you later decide to block TRACERT and NSLOOKUP.
Except, in my case, NSLOOKUP still wouldn't work - all I got was "UnKnown" (note the interesting letter case) for the DNS server name, and an IPv6 address instead of the usual xxx.xxx.xxx.xxx format one I was expecting. Typing "nslookup", then "set d2", and then a domain name produced the interesting response that the request was too long. In the end, my "large hammer" solution was to simply disable the IPv6 protocol in the properties for the network adapter and normal service was resumed. I guess this is something I'll need to come back and look at again.
What I can't help wondering, though, is when we'll finally solve the problems with spoofing that are already so widespread with email, and are obviously becoming just as common with DNS. I can (and do) use Sender Policy Framework (SPF) to advertise a list of valid IP addresses for email I send, but how do I do the same for DNS? And would it make any difference...?