cipherdyne.org

Michael Rash, Security Researcher



Intrusion Detection and iptables    [Summary View]

TCP Options and Detection of Masscan Port Scans

After Errata Security scanned port 22 across the entire Internet earlier this month, I thought I would go back through my iptables logs to see how the scan appeared against one of my systems. Errata Security published the IP they used for the scan as 71.6.151.167 so that it is easy to differentiate their scan from all of the other scans and probes:
[minastirith]# grep 71.6.151.167 /var/log/syslog | grep "DPT=22 "
Sep 12 21:19:15 minastirith kernel: [555953.034807] DROP IN=eth0 OUT= MAC=00:13:46:11:11:11:78:cd:11:6b:11:7e:11:00 SRC=71.6.151.167 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x20 TTL=241 ID=17466 PROTO=TCP SPT=61000 DPT=22 WINDOW=0 RES=0x00 SYN URGP=0
Interestingly, the SYN packet that produced the log message above does not contain TCP options. The LOG rule in the iptables policy was built with the --log-tcp-options switch, and yet the OPT field for TCP options is not included. Looking through the Masscan sources, TCP SYN packets are created with the tcp_create_packet() function which does not appear to include code to set TCP options, and neither does the default template used for describing TCP packets. This is most likely done in order to maximize performance - not from the perspective of the sender since a static hard-coded TLV encoded buffer would have done nicely - but rather to minimize the time that scanned TCP stacks must spend processing the incoming SYN packets before a response is made. While this processing time is trivial for individual TCP connections, it would start to become substantial when trying to rapidly scan the entire IPv4 address space.

A consequence of this strategy is that SYN packets produced by Masscan look different on the wire from SYN packets produced by most operating systems (at least according to p0f), and they also differ from SYN scans produced by Nmap (which do include options as we'll see below). This is not to say that every SYN packet without options necessarily comes from Masscan. There are operating systems in the p0f signature set (such as Ultrix-4.4) that do not include options, and the Scapy project also seems not to set options by default when producing SYN scans like this. In addition it looks like Zmap also does not include TCP options in SYN scans. For reference, here are three iptables LOG messages for SYN packets produced by a standard TCP connect() call from an Ubuntu 12.04 system, an Nmap SYN (-sS) scan, and Scapy (source and destination IP's and MAC addresses have been obscured):
### TCP connect() SYN:
Sep 29 21:16:00 minastirith kernel: [171470.436701] DROP IN=eth0 OUT= MAC=00:13:46:11:11:11:78:cd:11:6b:11:7e:11:00 SRC=2.2.2.2 DST=1.2.3.4 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=15593 DF PROTO=TCP SPT=58884 DPT=12345 WINDOW=14600 RES=0x00 SYN URGP=0 OPT (020405B40402080A0CE97C070000000001030306)

### Nmap SYN (-sS):
Sep 29 21:16:12 minastirith kernel: [171482.094163] DROP IN=eth0 OUT= MAC=00:13:46:11:11:11:78:cd:11:6b:11:7e:11:00 SRC=2.2.2.2 DST=1.2.3.4 LEN=44 TOS=0x00 PREC=0x00 TTL=39 ID=26480 PROTO=TCP SPT=48271 DPT=12345 WINDOW=4096 RES=0x00 SYN URGP=0 OPT (020405B4)

### Scapy SYN via: sr1(IP(dst="1.2.3.4")/TCP(dport=12345,flags="S"))
Sep 29 21:35:15 minastirith kernel: [172625.207745] DROP IN=eth0 OUT= MAC=00:13:46:11:11:11:78:cd:11:6b:11:7e:11:00 SRC=2.2.2.2 DST=1.2.3.4 LEN=40 TOS=0x00 PREC=0x00 TTL=64 ID=1 PROTO=TCP SPT=20 DPT=12345 WINDOW=8192 RES=0x00 SYN URGP=0
As a result, we can infer that SYN scans without options may originate from Masscan (the Scapy example not withstanding since Scapy's usage as a port scanner is a probably a lot less common than Masscan usage), and this will become more likely true over time as Masscan's popularity increases. (It is included as a tool leveraged by Rapid7's Sonar project for example.)

The upcoming 2.2.2 release of psad will include detection of scans that originate from Masscan, and if you check out the latest psad sources from git, this feature has already been added. To make this work, the iptables LOG rule needs to be instantiated with the --log-tcp-options switch, and a new psad.conf configuration variable "EXPECT_TCP_OPTIONS" has been added to assist. When looking for Masscan SYN scans, psad requires at least one TCP options field to be populated within a LOG message (so that it knows --log-tcp-options has been set for at least some logged traffic), and after seeing this then subsequent SYN packets with no options are attributed to Masscan traffic. All usual psad threshold variables continue to apply however, so (by default) a single Masscan SYN packet will not trigger a psad alert. Masscan detection can be disabled altogether by setting EXPECT_TCP_OPTIONS to "N", and this will not affect any other psad detection techniques such as passive OS fingerprinting, etc.

Design of a New 'xbits' Cross-Stream IDS Keyword

In the previous blog post a proposal was made for a new Snort and Suricata keyword "xbits" for cross-stream signature matching. This post had little discussion of implementation tradeoffs, and some have reacted to the blog post by saying that it is difficult to properly design and implement the type of cross-stream state tracking that would be necessary for xbits to work. I agree. However, the initial xbits proposal did not assume that such an implementation would be easy or straightforward, and this blog post will attempt to illustrate where some of the pitfalls are likely to be. In the end, I'm confident that is possible to develop something similar to xbits. Victor Julien, lead developer of Suricata, commented on my Google+ posting on the xbits keyword to state that Suricata has been considering implementing something similar to xbits for a while.

First, before diving into xbits itself, I would argue that there is already precedent in IDS/IPS engines for detecting an important class of communications that cross multiple transport layer "conversations" (using this term loosely for a moment): port scans and sweeps. Detecting such traffic is mostly about setting thresholds on various things such as the number of ports and IP addresses that are contacted within a given period of time, prioritizing on sets of ports that are usually associated with important services (with sweeps for certain ports sometimes spiking after a new vulnerability is discovered), and differentiating TCP flags that are used (a TCP FIN scan looks a lot different than a connect() scan and indicates some things about the adversary such as privileged OS access). By definition, the raw ability to differentiate scans and sweeps vs. normal traffic requires the capability of keeping some state across transport layer conversations. It just so happens that Snort and Suricata track this state within dedicated preprocessors and do not also expose port scan detection configuration in the signature language itself. By contrast, other preprocessors do offer signature language interfaces such as stream5 with the flow keyword. To be clear, I'm not at all advocating that configuration aspects of the sfPortscan preprocessor actually belong in the signature language (that would be unnatural to say the least) - I'm merely making the point that the idea of maintaining some state across transport layer conversations is something that Snort and Suricata already do. So, in this area at least, such a concept is not foreign.

Traffic Visibility

Now, in terms of factors that would affect an xbits implementation, it should be noted that not every IDS/IPS necessarily has a global view of all traffic on a given network. This can be the result of several different factors that depend not only on the physical hardware, but also how the IDS/IPS itself is developed (multi-threaded or not), and configured. Starting with the hardware, I've seen major IDS deployments that require multiple IDS appliances working together in order to inspect all of the network traffic. Essentially the IDS appliances form a cluster of systems (not in the HPC sense) where portions of the traffic are split across each appliance with a device such as a Gigamon tap. This allows each IDS appliance to handle a fraction of the traffic that it would otherwise have had to inspect, and this in turn enables the cluster as a whole to scale to massive amounts of network traffic. But, this also means portions of the traffic are physically separated from one IDS to the next, and therefore xbits on a single IDS can only apply to the set of transport layer conversations that actually traverse it. So, is there an opportunity for an attacker to evade xbits when deployed in this fashion? Sure, if the attacker knows how the traffic splitting is done, then attacks could most likely be sent against systems in ways that nullify xbits criteria just by using different source networks to force each individual IDS appliance into having only a limited view of a cross-stream attack. There are potential evasions at every level, but many attackers are not going to have access to such traffic splitting details.

Other IDS/IPS architectures such as those that rely on network processors or specialized packet acquisition hardware can also result in limited traffic visibility. Some organizations run multiple Snort processes on a single appliance, and have a network processor split packet data based on IP network ranges or transport layer port number across the Snort instances. Once again, each Snort process has a limited few of the traffic. Similarly, in a multi-threaded IDS/IPS such as Suricata, each thread may be tasked with processing a portion of the total network traffic, and this can result in a limited view within software even if there is no hardware enforced traffic splitting mechanism.

The Stream Preprocessor

Modifying the stream preprocessor to handle the xbits keyword could be tricky. By its nature, xbits would force the stream preprocessor to consider information that is not derived from single transport layer conversations, so locking issues against a global xbits tracking data structure in a multi-threaded context would become important. Also, it would be nice to not place onerous restrictions on xbits such as requiring that a connection close before a set xbit can be tested within a different connection. My guess is that stream preprocessor modifications have previously been a barrier to implementing something like xbits.

xbits Design

Given all of the above, what would be the ideal xbits design? Stepping back for a moment, when any signature language feature is implemented in an IDS, what should be the primary goal? Better attack detection. Performance is certainly a consideration too, and performance features sometimes bleed into the signature language (see the fast_pattern keyword for example), but usually a new signature language feature is added because it enables better detection of threats at acceptable performance levels. Further, some features of the language are important enough to expend lots of CPU cycles and consume precious memory anyway because the detection accuracy would be significantly harmed without them - see the pcre keyword for example. So, we should strive for the ideal xbits design from a detection perspective and let performance and other tradeoffs take place where they must:

  1. Allow an xbit to be set on one transport layer conversation and inspected in a different conversation before the first is closed.
  2. Allow an xbit to be set on a conversation that involves one IP protocol, and tested in a conversation that involves a different IP protocol. E.g. set a bit on a UDP flow and test it in a subsequent TCP connection.
  3. Interface with the current stream preprocessor to allow the setting and testing of xbits to take advantage of existing connection tracking capabilities. This most likely can be implemented as an extension to stream5 without requiring a wholly new preprocessor.
  4. For multi-threaded intrusion detection engines such as Suricata, some of the same tradeoffs that allow port scan detection to apply across threads could be used for xbits. Ideally, xbits would not be limited to traffic that is seen within a single thread.

Crossing the Streams in IDS Signature Languages

This blog post is a proposal for a new SNORT®/Suricata keyword "xbits" that could change how IDS signature developers approach detection of exploits that cross multiple streams. Today, in both Snort and Suricata, it is possible to build up a state machine out of a set of related signatures with the flowbits keyword in order to track how an exploit progresses through a vulnerable application layer protocol. This is an important feature, and is used in many standard Snort signature sets - as of this writing it is in about 6% of all active Snort rules in the Emerging Threats rule set. However, flowbits has an important limitation: it can only apply within single TCP connections or single UDP conversations (forward and reverse flows). So, it is not possible to set a flowbit on one TCP connection and then test whether this flowbit is set in a completely separate connection. This limitation represents an opportunity for innovation, and it is my belief that this shortcoming partially helped to fuel the demand for products offered by SIEM vendors (more on this below).

flowbits Example

First, let's see an example of flowbits usage within two Snort signatures - this comes from the online Snort manual. The following Snort signatures (or "rules" if you like) show a basic example of tracking the IMAP protocol and generate an event for the IMAP "LIST" command but only after a successful login:
alert tcp any 143 -> any any (msg:"IMAP login"; content:"OK LOGIN"; flowbits:set,logged_in; flowbits:noalert;)
alert tcp any any -> any 143 (msg:"IMAP LIST"; content:"LIST"; flowbits:isset,logged_in;)
So, the first signature sets the flowbit "logged_in", and then the second tests whether this flowbit has been set on the same TCP connection along with the "LIST" content match. Note the "noalert" modifier suppresses the alert from the first rule and triggering it merely signals internally to Snort that a necessary precondition for the second signature has been met. It is the second signature that causes an alert to be generated if it is triggered. Note also that the "logged_in" flowbit is set on traffic returned to the client from the IMAP server vs. the "isset" criteria which tests the logged_in flowbit in the second rule on traffic coming from the client. This illustrates the ability of flowbits to place match criteria on communications emanating from both sides of a connection.

Why is flowbits useful?

flowbits is important because it allows the signature developer to let network traffic progress in a natural way and have Snort/Suricata track it as it unfolds. It lets sets of signatures work together as group for more reliable exploit detection, and there are plenty of exploits that can be detected using the current flowbits implementation. However, what types of exploits are missed because flowbits cannot apply across multiple TCP connections? Put another way, would detection of malicious traffic be significantly improved with a cross-stream flowbits keyword?

A New Keyword: xbits

The proposed xbits keyword would function as follows:

  • Fundamentally, an xbit could be set on one TCP connection or UDP conversation, and tested in a different connection or conversation. This would require a different interface to the stream tracking portions of Snort and Suricata than currently implemented by flowbits.
  • All xbits semantics would match those in flowbits for existing flowbits modifiers such as set, unset, noalert, etc.
  • xbits would offer a new modifier "track" that accepts arguments "ip_pair" (to associate xbits by pairs of IP addresses), and "expire" (to allow xbits to be cleared automatically after a specified number of seconds).
This blog post is not to say that existing Snort rules that use flowbits don't work properly, or that flowbits is fundamentally flawed. Rather, in some cases these rules could be made more reliable and harder to evade with the proposed xbits keyword. It is the sequence of connections that look a certain way that is important, and xbits tries to promote this concept directly into the Snort signature language. Without xbits, the information conveyed by such sequences is lost. One could imagine other cases where having xbits would be useful such as:

  1. Use of a compromised system after successful exploitation. An attacker sends an exploit against a system where the exploit itself is difficult to detect, but following the exploit connection a new successful connection is made to a backdoor port that the exploit forces the compromised system to open (or a connect-back shell can be initiated the other way - the detection rules can be written to take either scenario into account). If the exploit itself can only be described by a signature that may also produce unacceptably high rates of false positives on legitimate traffic, then xbits provides an alternative since this rule never has to trigger any alert at all. Only the use of the compromised system after successful exploitation causes an alert.

    Now, why not just have a signature that triggers on every connection to the backdoor port? Well, sure, but if xbits existed then a set of higher confidence signatures related together by xbits can be created for the same thing. Once again, the sequence of communications contains information that is important for better exploit detection. Further, nothing prohibits both strategies from being used simultaneously; existing non-xbits signatures can be used at the same time.

  2. Better detection of Metasploit traffic, and by extension better detection of sophisticated adversaries. Here are a few examples of Metasploit modules that require multiple streams for successful exploitation:

                - SCADA 7-Technologies IGSS Rename Overflow
                - Apache ISAPI DoS
                - ContentKeeper Web Remote Code Execution

    There are many more modules that require multiple streams, and here is a quick way to identify those that may fall into this category (requires additional investigation). We just look for calls to connect(), connect_udp(), and send_request_cgi():
    $ git grep -c "^[[:space:]]*connect" modules | grep -v ":1" | wc -l
    125
    $ git grep -c "send_request_cgi" modules | grep -v ":1" | wc -l
    154
    

  3. Detection of network trickery that by its nature requires multiple streams. How about detecting SSH connections that have been authenticated with Single Packet Authorization? In this case, one needs a way to trigger an alert if a base64-encoded blob of data goes to UDP port 62201 followed closely after this by an SSH connection to the same system. Note that there are many ways SPA can deployed where this technique would not be effective (port randomization, SPA packet spoofing, sending SPA packets over Tor, and more), but still it is useful to consider how xbits could be applied to detect styles of communications that can't easily be expressed with current Snort/Suricata signature languages.

Metasploit Example

Let's examine the ContentKeeper Web remote code execution exploit mentioned above in a little more depth, and show how xbits can offer a detection alternative. This exploit attempts to escalate privilege and execute code as either the Apache or root user on the webserver as follows:

  1. Check for a vulnerable version of ContentKeeper by looking for a '500 Internal' error in response to issuing an HTTP request to /cgi-bin/ck/mimencode as seen here. This is an optional step (part of the Metasploit check() function for this exploit), but is generally a good idea since the Metasploit user probably does not want to upload a payload to a patched version of ContentKeeper (and thereby needlessly expand their own risk).
  2. Upload a base64-encoded perl script payload via an HTTP POST to the ContentKeeper webserver. Due to the vulnerability, the payload overwrites a specified file in the webserver filesystem. This connection is made to /cgi-bin/ck/mimencode as seen here.
  3. Wait three seconds after the HTTP upload connection is closed.
  4. Connect to the webserver via an HTTP GET and execute the uploaded payload script via /cgi-bin/ck/<script>. This step can be seen here.
In the context of this post, the important thing to note about the exploit steps above is that two separate HTTP connections are required - one to upload the payload via an HTTP POST, and the second to execute the payload via an HTTP GET. These two requests are required for exploitation regardless of whether the reconnaissance check is made in step one, though in our example rule set below we'll assume the recon check is issued as well. With the xbits keyword, the following signatures can detect this pattern of communications:

  1. Set xbit "Metasploit.ContentKeeper.recon" on initial HTTP connection in Metasploit step 1 above. Track by ip_pair.
  2. Test "Metasploit.ContentKeeper.recon" xbit with 'isset' and if it matches, then set xbit "Metasploit.ContentKeeper.recon_status_is_vuln" on '500 Internal' webserver response. Track by ip_pair.
  3. Look for an HTTP POST that uploads the base64 encoded perl script and test "Metasploit.ContentKeeper.recon_status_is_vuln" xbit. If this xbit is set, then set xbit "Metasploit.ContentKeeper.payload_uploaded" and track by ip_pair.
  4. Look for an HTTP GET to /cgi-bin/ck/ and test the "Metasploit.ContentKeeper.payload_uploaded" xbit. If it is set then generate an event "Metasploit ContentKeeper Web remote code exec".
The complete example rule set can be downloaded here. Obviously these won't run properly in Snort or Suricata until xbits is actually implemented. In this rule set a number of trade offs have been made such as: looking for the recon '500 Internal' error check, tracking by IP pair for the recon check and the exploit step, tracking by IP pair between the payload upload connection and the exploit connection, and allowing xbit values to expire after 30 seconds (see the xbits "track" criteria). The attacker is always free to slow things down and not use the same IP pair across each of these connections and still gain successful code execution. But, the attacker may not utilize different source IP's across these connections, and if not then the pattern above becomes a highly reliable indicator of a successful attack. Further, as mentioned previously, individual non-xbits rules can be deployed at the same time to catch each step by itself as well.

The SIEM Connection

At its core, xbits allows the IDS engine to process network traffic such that associations are made among groups of signatures in a manner that is not restricted to single TCP connections or UDP conversations. There is much analogous precedent for this concept in the SIEM world. SIEM vendors commonly allow security alerts to be built up from multiple independent sources of information such as syslog data, firewall logs, IDS events, webserver logs, netflow data, and more. A good example is "send an alert if source IP a.b.c.d triggers a port sweep event in my IDS, followed by SQL errors via my webserver, followed by a connection initiated back from the webserver to IP a.b.c.d". Having the SIEM automatically trigger an alert based on all of these events coming together in the specified order is far more valuable than trying to arrive at the same result through manual interpretation of each indicator by itself (which isn't practically possible for any decently large network).

This doesn't mean the raw event data is turned off or thrown away - far from it. The event data is simply processed by the SIEM in a way that gives priority to the sequence of events without necessarily caring about the sources of the events themselves (firewall log, etc.). Making an analogy to Snort/Suricata, the absence of xbits would be like a SIEM that could only generate an alert based on looking at one event source at a time. I.e., SSH brute force attempts logged via syslog could not be correlated with, say, a port sweep event from an IDS.

Further, in the absence of xbits, in the Metasploit example above a set of regular Snort rules could be deployed to detect each stage of the Metasploit traffic individually and log this data to a SIEM. From there, the SIEM itself could implement the same logic as xbits does - i.e. generate an alert if the raw events come together in the same sequence as what the xbits rules require. Given that this style of matching is useful, why not implement such a capability within the Snort signature language directly?

Performance?

The implementation of xbits would certainly impact performance, and this may be one reason neither Snort nor Suricata have developed something similar. However, some portion of xbits is clearly possible to implement with a reasonable set of constraints. These constraints may include limitations on the maximum number of xbits, enforcing a maximum amount of time that a set xbit is allowed to remain set, or limiting the number of comparisons that a signature is allowed to trigger for checking whether an xbit is set. It is my belief that a workable performance level could be achieved for many deployments of Snort/Suricata.

What About Shared Object (SO) Rules?

The whole point of binary shared object (SO) rules is to allow the signature author to alter the detection capabilities offered in the standard Snort signature language. However, implementing an SO rule is significantly more complex than just writing a regular Snort rule for most signature authors. If the concept of xbits could improve the detection landscape, then it would be desirable to have it built directly into the signature language itself where it is more accessible and readily applied to real world network traffic.

Why Not Just Extend flowbits?

Instead of creating a new Snort/Suricata signature language keyword, why not just extend flowbits so that it can apply to multiple transport layer conversations? Because maintaining backwards compatibility with existing Snort rule sets would become difficult and error prone. Matching across mutiple conversations is such a fundamentally different model that it would be better to clearly differentiate this capability with a new keyword.

Conclusion

This blog post proposes an extension to the Snort/Suricata signature language to allow those IDS engines to detect malicious traffic that crosses multiple streams. Attackers do not limit themselves to traffic patterns that must stay within a single transport layer conversation, and neither should standard intrusion detection engines. One only needs to look at the Metasploit project for excellent examples of exploits that span multiple streams. For the record, I had proposed this idea on a panel discussion with Ron Gula and Marty Roesch at the SANS "What Works in Incident Detection and Log Management Summit" in 2010. At the time I didn't use the "xbits" name, but (to me) writing up this blog post has solidified xbits as a decent name for the concept.

Software Release - psad-2.2.1

psad-2.2.1 released The 2.2.1 release of psad is available for download. This release adds new support for detecting a type of scan that psad has been previously blind to - IP protocol scanning (nmap -sO). Also in this release is the ability to detect Topera IPv6 scans and differentiate them from Nmap IPv6 scans. Other important changes include a new test suite, email throttling, and auto blocking timeouts that can be set for each danger level. Here is the complete ChangeLog:

  • Added IP protocol scan detection (nmap -sO). A new psad.conf variable PROTOCOL_SCAN_THRESHOLD defines the minimum number of different IP protocols (default = 5) that must be scanned before an alert is triggered.
  • Added detection for Topera IPv6 scans when --log-ip-options is used in the ip6tables logging rule. When this option is not used, the previous psad-2.2 release detected Topera scans. An example TCP SYN packet generated by Topera when --log-ip-options is used looks like this (note the series of empty IP options strings "OPT ( )":
    Dec 20 20:10:40 rohan kernel: [  488.495776] DROP IN=eth0 OUT=
    MAC=00:1b:b9:76:9c:e4:00:13:46:3a:41:36:86:dd
    SRC=2012:1234:1234:0000:0000:0000:0000:0001
    DST=2012:1234:1234:0000:0000:0000:0000:0002 LEN=132 TC=0 HOPLIMIT=64
    FLOWLBL=0 OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( ) OPT ( )
    OPT ( ) OPT ( ) PROTO=TCP SPT=61287 DPT=1 WINDOW=8192 RES=0x00 SYN
    URGP=0
    
  • Bug fix in --Analyze mode when IP fields are to be searched with the --analysis-fields argument (such as --analysis-fields "SRC:1.2.3.4"). The bug was reported by Gregorio Narvaez, and looked like this:
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Use of uninitialized value $_[0] in length at
    ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/hasbits.al) line 126.
    Bad argument length for NetAddr::IP::UtilPP::hasbits, is 0, should be
    128 at ../../blib/lib/NetAddr/IP/UtilPP.pm (autosplit into
    ../../blib/lib/auto/NetAddr/IP/UtilPP/_deadlen.al) line 122.
    
  • Added --stdin argument to allow psad to collect iptables log data from STDIN in --Analyze mode. This makes it easier to run an iptables logs through psad from arbitrary files like so:
    # grep "IN=.*OUT=" /var/log/kern.log | psad -A --stdin
    
  • Added the ability to acquire Snort rule 'msg' fields from fwsnort if it's also installed. A new variable FWSNORT_RULES_DIR tells psad where to look for the fwsnort rule set. This fixes a problem reported by Pui Edylie to the psad mailing list where fwsnort logged an attack that psad could not map back to a descriptive 'msg' field.
  • Added the ability to set per-danger level timeouts when psad is configured to run in auto-blocking mode. These timeouts are implemented with new AUTO_BLOCK_DL*_TIMEOUT variables - one for each of the five possible danger levels that may be assigned to a scanning IP address.
  • Added the ability to throttle emails generated by psad via a new EMAIL_THROTTLE variable which is implemented as a per-IP threshold. That is, if EMAIL_THROTTLE is set to "10", then psad will only send 1/10th as many emails for each scanning IP as it would have normally.
The complete psad-2.2.1 ChangeLog can also be found here via the psad gitweb interface.

Software Release - fwsnort-1.6.3

fwsnort-1.6.3 released The 1.6.3 release of fwsnort is available for download. This release adds a new test suite in the test/ directory that sends fwsnort through its paces for both iptables and ip6tables firewalls, speeds up iptables/ip6tables capabilities testing, and fixes a few bugs. In addition, one of the more significant changes is to ensure that Snort rules with HOME_NET=any -> EXTERNAL_NET=any are placed into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis. I would also like to thank Franck Joncourt for his support on the Debian side. Other changes were contributed by the open source community, and these are acknowledged in the complete fwsnort-1.6.3 ChangeLog below:

  • Bug fix to ensure that !, <, >, and = chars in content strings are converted to the appropriate hex equivalents. All content strings with characters outside of [A-Za-z0-9] are now converted to hex-string format in their entirety. This should also fix an issue that results in the following error when running /var/lib/fwsnort/fwsnort.sh:
        Using intrapositioned negation (`--option ! this`) is deprecated in
        favor of extrapositioned (`! --option this`).  Bad argument `bm'
        Error occurred at line: 64
        Try `iptables-restore -h' or 'iptables-restore --help' for more information.
        Done.
    
  • Bug fix to set default max string length in --no-ipt-test mode where iptables capabilities are not tested.
  • (Andrew Merenbach) Bug fix to properly honor --exclude-regex filtering option.
  • Added fwsnort test suite to the test/ directory. This mimics the test suites from the psad and fwknop projects, and it designed to examine many of the run time results of fwsnort.
  • Added the ability to easily revert the fwsnort policy back to the original iptables policy with "/var/lib/fwsnort/fwsnort.sh -r". Note that this reverts back to the policy as it was when fwsnort itself was executed.
  • Implemented a single unified function for iptables match parameter length testing, and optimized to drastically reduce run time for iptables capabilities checks (going from over 20 seconds to less than one second in some cases).
  • (Dwight Davis) Contributed patches for several bugs including not handling --exclude-regex properly, not ignoring the deleted.rules file, not handling --strict mode operations correctly, and more. These issues and the corresponding patch were originally reported here: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=693000
  • Bug fix for Snort rules with HOME_NET(any) -> EXTERNAL_NET(any) to ensure they go into the OUTPUT chain instead of the INPUT chain. This bug was reported by Dwight Davis.
  • Updated to bundle the latest Emerging Threats rule set.
The complete fwsnort-1.6.3 ChangeLog can also be found here via the fwsnort gitweb interface.

iptables Script Update - Logging and IPv6 Issues

ipv6 traffic vs. iptables Recently, Bobby Krupczak, a reader of "Linux Firewalls" pointed out to me that the iptables script used in the book does not log traffic over the loopback interface, and that such traffic is also blocked because of the INPUT and OUTPUT policies of "DROP" (instead of having a separate DROP rule). This should be made more clear in the book. Quite right - all logging is excluded for traffic that is sent or received over the loopback interface, and the iptables policy also drops loopback traffic because no ACCEPT rule exists. The lack of a logging rule is mostly because logging traffic generated locally and restricted to the loopback interface is usually a distraction from logging more important (and potentially malicious) traffic from remote networks. However, if a local process seems to have connectivity issues, then making sure that loopback traffic flows unimpeded is important. The iptables.sh script has been updated to ACCEPT all loopback traffic handled by the INPUT and OUTPUT chains.

On another note, I would also like to mention that the script has been updated to block IPv6 traffic altogether. With more networks routing IPv6 these days, and with things like Federal mandates for IPv6 compliance on Federal networks, IPv6 adoption is... still slow. However, Linux has had the ability to speak IPv6 for a long time, and Nmap can scan for IPv6-enabled services. Hence it is important to apply iptables controls to IPv6 traffic via ip6tables. The consequences of not doing this could be a system compromise via a service that can communicate over IPv6, but that is normally firewalled off in the IPv4 world.

Here is an example of scanning ::1 on an Ubuntu-9.04 system with Nmap without any ip6tables controls applied. Note that three important services are available over IPv6: [root@isengard ~]# nmap -6 -P0 ::1

Starting Nmap 5.00 ( http://nmap.org ) at 2009-07-28 21:10 EDT
Interesting ports on ip6-localhost (::1):
Not shown: 997 closed ports
PORT STATE SERVICE
22/tcp open ssh
53/tcp open domain
80/tcp open http

Nmap done: 1 IP address (1 host up) scanned in 0.11 seconds

With the updated iptables script deployed, Nmap no longer sees these services.

Have you checked the output of ip6tables -v -nL | grep DROP lately on your Linux system? If you are running a different operating system, are you confident that IPv6 traffic is being filtered appropriately? [root@isengard ~]# ip6tables -v -nL | grep DROP
Chain INPUT (policy DROP 0 packets, 0 bytes)
Chain FORWARD (policy DROP 0 packets, 0 bytes)
Chain OUTPUT (policy DROP 0 packets, 0 bytes)

Disrupting Conficker Worm Traffic with iptables and fwsnort

fwsnort vs. Conficker Although the media blitz surrounding the Conficker worm has died down, the worldwide computing infrastructure that the worm has cobbled together still exists and remains under the control of its masters. The resulting botnet is an impressive demonstration of distributed computing control and recoverability. Many organizations - from companies to governments - would be envious of such automation. Most likely the botnet is being used as a money making machine by renting out "botnet time" to criminals who then use it for their own purposes. New Scientist has a good summary of the Conficker saga, and includes a discussion of its switch from HTTP to a peer-to-peer module for communications and updates. Even though Conficker has perhaps not yet been used to issue DoS attacks against high profile sites, it has had measurable impacts such as leaving Manchester unable to issue parking tickets and Microsoft announcing a $250,000 bounty on the Conficker authors. On the defense side, the Conficker Working Group has produced some nice infection distribution maps and Nmap added Conficker scan detection based on an excellent paper written by Tillmann Werner and Felix Leder.

In the context of iptables and fwsnort, the goal is to give Linux systems the ability to detect and interfere network traffic associated with Conficker (at least as much as possible), and this process starts with Snort rules from the Emerging Threats rule set. There are currently six active Snort rules designed to detect Conficker in the Emerging Threats set, and an additional four that have been commented out. The six active rules so far detect the Conficker.A and Conficker.B variants, but hopefully more rules will become available as better detection techniques are developed.

Now, how does fwsnort do with translating the six Emerging Threats rules? Let's find out with the command below. This uses the --include-regex feature to restrict fwsnort to just those rules that contain the string "conficker", and we also add the new --include-perl-trigger argument (to be released in fwsnort-1.0.7) that builds a perl command to mimic the application layer data in each Snort rule. By combining this perl command with netcat, it is possible to test whether the iptables policy built by fwsnort properly detects attacks. Finally, we also use the --ipt-reject argument to have iptables drop any packet that matches the Conficker signatures and reset the connection at the same time: # fwsnort --include-regex conficker --include-re-caseless --snort-rfile /etc/fwsnort/snort_rules/emerging-all.rules --include-perl-triggers --ipt-reject | tail -n 4
[+] Generated iptables rules for 3 out of 6 signatures: 50.00%

[+] Logfile: /var/log/fwsnort.log
[+] iptables script: /etc/fwsnort/fwsnort.sh
Ok, so three out of the six signatures (I'm using 'signature' and 'rule' interchangeably in this blog post) converted properly to iptables rules. Those that did not convert contain elements such as pcre and threshold that are not currently supported by fwsnort.

Below is an example of one Snort signature that did convert correctly. This is rule ID 2009201, and it detects shellcode directed at TCP/445 from Conficker.B: alert tcp $EXTERNAL_NET any -> $HOME_NET 445 (msg:"ET CURRENT_EVENTS Conficker.b Shellcode"; flow:established,to_server; content:"|e8 ff ff ff ff c2|_|8d|O|10 80|1|c4|Af|81|9MSu|f5|8|ae c6 9d a0|O|85 ea|O|84 c8|O|84 d8|O|c4|O|9c cc|Ise|c4 c4 c4|,|ed c4 c4 c4 94|&<O8|92|\;|d3|WG|02 c3|,|dc c4 c4 c4 f7 16 96 96|O|08 a2 03 c5 bc ea 95|\;|b3 c0 96 96 95 92 96|\;|f3|\;|24 |i|95 92|QO|8f f8|O|88 cf bc c7 0f f7|2I|d0|w|c7 95 e4|O|d6 c7 17 cb c4 04 cb|{|04 05 04 c3 f6 c6 86|D|fe c4 b1|1|ff 01 b0 c2 82 ff b5 dc b6 1f|O|95 e0 c7 17 cb|s|d0 b6|O|85 d8 c7 07|O|c0|T|c7 07 9a 9d 07 a4|fN|b2 e2|Dh|0c b1 b6 a8 a9 ab aa c4|]|e7 99 1d ac b0 b0 b4 fe eb eb|"; reference:url,www.honeynet.org/node/388; reference:url,doc.emergingthreats.net/2009201; reference:url,www.emergingthreats.net/cgi-bin/cvsweb.cgi/sigs/CURRENT_EVENTS/CURRENT_Conficker; classtype:trojan-activity; sid:2009201; rev:4;) Here is the equivlent iptables command built by fwsnort and included in the /etc/fwsnort/fwsnort.sh script. Note the usage of the FWSNORT_FORWARD_ESTAB chain which is reserved for packets that are part of established TCP connections: $IPTABLES -A FWSNORT_FORWARD_ESTAB -p tcp --dport 445 -m string --hex-string "|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|" --algo bm -m comment --comment "sid:2009201; msg:ET CURRENT_EVENTS Conficker.b Shellcode; classtype:trojan-activity; reference:url,www.honeynet.org/node/388; rev:4; FWS:1.0.6;" -j LOG --log-ip-options --log-tcp-options --log-prefix "[3] DRP SID2009201 ESTAB " Because the pattern in the above signature is longer than 128 bytes, we'll increase the value of the MAX_STRING_LEN variable to 256 in the /etc/fwsnort/fwsnort.conf file. With that done, let's execute the /etc/fwsnort/fwsnort.sh script now and see how iptables handles such traffic on the wire: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
iptables v1.4.1.1: STRING too long `|e8ffffffffc2|_|8d|O|1080|1|c4|Af|81|9MSu|f5|8|aec69da0|O|85ea|O|84c8|O|84d8|O|c4|O|9ccc|Ise|c4c4c4|,|edc4c4c494|&<O8|923bd3|WG|02c3|,|dcc4c4c4f7169696|O|08a203c5bcea953bb3c096969592963bf33b24|i|9592|QO|8ff8|O|88cfbcc70ff7|2I|d0|w|c795e4|O|d6c717cbc404cb|{|040504c3f6c686|D|fec4b1|1|ff01b0c282ffb5dcb61f|O|95e0c717cb|s|d0b6|O|85d8c707|O|c0|T|c7079a9d07a4|fN|b2e2|Dh|0cb1b6a8a9abaac4|]|e7991dacb0b0b4feebeb|' Try `iptables -h' or 'iptables --help' for more information.
Ok, that is disappointing. It turns out that iptables currently enforces a 128-byte maximum on all strings supplied to the string match extension for inspecting payload data. Normally this is not a problem since the individual patterns in most Snort rules are typically less than 128 bytes, but in this case we'd like to work around this limitation. To do so requires that we patch and recompile the xt_string kernel module (assuming xt_string is configured as a module) with the following patch: # git diff
diff --git a/include/linux/netfilter/xt_string.h b/include/linux/netfilter/xt_string.h
index 8a6ba7b..afc60a2 100644
--- a/include/linux/netfilter/xt_string.h
+++ b/include/linux/netfilter/xt_string.h
@@ -1,7 +1,7 @@
#ifndef _XT_STRING_H
#define _XT_STRING_H

-#define XT_STRING_MAX_PATTERN_SIZE 128
+#define XT_STRING_MAX_PATTERN_SIZE 256
#define XT_STRING_MAX_ALGO_NAME_SIZE 16

enum {
@@ -15,7 +15,7 @@ struct xt_string_info
u_int16_t to_offset;
char algo[XT_STRING_MAX_ALGO_NAME_SIZE];
char pattern[XT_STRING_MAX_PATTERN_SIZE];
- u_int8_t patlen;
+ u_int16_t patlen;
union {
struct {
u_int8_t invert;
With the new xt_string module loaded let's execute the fwsnort.sh script once again: # /etc/fwsnort/fwsnort.sh
[+] Adding emerging-all rules:
Rules added: 12
[+] Finished.
Ah, that's better. The fwsnort iptables policy loaded properly in the running kernel. Now, let's use the perl trigger command along with netcat to send data across the wire that should match the signature. The trigger itself can be found in the /etc/fwsnort/fwsnort.sh script. First, we fire up a netcat server on TCP port 445 on a target system which is protected by another system running the fwsnort iptables policy, and then with the perl trigger we send bytes that match the Conficker.B shell code signature across the wire to the target. The complete perl command is listed below even though it certainly is obtuse looking. You can see how the bytes it is printing match the content strings in the original signature: [target]# nc -l -p 445
[attacker]$ perl -e 'print "\xe8\xff\xff\xff\xff\xc2_\x8dO\x10\x801\xc4Af\x819MSu\xf58\xae\xc6\x9d\xa0O\x85\xeaO\x84\xc8O\x84\xd8O\xc4O\x9c\xccIse\xc4\xc4\xc4,\xed\xc4\xc4\xc4\x94&<O8\x92\x3b\xd3WG\x02\xc3,\xdc\xc4\xc4\xc4\xf7\x16\x96\x96O\x08\xa2\x03\xc5\xbc\xea\x95\x3b\xb3\xc0\x96\x96\x95\x92\x96\x3b\xf3\x3b\x24i\x95\x92QO\x8f\xf8O\x88\xcf\xbc\xc7\x0f\xf72I\xd0w\xc7\x95\xe4O\xd6\xc7\x17\xcb\xc4\x04\xcb{\x04\x05\x04\xc3\xf6\xc6\x86D\xfe\xc4\xb11\xff\x01\xb0\xc2\x82\xff\xb5\xdc\xb6\x1fO\x95\xe0\xc7\x17\xcbs\xd0\xb6O\x85\xd8\xc7\x07O\xc0T\xc7\x07\x9a\x9d\x07\xa4fN\xb2\xe2Dh\x0c\xb1\xb6\xa8\xa9\xab\xaa\xc4]\xe7\x99\x1d\xac\xb0\xb0\xb4\xfe\xeb\xeb"' |nc 10.1.1.1 445
The fwsnort iptables policy has reset the connection, and the following iptables log message was also produced: Jul 4 13:23:18 fwsnort kernel: [10966.350782] [2] REJ SID2009201 ESTAB IN=lo OUT= MAC=AB:00:00:AB:00:00:AB:00:00:AB:00:00:08:00 SRC=192.168.10.1 DST=10.1.1.1 LEN=244 TOS=0x00 PREC=0x00 TTL=64 ID=5976 DF PROTO=TCP SPT=49053 DPT=445 WINDOW=513 RES=0x00 ACK PSH URGP=0 OPT (0101080A0028B05B0028B058) Of course, the best defense against Conficker is to patch Windows systems against the MS08-067 vulnerability, and to use Nmap to scan for systems that are already infected. Those that are should be completely reimaged.

Software Release - fwsnort-1.0.6

software release fwsnort-1.0.6 The 1.0.6 release of fwsnort is ready for download. This release fixes a bug that caused some Snort rules to not be translated into iptables rules due to improper handling of escaped semicolons. Now that this bug has been fixed, an additional 58 rules from the Emerging Threats rule set are now properly supported. Also made it easier to point fwsnort at a single file with a Snort rule set to be converted (see the --fwsnort-rfile command line argument).

Here is the complete ChangeLog:

  • (Franck Joncourt) Updated fwsnort to use the "! <option> <arg> syntax instead of the older "<option> ! <arg> for the iptables command line.
  • (Franck Joncourt) For the --hex-string and --string matches, if the argument exceeds 128 bytes (iptables 1.4.2) then iptables fails with an error "iptables v1.4.2: STRING too long". Fixes this with a patch that adds a new variable in fwsnort.conf "MAX_STRING_LEN", so that the size of the content can be limited. If the content (null terminated string) is more than MAX_STRING_LEN chars, fwsnort throws the rule away.
  • Bug fix to allow fwsnort to properly translate snort rules that have "content" fields with embedded escaped semicolons (e.g. "\;"). This allows fwsnort to translate about 58 additional rules from the Emerging Threats rule set.
  • Bug fix to allow case insensitive matches to work properly with the --include-re-caseless and --exclude-re-caseless arguments.
  • Bug fix to move the 'rawbytes' keyword to the list of keywords that are ignored since iptables does a raw match anyway as it doesn't run any preprocessors in the Snort sense.
  • Added the --snort-rfile argument so that a specific Snort rules file (or list of files separated by commas) is parsed.
  • Added a small hack to choose the first port from a port list until the iptables 'multiport' match is supported.
  • Updated to consolidate spaces in hex matches in the fwsnort.sh script since the spaces are not part of patterns to be searched anyway.
  • Updated to the latest complete rule set from Emerging Threats (see http://www.emergingthreats.net/).
  • Added the "fwsnort-nobuildreqs.spec" file for building fwsnort on systems (such as Debian) that do not install/upgrade software via RPM. This file omits the "BuildRequires: perl-ExtUtils-MakeMaker" directive, and this fixes errors like the following on an Ubuntu system when building fwsnort with rpmbuild: rpm: To install rpm packages on Debian systems, use alien. See README.Debian.
    error: cannot open Packages index using db3 - No such file or directory (2)
    error: cannot open Packages database in /var/lib/rpm

Handling Escaped Semicolons in Snort Rules with fwsnort

fwsnort and escaped semicolons in Snort rules Recently I ran into a situation in which several Snort rules from the Emerging Threats rule sets were not being properly translated into iptables rules by fwsnort. It turned out that fwsnort did not correctly parse Snort content fields that contained escaped semicolons (e.g. "\;"). In the Snort signature language, the argument to every keyword in the body of a Snort rule such as content, pcre, and flowbits is terminated with a semicolon, and some keywords also use opening and closing double quotes. But, Snort supports escaping with a backslash so that these characters can easily be made to be part of a keyword argument as opposed to the delimiting syntax. Snort does not allow the argument of a content keyword to contain an embedded semicolon that is not escaped (e.g. content:"distloc=;";), and will generate an error similar to the following if a rule does not conform to this: Initializing rule chains...
ERROR: /etc/snort/rules/web-cgi.rules(3) =& Content data needs to be enclosed in quotation marks (")!
Fatal Error, Quitting..
In this case, we change content:"distloc=;"; to content:"distloc=\;"; and the error goes away. However, in addition to the escaping mechanism, any double quote or semicolon that is part of a content field can just be specified in hex notation between pipe "|" characters instead.

So, what are the tradeoffs in using one convention vs. the other?

Using backslashes can complicate the way an argument looks (since backslashes are not part of the content that is actually searched for in network traffic), but they can also make the argument more intuitive to look at than the hex syntax. This can be important when looking at lots of packet traces. For example, in web traffic the semicolon is used in HTTP request headers as a separator and therefore has special significance in HTTP, and the semicolon is also a separator for multiple commands launched from a command shell. So, for those that don't automatically know the hex equivalent of a semicolon (0x3b), it might be better to look at content:"distloc=\;"; instead of content:"distloc=|3B|"; when interpreting signature matches against raw packet traces since it emphasizes the importance of the semicolon.

There are important examples of Snort rule sets that use each strategy for the arguments to content fields (escaped semicolons vs. the hex equivalent). The complete Emerging Threats rule set contains 58 signatures with escaped semicolons: $ perl -lwne 'while (/content:"(.*?)"/g) { $tmp = $1; if ($tmp =~ /\x3b/) { print $tmp; }} ' emerging-all.rules |wc -l
58
Note that the 'while (/content:"(.*?)"/g)' loop is necessary above in order to parse all content fields from each Snort rule - using something like 'if (/content:"(.*?)"/' would just parse the very first content field in each Snort rule. Here is an example content field from the "ET MALWARE Vaccine-program.co.kr Related Spyware Checkin" signature: |0d 0a|User-Agent\: Mozilla/3.0 (compatible\; Indy Library)|0d 0a| By contrast, I've seen a few Sourcefire VRT rule sets, and none of them appear to use escaped semicolons in any of their signatures. They always prefer to use the "|3B|" hex notation.

Now, why is this important for fwsnort? The reason is that the current version - fwsnort-1.0.5 - does not properly parse content fields with escaped semicolons. However, this will be corrected in the upcoming fwsnort-1.0.6 release, which will be completed within the next two days or so. In the meantime, here is a link to fwsnort-1.0.6-pre4 that corrects this issue.

ISSA Journal's Toolsmith on fwsnort and iptables IDS

ISSA Toolsmith on fwsnort Russ McRee of holisticinfosec.org has written the October Toolsmith issue from the ISSA Journal about fwsnort and intrusion detection. Russ emphasizes practical usages of fwsnort with tuning the Snort signature set it translates into an iptables policy, and after using fwsnort to build a detection policy, he sends several example attacks at a webserver. These attacks include directory traversal, XSS, and CSRF attacks, and he illustrates how the attacks are detected by fwsnort via the iptables log messages it generates. Here is an attack identified as ET WEB-MISC Poison Null Byte by Snort rule ID 2003099, and below is the log message generated by fwsnort when the attack is sent against a webserver: alert tcp $EXTERNAL_NET any -> $HTTP_SERVERS $HTTP_PORTS (msg:"ET WEB-MISC Poison Null Byte"; flow:established,to_server; uricontent:"|00|"; depth:400; reference:cve,2006-4542; reference:cve,2006-4458; reference:cve,2006-3602; reference:url,www.security-assessment.com/Whitepapers/0x00_vs_ASP_File_Uploads.pdf; classtype: web-application-activity; sid:2003099; rev:3;)

Sep 4 16:59:45 holisticinfosec01 kernel: [29567.666562] [234] SID2003099 ESTAB IN=eth0 OUT= MAC=00:0c:6e:3c:b4:71:00:13:e8:e8:81:37:08:00 SRC=192.168.248.101 DST=192.168.248.104 LEN=40 TOS=0x00 PREC=0x00 TTL=128 ID=24638 DF PROTO=TCP SPT=7987 DPT=80 WINDOW=16126 RES=0x00 ACK URGP=0
Note the iptables log prefix "[234] SID2003099 ESTAB" above. This indicates that rule number 234 (from the custom FWSNORT_INPUT_ESTAB chain in this case which is jumped to from the built-in INPUT chain) matched traffic described by Snort rule ID 234, and that the match took place in an ESTABLISHED TCP connection. By using the iptables state tracking extensions, fwsnort matches up TCP connection states to packets on the wire in order to emulate the Snort flow keyword - at least to the extent that bi-directional communication is required before an event is generated.

Russ also mentions the fwsnort --ipt-drop and --ipt-reject command line arguments that change the stance of fwsnort from a detection engine to a mechanism that can drop malicious packets before forwarding them to their intended target. Given that iptables is firewall, it runs inline to network traffic by definition, and therefore is in ideal position to respond in this way.