CipherDyne http://www.cipherdyne.org The network and system security blog of Michael Rash en-us Copyright 2000-2008, Michael Rash. All Rights Reserved. Michael Rash 2006-02-28T16:48:01-05:00 http://www.cipherdyne.org/blog/2008/07/software-release-fwknop-1.9.6.html Software Release - fwknop-1.9.6 2008-07-22T21:01:30-05:00 Michael Rash software release fwknop-1.9.6 The 1.9.6 release of fwknop is ready for download. This release was made at The Last HOPE conference over the weekend in NYC, and introduces several new features that make SPA traffic more difficult to detect on a network even when an IDS is watching. Previous to the 1.9.6 version of fwknop, it was possible to write Snort rules to detect SPA traffic by looking for invariant artifacts from base-64 encoding and also from encryption operations (both Rijndael and GnuPG). These artifacts are now stripped out by the fwknop client before being transmitted on the wire, and the result is that SPA packets are now more highly randomized. This implies that SPA packets generated by the 1.9.6 release are not compatible with older fwknop deployments by default, but the client does offer command line arguments to maintain compatibility if necessary.

Here is an excerpt from the ChangeLog:

  • SPA packets are base64-encoded by the fwknop client, and this encoding pads data with '=' chars until the total length of the encoded data is a multiple of four. This characteristic can be used within a Snort rule to assist in the detection of SPA communications. The 1.9.6 release of fwknop strips out these padding characters before the client sends an SPA packet, and the fwknopd server adds them back in (to form a multiple of four) before base64 decoding the packet data. This reduces the level of identifying information in SPA packets and therefore makes it more difficult to detect the usage of SPA for service access. For reference, a Snort rule that would detect SPA packets via the trailing '=' chars (previous to this release) would be:

    alert udp any any -> any 62201 (msg:"fwknop SPA traffic"; dsize:>150; pcre:"/==$/"; sid:20080001; rev:1;)

  • According to the 'file' command (via it's 'magic') database, files that are encrypted with GnuPG begin with 0x8502, and this is true for SPA packets generated by fwknop (previous to this release). In fwknop-1.9.6, the "hQ" prefix is removed by the fwknop client and added back in by the fwknopd server if it doesn't exist. This measure is another effort to make SPA packets more difficult to detect on the wire, such as with the following Snort rule:

    alert udp any any -> any 62201 (msg:"fwknop GnuPG encrypted SPA traffic"; content:"hQ"; depth:2; dsize:>1000; sid:20080003; rev:1;)

  • Updated the fwknop client to randomize the UDP source port for default SPA packet generation. There is also a new command line argument --Source-port <port> to allow the user to manually set the source port on the fwknop client command line. A lot more attention is given now to source ports after the Dan Kaminsky DNS caching exploit, and it turns out that even on Linux that the kernel did not randomize UDP source ports until the 2.6.24 kernel. Of course, any userspace process is free to request a random port itself, but if a userspace application did not build this in then it would be up to the kernel to assign a source port. In the case of Linux, here are two links that show the change to the kernel code as well as the ChangeLog entry for UDP source port randomization:

    Kernel commit

    ChangeLog-2.6.24
]]> http://www.cipherdyne.org/blog/2008/07/mitigating-dns-cache-poisoning-attacks-with-iptables.html Mitigating DNS Cache Poisoning Attacks with iptables 2008-07-14T21:01:30-05:00 Michael Rash Digg Mitigating the Kaminksy DNS attack
Kaminsky DNS Exploit It is well known that last week Dan Kaminsky publicized a cache poisoning exploit against DNS. The details of this exploit have not yet been released, but Dan will present a talk at the Blackhat Briefings next month that will clarify the technical specifics behind the vulnerability - and therefore how to exploit it. Even though Dan has yet to release the details, enough information has already been released to show that iptables may be able to mitigate the vulnerability with a single well-chosen rule in your iptables policy. Further, this technique can be used right now whenever a vulnerable DNS server is deployed on (or behind) a system running iptables. Although updates have been released in a coordinated effort for most platforms, not everyone patches their systems immediately, and in some cases it can be easier to execute a few iptables commands than to deploy such updates. Of course, patching the vulnerability should be the top priority, but the exact details of the attack haven't been disclosed so it is hard to gauge risk. That said, if Dan Kaminsky says to patch, then it's probably a good idea to do so. In the meantime, let's press on.

The CERT advisory highlights three problems in the existing DNS infrastructure:

  1. Lack of sufficient randomness in the selection of source ports for DNS queries
  2. DNS transaction ID values that also exhibit insufficient randomness
  3. Multiple outstanding requests for the same resource record
For this blog post we are concerned about source port predictability for DNS queries, but Dan's cache poisoning attack (most likely) relies heavily on the first two problems in combination. Dan has probably found a clever way to weaponize his attack so that it can be leveraged against lots of DNS servers.

How can iptables help? First, let's assume that source port predictability is a necessary prerequisite to poison a DNS cache by Dan's technique. So, if iptables can introduce additional randomness into the source port that bind uses - at least as far as any upstream DNS server can tell - for each DNS query then an attacker would not be able to rely on source port predictability. The iptables SNAT target supports the ability to randomize both TCP and UDP source ports (see the --random option) even if a userspace application chooses a specific source port. This applies to both locally generated and forwarded packets. Hence, if a piece of userland software (such as bind) uses predictable source ports, iptables can rectify this via the SNAT target subject to some connection tracking restrictions.

To illustrate this, we use nmap with its --source-port option to set the source port to 44444 for a UDP scan of port 53 on two different servers, and verify with a packet trace that the source port is indeed set to this value. We'll assume the IP's 22.2.2.2 and 33.3.3.3 simulate upstream DNS servers from a local DNS server on the system with IP 11.1.1.1: 

# nmap --source-port 44444 -P0 -p 53 -sU 22.2.2.2 33.3.3.3

19:26:19.625637 IP 11.1.1.1.44444 > 22.2.2.2.53: [|domain]
19:26:19.625790 IP 11.1.1.1.44444 > 33.3.3.3.53: [|domain]
19:26:19.729520 IP 11.1.1.1.44444 > 33.3.3.3.53: [|domain]
19:26:20.626527 IP 11.1.1.1.44444 > 22.2.2.2.53: [|domain]
As you can see, the source port for each UDP datagram from 11.1.1.1 is set to 44444. Now, to force the source port to be randomized when each packet is transmitted despite setting it to 44444 from the nmap command line, let's use the SNAT target with the --random option (note that it is assumed that the iptables policy is also using connection tracking to allow expected replies through as well, etc.): 

# iptables -t nat -I POSTROUTING 1 -p udp -s 11.1.1.1 --dport 53 \
-j SNAT --to 11.1.1.1 --random
By executing the same nmap command again, now we see that the SNAT target has changed the source port on the outgoing UDP datagrams to 9374 (for 22.2.2.2) and 54743 (for 33.3.3.3): 

# nmap --source-port 44444 -P0 -p 53 -sU 22.2.2.2 33.3.3.3

19:28:05.613637 IP 11.1.1.1.9374 > 22.2.2.2.53: [|domain]
19:28:05.613792 IP 11.1.1.1.54743 > 33.3.3.3.53: [|domain]
19:28:05.717536 IP 11.1.1.1.54743 > 33.3.3.3.53: [|domain]
19:28:06.617553 IP 11.1.1.1.9374 > 22.2.2.2.53: [|domain]
Hence, iptables can introduce randomness into source ports via a NAT operation regardless of whether a userspace application requests a specific source port.

However, you may have noticed that nmap generates two packets for each scan of UDP/53, and you may have also noticed that the source port is 9374 for both packets sent to 22.2.2.2, and 54743 for both packets sent to 33.3.3.3. While the original source port of 44444 has definitely been randomized, it appears that this randomization is not taking place on a per-packet basis. What is really happening is that iptables has to use its connection tracking mechanism to map packets translated with the SNAT target to the correct socket, and this mapping is created from the srcIP/dstIP/sport/dport/protocol tuple along with a set of timers (see the nf_ct_udp_timeout and nf_ct_udp_timeout_stream variables in the kernel linux/net/netfilter/nf_conntrack_proto_udp.c file). So, if multiple requests are made to the same upstream DNS server within a 30 second time window, then the SNAT --random rule will map the source port for each request to the same (randomly assigned) source port - the initial tuples are the same after all.

At first glance this might seem to be a show stopper, but assuming that Dan's cache poisoning attack requires a process of querying different DNS servers - at least one of which is under the control of the attacker so that the source port behavior of the targeted server can be monitored - then the SNAT --random strategy would provide an effective defense. That is, the iptables state tracking code will assign a different random source port on (at least) a per-server basis even for rapid sets of DNS queries. Further, for queries to the same DNS server that are 30 seconds apart, the source port will change on a per-query basis as well.

This should be an effective defense against the attack, but we'll have to wait for Blackhat to be sure.

On another note, applications commonly do not bother requesting specific source ports for client sockets because they assume that the networking stack provided by the local kernel will assign a sensible source port. Or, they don't have to worry about the source port because they build in enough security at the application layer (unlike DNS implementations that don't sufficiently randomize transaction ID's) such that source port prediction does not give an advantage to the attacker. Still, those stacks that don't choose a random source port are not helping the state of network security, and the Kaminsky DNS attack is a perfect example of why. It turns out that the Linux kernel has only recently started randomizing UDP source ports as of 2.6.24.

Update 07/16/2008: Jon Hart has also written a blog post that illustrates using OpenBSD's pf firewall to implement a similar mitigation strategy. Hence, multiple pieces of firewall infrastructure can offer effective protection from the cache poisoning attack.

]]> http://www.cipherdyne.org/blog/2008/07/analyzing-a-trac-spam-attempt.html Analyzing a Trac SPAM Attempt 2008-07-05T21:01:30-05:00 Michael Rash Spam Attempts Through Trac One of the best web interfaces for visualizing Subversion repositories (as well as providing integrated project management and ticketing functionality) is the Trac Project, and all Cipherdyne software projects use it. With Trac's success, it also becomes a target for those that would try to subvert it for purposes such as creating spam. Because I deploy Trac with the Roadmap, Tickets, and Wiki features disabled, my deployment essentially does not accept user-generated content (like comments in tickets for example) for display. This minimizes my exposure to Trac spam, which has become a problem significant enough for various spam-fighting strategies to be developed such as configuring mod-security and a plugin from the Trac project itself.

Even though my Trac deployment does not display user-generated content, there are still places where Trac accepts query strings from users, and spammers seem to try and use these fields for their own ends. Let's see if we can find a few examples. Trac web logs are informative, but sifting through huge logfiles can be tedious. Fortunately, simply sorting the logfiles by line length (and therefore by web request length) allows many suspicious web requests to bubble up to the top. Below is a simple perl script sort_len.pl that sorts any ascii text file by line length, and precedes each printed line with the line length followed by the number of lines equal to that length. That is, not all lines are printed since the script is designed to handle large files - we want the unusually long lines to be printed but the many shorter lines (which represent the vast majority of legitimate web requests) to be summarized. This is an important feature considering that at this point there is over 2.5GB of log data specifically from my Trac server. 

$ cat sort_len.pl
#!/usr/bin/perl -w
#
# prints out a file sorted by longest lines
#
# $Id: sort_len.pl 1739 2008-07-05 13:44:31Z mbr  $
#

use strict;

my %url       = ();
my %len_stats = ();
my $mlen = 0;
my $mnum = 0;

open F, "< $ARGV[0]" or die $!;
while (<F>) {
    my $len = length $_;
    $url{$len} = $_;
    $len_stats{$len}++;
    $mlen = $len if $mlen < $len;
    $mnum = $len_stats{$len}
        if $mnum < $len_stats{$len};
}
close F;

$mlen = length $mlen;
$mnum = length $mnum;

for my $len (sort {$b <=> $a} keys %url) {
    printf "[len: %${mlen}d, tot: %${mnum}d] %s",
            $len, $len_stats{$len}, $url{$len};
}

exit 0;
To illustrate how it works, below is the output of the sort_len.pl script used against itself. Note that at the top of the output the more interesting code appears whereas the most uninteresting code (such as blank lines and lines that contain closing "}" characters) are summarized away at the bottom: 
$ ./sort_len.pl sort_len.pl
[len: 51, tot: 1] # $Id: sort_len.pl 1739 2008-07-05 13:44:31Z mbr  $
[len: 50, tot: 1]     printf "[len: %${mlen}d, tot: %${mnum}d] %s",
[len: 48, tot: 1]             $len, $len_stats{$len}, $url{$len};
[len: 44, tot: 1] # prints out a file sorted by longest lines
[len: 43, tot: 1] for my $len (sort {$b <=> $a} keys %url) {
[len: 37, tot: 1]         if $mnum < $len_stats{$len};
[len: 34, tot: 1]     $mlen = $len if $mlen < $len;
[len: 32, tot: 1] open F, "< $ARGV[0]" or die $!;
[len: 29, tot: 1]     $mnum = $len_stats{$len}
[len: 25, tot: 1]     my $len = length $_;
[len: 24, tot: 1]     $len_stats{$len}++;
[len: 22, tot: 2] $mnum = length $mnum;
[len: 21, tot: 1]     $url{$len} = $_;
[len: 20, tot: 2] my %len_stats = ();
[len: 19, tot: 1] #!/usr/bin/perl -w
[len: 14, tot: 3] while (<F>) {
[len: 12, tot: 1] use strict;
[len:  9, tot: 1] close F;
[len:  8, tot: 1] exit 0;
[len:  2, tot: 5] }
[len:  1, tot: 6]
Now, let's execute the sort_len.pl script against the trac_access_log file and look at one of the longest web requests. (The sort_len.pl script was able to reduce the 12,000,000 web requests in my Trac logs to a total of 610 interesting lines.) This particular request is 888 characters long, but there were some other similar suspicious requests that had over 4,000 characters that are not displayed for brevity: 
[len: 888, tot: 1] 195.250.160.37 - - [02/Mar/2008:00:30:17 \
-0500] "GET /trac/fwsnort/anydiff?new_path=%2Ffwsnort%2Ftags%2Ffwsnort\
-1.0.3%2Fsnort_rules%2Fweb-cgi.rules&old_path=%2Ffwsnort%2Ftags%2F\
fwsnort-1.0.3%2Fsnort_rules%2Fweb-cgi.rules&new_rev=http%3A%2F%2Ff\
1234.info%2Fnew5%2Findex.html%0Ahttp%3A%2F%2Fa1234.info%2Fnew4%2F\
map.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew2%2Findex.html%0Ahttp%3A%2F\
%2Fs1234.info%2Fnew9%2Findex.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew6%2F\
map.html%0A&old_rev=http%3A%2F%2Ff1234.info%2Fnew5%2Findex.html%0Ahttp\
%3A%2F%2Fa1234.info%2Fnew4%2Fmap.html%0Ahttp%3A%2F%2Ff1234.info%2Fnew2\
%2Findex.html%0Ahttp%3A%2F%2Fs1234.info%2Fnew9%2Findex.html%0Ahttp%3A\
%2F%2Ff1234.info%2Fnew6%2Fmap.html%0A HTTP/1.1" 200 3683 "-" \
"User-Agent: Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; \
Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR \
1.1.4322; .NET CLR 2.0.50727; InfoPath.2)"
My guess is that the above request is a bot that is trying to do one of two things: 1) force Trac to accept the content in the request (which contains a bunch of links to pages like "http://f1234.info/new/index.html" - note that I altered the domain so as to not legitimize the original content) and display it for other Trac users or to search engines, or 2) force Trac itself to generate web requests to the provided links (perhaps as a way to increase hit or referrer counts from domains - like mine - that are not affiliated with the spammer). Either way, the strategy is flawed because the request is against the Trac "anydiff" interface which doesn't accept user content other than svn revision numbers, and (at least in Trac-0.10.4) such requests do not cause Trac to issue any external DNS or web requests - I verified this with tcpdump on my Trac server after generating similar requests against it.

Still, in all of my Trac web logs, the most suspicious web requests are against the "anydiff" interface, and specifically against the "web-cgi.rules" file bundled within the fwsnort project. But, the requests never come from the same IP address, the "anydiff" spam attempts never hit any other link besides the web-cgi.rules page, and they started with regularity in March, 2008. This makes a stronger case for the activity coming from bot that is unable to infer that its activities are not actually working (no surprise there). Finally, I left the original IP address 195.250.160.37 of the web request above intact so that you can look for it in your own web logs. Although 195.250.160.37 is not listed in the Spamhaus DNSBL service, a rudimentary Google search indicates that 195.250.160.37 has been noticed before by other sites as a comment spammer.

]]> http://www.cipherdyne.org/blog/2008/06/spa-talk-at-the-last-hope-computer-security-conference.html SPA Talk at the Last HOPE Computer Security Conference 2008-06-28T21:01:30-05:00 Michael Rash SPA talk at the Last HOPE conference Next month in NYC the final Hackers On Planet Earth (HOPE) conference will take place from July 18th through the 20th. I will be giving a talk there entitled "Port Knocking and Single Packet Authorization: Practical Deployments", and here is the abstract:

   Port Knocking and its big brother, Single Packet Authorization (SPA), can provide a robust additional layer of protection for services such as SSH, but there are many competing Port Knocking and SPA implementations. This talk will present practical usages of fwknop in Port Knocking and SPA modes, and discuss what works and what doesn't from a protocol perspective. Integration points for both iptables and ipfw firewalls on Linux and FreeBSD systems will be highlighted, and client-side support on Windows will be demonstrated. Finally, advanced functionality such as inbound NAT support for authenticated connections, sending SPA packets over the Tor anonymity network, and covert channel usages will be discussed. With SPA deployed, anyone scanning for a service with Nmap cannot even tell that it is listening; let alone target it with an exploit (zero-day or not).

A goal for the talk will be to start with the most basic port knocking deployment (a shared sequence of only one port) and build from there into encrypted port knocking sequences, and then move into the SPA realm with SPA packets encrypted with Rijndael and finally with GnuPG. Along the way security tradeoffs will be discussed. For example, a shared sequence of a single port allows an extremely simplistic port knocking implementation (so there is less risk of a vulnerability in the port knocking software itself), but then any casual port scan or stray packet that hits the shared port also qualifies as a valid port knock sequence. At the high end, SPA packets encrypted with GnuPG solve all sorts of difficulties with simple port knocking from a protocol perspective, but there is the slight expense of a more complicated implementation (although it is still a lot harder to target an SPA implementation with an exploit than a complicated TCP-based service that advertises its existence to the world under any basic port scan).

At the talk I will also release the next version (1.9.6) of fwknop.

If you are going to be at the HOPE conference, please stop by and say 'hello'. No Starch Press will also be at the conference so I'm sure I will hang out at their booth much of the time as well. ]]> http://www.cipherdyne.org/blog/2008/06/software-release-gootrude-0.2.html Software Release - gootrude-0.2 2008-06-19T21:01:30-05:00 Michael Rash Gootrude queries Yahoo Gootrude-0.2 is available for download. This release changes the default search engine from Google to Yahoo because it appears that Yahoo's Terms of Service do not prohibit automated queries (although I'm not a lawyer). Gootrude retains the ability to query Google at your own risk if you so desire. As long as you use Gootrude as designed to collect search engine results only once per day with a limited number of search terms (say, 20 or less), it is unlikely to provoke a negative reaction from search engines. Over time, support for many additional search engines will be added to Gootrude so that search results can be trended from all sorts of different data sources.

Also, this post should really have announced Gootrude as a "search results trender" instead of a "search trender". The goal of the project is made clear in the README, and here is the complete ChangeLog, for the Gootrude-0.2 release:

  • Added support for querying Yahoo for search results and made this the default because Google's TOS does not technially allow automated queries. However, Gootrude continues to support querying Google for search results, and responsible use of Gootrude once per day with a limited set of search terms should most likely not be a cause for concern for Google. A new syntax of the searchterms.conf file allows search terms to express which search engine should be queried on an individual basis according to the following syntax (note the middle element): [Linux "highspeed firewalls"] [yahoo:count] [Linux_highspeed_firewalls.dat]
  • Changed the default USER_AGENT variable to "Gootrude <version>".
  • Added a percentage-based offset for the plot minimum and maximum ranges on the y-axis (with a default of 10% controlled by two new variables MIN_PERCENT_DIFF and MAX_PERCENT_DIFF). This value can also be set by the search terms themselves via an additional config element as follows (for a 20% offset instead for example): [yrange:20%/20%]
  • Increased SEARCH_TERM_DELAY to two seconds to be nicer to search engines.
]]> http://www.cipherdyne.org/blog/2008/06/trending-low-volume-google-searches-introducing-gootrude.html Trending Low-Volume Google Searches - Introducing Gootrude 2008-06-15T21:01:30-05:00 Michael Rash Introducing Gootrude UPDATE 06/16/08: 1) After making Slashdot today and reading the comments, I feel I should clarify a few things regarding this post. First, it was pointed out (quite rightly) that Gootrude trends the number of documents in Google's index that contain each search term, whereas Google Trends tracks the search volume associated with a search term. These are not the same thing. However, I would wager that they are related, though certainly not in some nice 1:1 correspondence. That is, if a relatively unique word - say, "myspace" - is mentioned within 100,000 web sites, then the number of times this word is used as a search term in Google will be higher than if "myspace" were mentioned in only 10 web sites just because it is exposed to many more people who would then be interested in all things "myspace". I would also wager that the reverse is true. So, if "myspace" is an extremely popular search term, then it will probably also appear in many more web sites around the Internet. Gootrude attempts to trend the number of hits a given search term returns in Google, and by the above reasoning, there is a loose correspondence between this and the number of times the term is searched for in Google. Perhaps this is not useful, but only time will tell after Gootrude is used by a lot of people. If you will allow the above interpretation, then the remainder of the post below makes sense.

2) It was pointed out that Gootrude violates the Google Terms of Service. After actually reading this document for the first time today, I think this is a correct interpretation, and if Google were to ask me to stop making Gootrude able to make web queries then I would comply with their request (and switch Gootrude to query Yahoo instead). However, let me just say that Gootrude is designed to be run once per day with a limited set of search terms (say, about 10). I personally make many more manual queries to Google every day than this with my web browser, and my usage of Gootrude cannot even remotely be interpreted as interfering with their server operations. I'm just interested in how Google (being the best search engine) views my open source projects. As a precautionary measure, I may switch Gootrude to query Yahoo by default anyway since their TOS does not seeem to prohibit this. Also, I would like to mention that other projects seem to make use of Google services in ways that may be questionable in view of the Google TOS (see the Gmail Filesystem for example) but Google seems to allow it. Google probably derives more benefits from open discussions that such projects foster than would justify shutting them down.

Now for the original post:

The Google Trends project allows you to input search terms like "Myspace", "2008 election", or "Linux", and see how Google tracks how popular these search terms are over time. The resulting graphs can be quite interesting - spikes in search volume can sometimes be correlated against particular news articles and world events, and the Google Trends interface points these out.

This is a handy tool, but there are many search terms that Google Trends does not display any results for. Such terms (such as "Linux Firewalls" - with the quotes) have insufficient search volumes to display graphs according to the error message that Google Trends generates. Fair enough. I suppose that Google sets an internal threshold on search volume, and this threshold could be set for reasons that range anywhere from Google Trends is still experimental to Google not wanting to provide data on how it builds its massive search index for emerging search terms. Either way, I would like a way to see search term trends that Google doesn't currently make available to me.

Although I'm an open source developer and author, search terms related to my projects are not popular enough yet to be displayed by Google Trends. So, I had to roll my own trending mechanism, and this blog post announces the release of a new open source tool Gootrude (see the quick start, source code, and download links) that I wrote to do just this. The basic strategy is to take a collection of search terms defined by the user, automatically query Google for the number of results associated with each of these search terms (this is displayed by Google when doing a web search), and graph these numbers over time with Gnuplot. At this point let me state up front that Gootrude only makes use of data that Google freely provides to everyone with normal web searches, and is meant to be run once per day (so as to not be a pest in terms of the numbers of queries it makes). As an example, if you type in the word "security" into the Google web interface, it will return a string like "Results 1 - 10 of about 1,010,000,000". The "1,010,000,000" number is collected by Gootrude and stored in a file along with the current time.

For the past year, I have sent a set of search terms through Google once per day with Gootrude and the results are displayed below. Visible within the data returned from Google are strange oscillations that vary quite a bit more than I would have expected, and also evidence for what happens when a large site (like linux.com) posts an article about a Cipherdyne project.

First, below is the graph of the fairly unique word "cipherdyne" since late June 2007. The filled-in red curve is the absolute number of search results (taken each day around 1am), and the green line is the 10-day moving average. Gootrude plot of cipherdyne search term As you can see, at the beginning of the graph around July 1st Google steadily shows about 28,000 results for "cipherdyne", but towards the end of July this dips to well below 20,000 only to rebound in August to about 30,000. Then, beginning around March 1st, 2008 the results shoot up to over 100,000 briefly and then back to around 70,000 in May. How does one interpret this data? It seems unlikely that these fluctuations can be entirely explained by "actual" day-by-day changes in how external sites reference the term "cipherdyne" - there must be some index updating component that is internal to Google at work here, and we'll see a better example of this below.

Now, here is the graph of the search term "gpgdir": Gootrude plot of gpgdir search term The most obvious feature of the gpgdir graph above is the large spike to around 60,000 results around May 1st. It turns out that an article was posted to linux.com on the 24th of April, so given that "gpgdir" is not a common word, the spike seems nicely correlated with the posting of the article as it got bounced around the Internet and blogosphere. A more interesting feature perhaps is the sharp cyclic oscillation between July and December 2007. During this time, search results for gpgdir bounced from 1,000 to around 10,000 and back again several times, and the transition each time was fast - making the jump to 10,000 over the course of two days and then stabilizing for about 10 days or so and then back down to 1,000. It is almost as though Google was trying to establish the proper order of magnitude for "gpgdir" search results during this time via a sort of step function.

Finally here is the graph of "single packet authorization": Gootrude plot of single packet authorization Again, we see a dramatic spike in search results - from around 5,000 to well over 50,000 and settling down to about 10,000 around the beginning of June. Although there has been some activity related to SPA in the Ubuntu forums and also in the Gentoo forums, if this caused Google to report the search results as over 50,000 why did this number return so precipitously back to around 10,000? The links have not gone away, but they were probably mentioned on other referencing sites and then moved to less important pages over time on those sites. Perhaps Google is trying to find the appropriate steady state for its search results, and there are many factors that Google takes into account that are not available to the public.

There are lots of unanswered questions this sort of data brings to mind:

  • All of the data for the above graphs was collected from a single Linux system. How different would the results be if several systems in different geographic locations collected the data and the average for each data point was used instead?
  • Each data point was collected around 1am every morning. If the data collection time were, say, 1pm, would the results have been significantly different?
  • What is the "optimal" time scale for the moving average? Given that Google's own Trends interface seems to show search results on the macro level, would a much longer moving average than 10 days - perhaps on the order of several weeks - be a more accurate reflection of search popularity?
One thing is clear - getting search results that are meaningful is much easier with unusual search terms. With the posting of this blog entry, the term "Gootrude" should evolve nicely within Google results, and the graph of these results will be updated daily on the main Gootrude page so you can see this evolution as it unfolds.

In closing, I would like to mention that Gootrude is just getting started, so there are lots of enhancements that need to be made. Some of the most important features to develop are:

  • Integration with the Google Charts API.
  • Development of an online web portal for Gootrude so that users don't have to have their own infrastructure to run Gootrude.
  • Ability to import search data from different Gootrude collection systems.
  • Add support for data collected from additional search engines.
If you are an open source developer and would like to contribute, see the TODO file for an updated list of development tasks, or send me an email (mbr[at]cipherdyne[dot]org). Also, if you have any ideas or feedback on why some of the graphs above look the way they do in the context of how Google builds its index, please email me.

Finally, here are a few additional graphs of search terms over the past year: Gootrude plot of fwsnort Gootrude plot of michael rash Gootrude plot of Linux Firewalls Attack Detection Gootrude plot of single packet authentication Gootrude plot of iptables attack visualization ]]> http://www.cipherdyne.org/blog/2008/06/three-software-releases-psad-fwknop-gpgdir.html Three software releases - psad, fwknop, gpgdir 2008-06-08T21:01:30-05:00 Michael Rash Three software releases The following releases of the Cipherdyne security projects are now available: psad-2.1.3, fwknop-1.9.5, and gpgdir-1.9.1. The motivation for a group release stems from the need to make similar changes to the fwknop and gpgdir projects to update to version 2.11 of the Class::MethodMaker perl module from CPAN - a dependency of GnuPG::Interface. This version fixes a build error under recent versions of perl (such as perl-5.10.0) which are distributed on systems like Fedora 9. Also, for both fwknop and gpgdir, thanks to a suggestion made by Jean-Denis Girard on the fwknop mailing list, the default locale has been set to "C" via the LC_ALL environmental variable so that GnuPG output can be properly interpreted even on systems where a different locale is used. The locale can be manually set or not used at all with two new command line arguments --locale <str> and --no-locale respectively. On another note, Kevin Hilton has written an excellent how-to for fwknop on Ubuntu systems.

For psad, it was time to make a new release after it became necessary to update the whois client so that IP addresses such as 116.125.35.98 (which was scanning a psad user) could be properly identified with whois records. In addition, psad was updated to parse syslog files directly for iptables log messages instead of requiring reconfiguration of the syslog daemon to write kern.info messages to the /var/lib/psad/psadfifo named pipe. This simplifies the proper installation of psad, and is now a default setting. Although there is a slight performance penalty since psad now parses all messages that are written to the /var/log/messages file (this is the default path), it should not be noticeable on most systems. Further, the old behavior of using the named pipe can be restored via the ENABLE_SYSLOG_FILE variable in the /etc/psad/psad.conf file.

Finally, Franck Joncourt has made excellent progress in developing Debian packages for the IPTables::ChainMgr and IPTables::Parse modules, and he is also close to a Debian package for the fwknop project.

The complete change logs for these new releases can be found as follows: psad-2.1.3, fwknop-1.9.5, and gpgdir-1.9.1 ]]> http://www.cipherdyne.org/blog/2008/06/single-packet-authorization-with-port-randomization.html Single Packet Authorization with Port Randomization 2008-06-01T21:01:30-05:00 Michael Rash Single Packet Authorization with Port Randomization After two months of development, the 1.9.4 release of fwknop is available for download. This release introduces new functionality that has implications for hardening both fwknop SPA communications and the follow-on connections that client programs make (such as SSH). Specifically, the 1.9.4 release adds two port randomization options to 1) send the SPA packet over a random port between 10,000 and 65,535 (requires updating the default PCAP_FILTER variable on the fwknopd server side), and 2) select a random port that is used as the destination port in a NAT operation on the fwknopd server system. These two options can be used individually or together, and are enabled with two new command line options, --rand-port and --NAT-rand-port respectively, to the fwknop client (see examples below).

The inspiration for adding this functionality came from a post to the "Documentation, Tips & Tricks". Gentoo forum, and I have credited John Brendler with the idea. In response to John's post, I would like to mention however that all Port Knocking / Single Packet Authorization implementations suffer from "piggy-back exploits", including those that select random ports for SPA communications or for NAT operations against follow-on sessions. A "piggy-back exploit" is where an attacker takes advantage of the fact that a firewall rule inserted by a PK/SPA system accepts a connection from an IP address to a destination port specified by the SPA packet(s). So, if the attacker can spoof packets from this source IP, or if the attacker happens to be on the same internal network as the SPA client (and hence sends connections out through the same NAT device), then the firewall rule will accept these packets just as though they originated from the legitimate SPA client system. If an attacker is in a privileged position and can sniff the legitimate session as it is initiated, then one can envision an automated attack that spoofs packets from the same source IP and directs them at the same service. Further, such an "attack" can be made just by watching outgoing connections without paying any attention whatsoever to whether or not a set of SPA packets are sent first - it doesn't matter if the real connection is made to a random translated port on the SPA system; the attacker can see this port in the real connection itself.

Now, should you be concerned about such a piggy-back attack? Not really. First, if the attacker is not going through the same NAT device as the real connection, then any response to a spoofed packet will go back to the spoofed source - not back to the attacker. So, for TCP connections, unless the attacker can effectively perform a sequence prediction attack an existing connection (and even then that is of little use against an encrypted application layer protocol such as SSH), this is not very effective. Second, even if an attacker is behind the same NAT device as the SPA client, just being able to access the targeted service over TCP/IP does not imply an automatic vulnerability; SPA is an additive measure to whatever existing security mechanisms are already in place (barring a vulnerability in libpcap itself in the SPA server for example). Third, there are an awful lot of networks out there to which an attacker will not have such privileged access, and therefore not be in a position able to sniff anything useful. Forth, fwknop minimizes the opportunity for an attacker to conduct a piggy-back attack by maintaining a small window of time (30 seconds by default) for any new firewall rules after receiving a valid SPA packet. By using a connection tracking mechanism built into iptables or ipfw, any connection established during the accept window is allowed to remain open but all attempts to create a new connection must first preceeded with a new SPA packet in order to gain access.

Finally, although port randomization is an enhancement, fwknop has had the ability for a long time to allow the user to select the destination port for SPA packets with the --Server-port argument as well as the destination port for a NAT'd connection to an internal system. Hence, fwknop SPA packets are not always sent over udp/62201. But, I agree that it is useful to add the port randomization features that John Brendler suggested, and this is why I've implemented them in fwknop. Randomizing the SPA destination port along with the destination port of the follow-on connection makes traffic analysis more difficult.

Now, let us see the new --rand-port and --NAT-rand-port options in a practical example. We'll assume that the fwknopd server is at hostname spaserver with IP 11.1.1.1, and the fwknop client runs on the spaclient system with IP 12.2.2.2. We ultimately want to gain access to SSHD on the spaserver system, and we assume that iptables is configured in a default-drop stance for all attempts to communicate with SSHD. Also, there is no requirement to necessarily attempt to gain access only to an SSHD instance running on an internal server via a forwarded port - the iptables PREROUTING chain can forward a port to a local socket as well (based on a routing calculation for the destination IP), and on the fwknop client command line we use the --NAT-local argument for this.

Because the --rand-port option sends the SPA packet over a random destination port, we first need to set the PCAP_FILTER variable as follows in the /etc/fwknop/fwknop.conf file: 
[spaserver]# vi /etc/fwknop/fwknop.conf
PCAP_FILTER                 udp dst portrange 10000-65535;

[spaserver]# /etc/init.d/fwknop restart
[+] knopwatchd is running (pid: 17584), stopping daemon
[+] knoptm is running (pid: 17582), stopping daemon
[+] fwknopd is running (pid: 17580), stopping daemon
Starting the fwknop daemons.
With the fwknopd server up and running, we now use the fwknop client to gain access to SSHD on the spaserver system via a randomly selected NAT'd port (and we show that SSHD is never accessible over the standard TCP port 22 even from the spaclient system): 
[spaclient]$ nmap -sT -n -P0 -p 22 11.1.1.1

Starting Nmap 4.20 ( http://insecure.org ) at 2008-06-02 01:48 EDT
Interesting ports on 11.1.1.1:
PORT   STATE    SERVICE
22/tcp filtered ssh

Nmap finished: 1 IP address (1 host up) scanned in 12.017 seconds

[spaclient]$  fwknop -A tcp/22 --NAT-local --NAT-rand-port \
--rand-port -R -D 11.1.1.1

[+] Starting fwknop client (SPA mode)...
[+] Requesting NAT access for randomized port: 16791
    Resolving external IP via: http://www.whatismyip.org/
    Got external address: 12.2.2.2

[+] Enter an encryption key. This key must match a key in the file
    /etc/fwknop/access.conf on the remote system.

Encryption Key:

[+] Building encrypted Single Packet Authorization (SPA) message...
[+] Packet fields:

        Random data:    4846125760033285
        Username:       mbr
        Timestamp:      1212386108
        Version:        1.9.4
        Type:           5 (Local NAT access mode)
        Access:         12.2.2.2,tcp/22
        NAT access:     11.1.1.1,16791
        SHA256 digest:  ZyWG+nRGYMfWFssJuOy7bhGmJHpHia6T1igaNnVVhqI

[+] Sending 206 byte message to 11.1.1.1 over udp/52949...
    Requesting NAT access to tcp/22 on 11.1.1.1 via port 16791

[spaclient]$  ssh -p 16791 mbr@11.1.1.1
mbr@11.1.1.1's password:
[spaserver]$
The above output shows that the client system indeed has access to the spaserver system via TCP port 16791, which is forwarded to the local SSH daemon. In the /var/log/messages file, fwknopd has written the following messages to syslog: 
[spaserver]# tail /var/log/messages
Jun  2 01:54:16 spaserver fwknopd: received valid Rijndael encrypted \
packet from: 12.2.2.2, remote user: mbr, client version: 1.9.4 (SOURCE \
line num: 151)
Jun  2 01:54:17 spaserver fwknopd: add FWKNOP_PREROUTING 12.2.2.2 \
-> 11.1.1.1(tcp/16791 to 22) DNAT rule 30 sec
Jun  2 01:54:17 spaserver fwknopd: add FWKNOP_INPUT 12.2.2.2 -> \
0.0.0.0/0(tcp/22) ACCEPT rule 30 sec
Jun  2 01:54:58 spaserver fwknop(knoptm): removed iptables \
FWKNOP_PREROUTING DNAT rule for 12.2.2.2 -> 11.1.1.1(tcp/22), 30 \
sec timeout exceeded
Jun  2 01:55:11 spaserver fwknop(knoptm): removed iptables \
FWKNOP_INPUT ACCEPT rule for 12.2.2.2 -> 0.0.0.0/0(tcp/22), 30 sec \
timeout exceeded
In closing, here is an abbreviated version (the randomization options are not duplicated here) of the fwknop-1.9.4 ChangeLog:

  • Added the ability to specify the port that SPA packets are sent over with the fwknop client by using the syntax "<host|IP>:<port>". So, for example, to have the client send an SPA packet to 11.1.1.1 over UDP port 12345 (instead of the default of 62201), one could use the following command:

    $ fwknop -A tcp/22 -R -D 11.1.1.1:12345

  • Bugfix to add a check for "keep-state" in ipfw policies in addition to the existing "check-state" check (noticed by Sebastien Jeanquier).
  • Updated the install.pl script to try to determine the OS type as early as possible during the install process.
  • Added the MIN_SPA_PKT_LEN variable with 160 (bytes) as the default. This allows fwknopd to ignore packets that are not at least this many bytes (including packet headers) before any decryption attempt is made.
  • Added --time-offset-plus and --time-offset-minus args to the fwknop client command line. This allows the time stamp within an SPA packet to be influenced without setting the system clock (which normal users cannot usually do). This is useful for when the client and server systems have clocks that are out of sync.
  • Bugfix on Ubuntu systems to make sure that the fwknop init script is installed with a priority of 99 instead of 20 - this puts fwknop as late as possible within the boot sequence so that the system is ready to run fwknop.
  • Bugfix to not open ports that are not specifically requested in an SPA packet even if those ports are listed in the OPEN_PORTS variable in the access.conf file.
  • Updated to version 5.47 of the Digest::SHA module.
  • Updated to version 0.7 of the IPTables::ChainMgr module (includes perldoc documentation).
  • Updated to version 0.6 of the IPTables::Parse module (includes perldoc documentation).
  • Added NAT, port randomization, and and time offset option discussions to fwknop(8) man page.
]]> http://www.cipherdyne.org/blog/2008/05/software-release-gpgdir-1.9.html Software Release - gpgdir-1.9 2008-05-31T21:01:30-05:00 Michael Rash gpgdir-1.9 released The 1.9 release of gpgdir is ready for download. This release introduces minor fixes to change the --Obfuscate file format and adds process locking against multiple instances of gpgdir from operating against the same directory.

On a side note, Ben Martin wrote an article on gpgdir entitled "Protecting directory trees with gpgdir" for linux.com. Ben emphasizes using the gpgdir --Wipe option to securely delete files from the filesystem after they are encrypted with GnuPG, and I agree with this. In addition, there is nothing that prevents gpgdir from being used in conjunction with an encrypted filesystem such as Cryptmount to achieve additional protections for directory structures (which gpgdir does not alter).

Here is the complete ChangeLog: for the 1.9 release:
  • Changed --Obfuscate-filenames format to not include the gpgdir PID. This allows directories to be encrypted/decrypted under -O multiple times without creating new filenames (which would pollute encrypted directories under rsync to other systems). The new -O encrypted filename format is just "gpgdir_<num>.gpg".
  • Added PID locking against directories so that multiple gpgdir processes cannot operate against the same top-level directory simultaneously. This is useful for users that typically operate with multiple shells and might launch gpgdir from any of them.
]]> http://www.cipherdyne.org/blog/2008/05/linux.com-article-on-fwknop.html linux.com Article on fwknop 2008-05-21T21:01:30-05:00 Michael Rash At linux.com Bob Currier has written a featured article on fwknop. He gives a good overview of how Single Packet Authorization (SPA) is different from port knocking, and illustrates how to use fwknop to harden SSH communications. Here is a quote from the article:

   Single packet authorization distills the essence of the port knocking concept down to a single packet. Rather than sending a series of packets to predefined ports, single packet authorization encodes the "knock" within the payload of one packet. Once a proper key has been received, SPA applications modify firewall rules to allow access to the authenticated host. We'll examine how this is accomplished by installing and testing the Firewall KNock Operator, better known as fwknop.

There are several comments attached to the article from interested users, and additional discussion of both port knocking and SPA topics can be found at Sebastien Jeanquier's online forum. ]]> http://www.cipherdyne.org/blog/2008/05/site-update-wordpress-theme-without-running-wordpress.html Site Update - WordPress Theme without Running WordPress 2008-05-08T21:01:30-05:00 Michael Rash WordPress site WordPress has a loyal following in the blogosphere and is used to power thousands of websites and blogs, and several books have been written about it. A compelling measure of WordPress popularity is the number of downloadable themes that can provide an easily deployed and consistent look and feel to WordPress sites. So, when I was looking to update the presentation of the cipherdyne.org site, it seemed a natural step to consider a WordPress theme.
I've updated cipherdyne.org to use a slightly modified version of the Daleri Selection (v1.0) Theme by Andreas Viklund, but with a twist: I don't run actually Word Press. All of that PHP code and MySQL database backend is throwing a lot of complex code (and associated potential security vulnerabilities) at a problem that (in my case) can be solved in a simpler way. That is, I just want to provide an outlet for my software and publications, and maintain a blog for my corner of the security technology landscape. For this, I just need a themed website in plain HTML and CSS along with RSS and Atom feeds, and I need an effective way to manage all of the pages and create new blog posts. This is where perl, rsync, and subversion comes in.
The cipherdyne.org website and blog is managed by a set of custom perl scripts that perform each the following tasks:
  • Provide a way to create new blog posts and link them into the cipherdyne.org blog hierarchy. For example, the forward and backward links at the end of each post (such as this one) are auto-generated based on the posting date, and the permanent blog links are created from each post by the blog post title. The number of blog posts is summarized by date and organized into categories that can be browsed via the blog archives, and updates to the RSS and Atom feeds are automatically generated.
  • Validate HTML and XML page structure. This is exceeding easy through the use of the perl XML::Simple module. But, using the W3C HTML validator is more thorough, so I periodically use it as well to spot check various pages on cipherdyne.org.
  • Validate software release MD5 digests and GnuPG signatures (see the fwknop download page for example), and automatically correct if any digest is invalid.
  • Search through the entire cipherdyne.org website and flag any broken links (parsing the output of a recursive wget against the cipherdyne.org staging server make this easy).
  • Send blog pings out via the XMLRPC::Lite module to various blog aggregation services such as FeedBurner and Technorati.
  • Drive the entire cipherdyne.org site via Subversion for source control, and deploy with rsync to separate staging and production servers. When it comes to deploying a more complex piece of software for visualizing changes to the source code for any of the cipherdyne.org projects, my first choice is Trac, but when it comes to the main cipherdyne.org site, a set of HTML pages maintained by a consistent WordPress theme is sufficient.
]]> http://www.cipherdyne.org/blog/2008/04/hakin9-march-2008-issue.html Hakin9 March 2008 Issue 2008-04-19T21:01:30-05:00 Michael Rash Hakin9 March 2008 Issue The March issue of Hakin9 Magazine contains some noteworthy articles and interviews. First is an article written by Ryan Maple entitled "Best Practices for Secure Shell" which discusses various security measures that an administrator can use to heighten the security of SSH. These measures include (among others) forcing the usage of SSH protocol version 2, restricting the address (via the ListenAddress variable) that SSHD binds to from the default of 0.0.0.0, using tcpwrappers, and configuring SSHD to listen on a port other than tcp/22. Covering Single Packet Authorization would have been difficult to include in the same article, but a reference is made to another online article Knock, Knock, Knockin' on EnGarde's Door (with FWKNOP) that does discuss protecting SSH with SPA. Next, is Matt Jonkman's fifth part in his series "Writing IPS Rules". This article covers the byte_test keyword in the Snort rules language and how to use it to write signatures against length encoded protocols. In particular, Matt illustrates using byte_test to look for specific byte values at particular offsets derived from data within the DHCP protocol. The offsets themselves are determined by the data on the wire and therefore cannot simply be hard coded within a signature beforehand. For those who are interested in the latest IDS signatures from Matt, he has updated the online home of the Bleeding Edge Snort ruleset to Emerging Threats. Finally, both Marcus Ranum and Richard Bejtlich were interviewed for the March issue. I would like to see Hakin9 do a more in-depth interview of Bejtlich though since he always has an insightful perspective on computer security - particularly as demonstrated in his book The Tao of Network Security Monitoring. ]]> http://www.cipherdyne.org/blog/2008/04/fwknop-windows-ui-update.html fwknop Windows UI Update 2008-04-17T21:01:30-05:00 Michael Rash Sean Greven has released the next version of his fwknop UI for Windows systems. This release is compatible with all fwknop daemons that are configured to accept SPA packets that have been encrypted with the Rijndael symmetric cipher. The executable can be downloaded here, and the source code is also available here. Finally, here is a screenshot to illustrate generating an SPA packet from Windows 2000 running under VMware on an Ubuntu 7.10 system. The fwknopd daemon is running in --debug mode in the terminal on the right, and you can see the addition of an iptables ACCEPT rule to allow access to SSHD:

fwknop Window UI update ]]> http://www.cipherdyne.org/blog/2008/04/port-forwarding-via-single-packet-authorization.html Port Forwarding via Single Packet Authorization 2008-04-05T21:01:30-05:00 Michael Rash Digg How to Safely Connect to Your Closed Internal Systems via SPA and NAT
Port Forwarding via Single Packet Authorization Most port knocking or Single Packet Authorization implementations offer the ability to passively authenticate clients for access only to a locally running server (such as SSHD). That is, the daemon that monitors a firewall log or that sniffs the wire for port knock sequences or SPA packets can only reconfigure a local firewall to allow the client to access a local socket. This is usually accomplished by allowing the client to connect to the server port by putting an ACCEPT rule in the INPUT chain for iptables firewalls, or adding a pass rule for ipfw firewalls for the client source IP address. For local servers, this works well enough, but suppose that you ultimately want to access an SSH daemon that is running on an internal system? If the SPA software is deployed on a Linux gateway that is protecting a non-routable internal network and has a routable external IP address, it is inconvenient to first have to login to the gateway and then login to the internal system.
       Since the 1.9.2 release, fwknop has supported the creation of inbound port forwarding rules for iptables firewalls via the DNAT target in the PREROUTING chain after receiving a valid SPA packet. This allows direct access to internal systems on non-routable address space from the open Internet. Here is an example - suppose that you are running fwknop on Linux gateway system that is protecting an internal network 192.168.10.0/24 and has an external routable address of 11.1.1.1. Now, suppose that you are on travel somewhere (so your source IP address is not predictable), and you would like to access an SSH server that is running on the internal system 192.168.10.23. Finally, because fwknop is deployed, iptables is configured in a default-DROP stance against all attempts to connect with any SSH daemon; nmap cannot even tell that there is any SSH server listening. Access is granted only after a valid SPA packet is passively sniffed by the fwknopd daemon.

With inbound NAT support configured in fwknop (the config is mentioned below), here is an illustration of fwknop usage in order to gain direct access to SSHD on the 192.168.10.23 from an external network (note that the -R argument instructs the fwknop client to automatically resolve the current IP address - 12.2.2.2 in this case): 
[externalhost]$ fwknop -A tcp/22 --Forward-access 192.168.10.23,5001 \
-R -D 11.1.1.1

[+] Starting fwknop client (SPA mode)...
    Resolving external IP via: http://www.whatismyip.org/
    Got external address: 12.2.2.2

[+] Enter an encryption key. This key must match a key in the file
    /etc/fwknop/access.conf on the remote system.

Encryption Key:

[+] Building encrypted Single Packet Authorization (SPA) message...
[+] Packet fields:

        Random data:    6862733471944039
        Username:       root
        Timestamp:      1207404612
        Version:        1.9.3
        Type:           2 (FORWARD access mode)
        Access:         12.2.2.2,tcp/22
        Forward access: 192.168.10.23,5001
        SHA256 digest:  hE4zGafLtQiQiFrep+cSq/wVO7SQhwh65hmLr+ehtrw

[+] Sending 206 byte message to 11.1.1.1 over udp/62201...
Now, port 5001 on the external IP address is forwarded through to the SSH server on the internal 192.168.10.23 system, but only for the client IP 12.2.2.2: 
[externalhost]$ ssh -p 5001 mbr@11.1.1.1
Password:
[internalhost]$
Graphically, this scenario is illustrated by the following picture. The dotted lines represent the SPA packet from the fwknop client (which only needs to be sniffed by the fwknopd daemon running on the Linux gateway), and the solid arrows represent the SSH connection from the external client through to the internal SSH server.

DNAT SPA access to internal SPA server
On the Linux gateway system that is running the fwknop daemon, executing the following command illustrates the additions to the iptables policy to allow the SSH connection to be forwarded to the internal system. These firewall rules are automatically deleted after a 30 second timeout (this is tunable), but any existing SSH connection remains open through the use of the iptables connection tracking capability. 
[gateway]# fwknopd --fw-list
[+] Listing rules in fwknop chains...
Chain FWKNOP_INPUT (1 references)
 pkts bytes target prot opt in out source   destination

Chain FWKNOP_FORWARD (1 references)
 pkts bytes target prot opt in out source     destination
   19  2740 ACCEPT tcp  --  *  *   12.2.2.2   0.0.0.0/0    tcp dpt:22

Chain FWKNOP_PREROUTING (1 references)
 pkts bytes target prot opt in out source     destination
    1    60 DNAT   tcp  --  *  *   12.2.2.2   0.0.0.0/0    tcp \
dpt:5001 to:192.168.10.23:22
Finally, the /etc/fwknop/access.conf is configured like so to facilitate this example, and the ENABLE_IPT_FORWARDING variable is also enabled in the /etc/fwknop/fwknop.conf file: 
SOURCE: ANY;
OPEN_PORTS: tcp/22;
PERMIT_CLIENT_PORTS: N;
ENABLE_FORWARD_ACCESS: Y;
FW_ACCESS_TIMEOUT: 30;
KEY: aesdemokey;
In addition to explaining the inbound NAT support offered by fwknop, this blog post also announces the 1.9.3 release of fwknop. Here is the complete ChangeLog:
  • Added MASQUERADE and SNAT support to complement inbound DNAT connections for SPA packets that request --Forward-access to internal systems. This functionality is only enabled when both ENABLE_IPT_FORWARDING and ENABLE_IPT_SNAT are set, and is configured by two new variables IPT_MASQUERADE_ACCESS and IPT_SNAT_ACCESS which define the iptables interface to creating SNAT rules. The SNAT supplements of DNAT rules are not usually necessary because internal systems usually have a route back out to the Internet, but this feature accommodates those systems that do not have such a route. By default, the MASQUERADE target is used if ENABLE_IPT_SNAT is enabled because this means that the external IP does not have to be manually defined. However, the external IP can be defined by the SNAT_TRANSLATE_IP variable.
  • Added hex_dump() feature for fwknop client so that raw encrypted SPA packet data can be displayed in --verbose mode.
  • When ENABLE_IPT_FORWARDING is set, added a check for the value of the /proc/sys/net/ipv4/ip_forward file to ensure that the local system allows packets to be forwarded. Unless ENABLE_PROC_IP_FORWARD is disabled, then fwknopd will automatically set the ip_forward file to "1" if it is set to "0" (again, only if ENABLE_IPT_FORWARDING is enabled).
  • Minor bugfix to remove sys_log() call in legacy port knocking mode.
  • Minor bugfix to expand both the Id and Revision tags via the svn:keywords directive.
]]> http://www.cipherdyne.org/blog/2008/04/software-release-gpgdir-1.8.html Software Release - gpgdir-1.8 2008-04-04T21:01:30-05:00 Michael Rash Software Release - gpgdir-1.8 The 1.8 release of gpgdir is ready for download. This is a minor feature enhancement release, and here is the Here is the complete ChangeLog:
  • Updated the test suite to validate the gpgdir --Obfuscate-filenames mode to ensure that files are encrypted as "gpgdir_<pid>_<num>.gpg".
  • Minor bug fix to remove the .gpgdir_map_file in --Obfuscate-filenames mode after a successful decryption cycle.
  • Updated to version 0.36 of CPAN GnuPG::Interface module.
]]> http://www.cipherdyne.org/blog/2008/04/software-release-psad-2.1.2.html Software Release - psad-2.1.2 2008-04-03T21:01:30-05:00 Michael Rash Software Release - psad-2.1.2 The 2.1.2 release of psad is ready for download. This is a minor bugfix release to improve support for Linux distributions that report a timestamp in syslog messages just before any iptables log prefix. Here is an example of such syslog messages reported in "psad --Status output (sent to the psad mailing list by Erik Heidt): 
[+] iptables log prefix counters:
      "[ 5370.901768] Default_Drop________": 1
      "[ 9411.899552] Default_Drop________": 1
      "[ 7560.529042] Default_Drop________": 1
      "[ 6454.263813] Default_Drop________": 1
      "[ 7038.635371] Default_Drop________": 1
      "[ 5648.026038] Default_Drop________": 1
      "[ 1207.663709] Default_Drop________": 1
      "[  394.423573] Default_Drop________": 1
      "[  186.740237] Default_Drop________": 1
      "[ 9405.138521] Default_Drop________": 1
      "[ 5364.208693] Default_Drop________": 1
      "[ 7297.121141] Default_Drop________": 1
The bug caused the timestamp portion of the syslog messages to be included within the iptables log prefix strings, and because timestamps are nearly unique, this forced even the same iptables log prefix to also be treated as unique.
Here is the complete ChangeLog:
  • Bugfix to not include kernel timestamps in iptables log prefixes that contain spaces like "[ 65.026008] DROP" (bug reported by Erik Heidt).
  • Bugfix to skip non-resolved IP addresses (bug reported by Albert Whale)
  • Better p0f output in --debug mode to display when a passive OS fingerprint cannot be calculated based on iptables log messages that include tcp options (i.e., with --log-tcp-options when building a LOG rule on the iptables command line).
]]> http://www.cipherdyne.org/blog/2008/03/ids-signature-matching-with-iptables-psad-and-fwsnort.html IDS signature matching with iptables, psad, and fwsnort 2008-03-20T21:01:30-05:00 Michael Rash IDS signature matching with iptables, psad, and fwsnort The UK's Unix & Open Systems User Group has re-printed an article I wrote originally for the December, 2007 security issue of USENIX ;login: Magazine The article is entitled "IDS signature matching with iptables, psad, and fwsnort" and concentrates on how to use the iptables infrastructure in the Linux kernel as a source of intrusion detection data. That is, iptables offers many features (such as application layer string matching) that allow a significant fraction of Snort rules to be converted into iptables rules, and fwsnort automates the conversion process. The end result is an iptables policy that is looking for evidence of malicious traffic. Also covered in the article is the concept of log analysis with an emphasis on passive OS fingerprinting. The completeness of the iptables logging format - which even includes the options portion of the TCP header when the --log-tcp-options argument is given on the iptables command line when building a LOG rule - allows psad to run the same algorithm that p0f uses to passively fingerprint remote operating systems.

In other news, Carla Schroder has written an article on psad for Enterprise Networking Planet. She recommends running psad alongside Snort, which falls nicely within the principle of defense-in-depth in order to maintain a strong defensive stance. Also, Noah Schiffman has written an article on port knocking for Network World. He mentions the usage of port knocking within some malware applications as an authentication mechanism, and he also touches on Single Packet Authorization. ]]> http://www.cipherdyne.org/blog/2008/03/source-boston-advanced-linux-firewalls-talk-slides.html SOURCE Boston Advanced Linux Firewalls Talk Slides 2008-03-17T21:01:30-05:00 Michael Rash SOURCE Boston Advanced Linux Firewalls Talk Slides At the SOURCE Boston conference in Boston last week I gave talk entitled "Advanced Linux Firewalls" (slides). The conference attendance was good considering that this is the first year the conference was offered, and I look forward to next year. I managed to see a few talks, and two that stood out from the crowd were Roger Dingledine's talk "How To Make Tor Play Well With The Rest Of The Internet", and Andrew Jaquith's talk "Not Dead But Twitching: Anti-Virus Succumbs to the Scourge of Modern Malware". Roger highlighted several technology research and development areas for the Tor project, including the ability to use UDP instead of TCP for Tor virtual circuits. This is of particular interest to me, since it would mean that SPA packets could be routed over the Tor network without having to resort to the establishment of full TCP connections (which breaks the "single packet" part of "SPA"). Andrew gave some interesting perspectives on malware trends, including the fact that malware over time is becoming more targeted while at the same time exhibiting high variability. The end result is that malware authors are able to attack the weakest link in the creation of signatures for malware detection - the people that reverse engineer malware. Because human resources are scarce and slow when it comes to reverse engineering (there is no fully automated mechanism for this yet), malware authors are able to essentially perpetrate a DoS against vendors that offer malware detection. ]]> http://www.cipherdyne.org/blog/2008/03/fwknop-1.9.2-release-at-source-boston.html fwknop-1.9.2 Release at SOURCE Boston 2008-03-12T21:01:30-05:00 Michael Rash fwknop-1.9.2 Release at SOURCE Boston Today at the SOURCE Boston computer security conference I will give a talk entitled "Advanced Linux Firewalls" in which I will present many of the themes I discuss in my book published late last year by No Starch Press. This talk will also launch the 1.9.2 release of fwknop, and present several new features such as client-derived access timeouts, the ability to select any of several digest algorithms (SHA-256, SHA-1, or MD5) for replay attack detection, the removal of the Salted__ prefix in SPA packets encrypted with Rijndael, and blacklist IP exclusions for incoming SPA packets. Many of these features were implemented by the SPAPICT team as well as several other contributors, and I wish to thank all who participated in the fwknop development process.

      You can download fwknop-1.9.2 here, and for those interested in the changes in the fwknop-1.9.2 release, here is the complete ChangeLog:
  • Crypt::CBC adds the string "Salted__" to the beginning of the encrypted text (at least for how fwknop interfaces with Crypt::CBC), so the fwknop client was updated to delete the encoded version of this string "U2FsdGVkX1" before sending a Rijndael-encrypted SPA packet on the wire. The fwknopd server will add this string back in before decrypting. This makes it harder to write an IDS signature that looks for fwknop traffic; e.g. look for the default prefix string "U2FsdGVkX1" over UDP port 62201, which would work for fwknop clients < 1.9.2 (as long as the port number is not changed with --Server-port).
  • Added more granular source IP and allowed IP tests so that access to particular internal IP addresses can be excluded in --Forward-access mode. A new keyword "INTERNAL_NET_ACCESS" is now parsed from the access.conf file in order to implemented these restrictions.
  • (SPAPICT Group) Added BLACKLIST functionality to allow source IP addresses to easily be excluded from the authentication process.
  • (Grant Ferley) Submitted patch to handle SIGCHLD in IPTables::ChainMgr.
  • (Grant Ferley) Submitted patch to handle Linux "cooked" interfaces for packet capture (e.g. PPPoE interfaces).
  • (SPAPICT Group) Applied modified version of the client-defined access timeout patches submitted by the PICT SPA Group. There are two new message types to facilitate client timeouts; one for normal access mode, and the other for the FORWARD access mode. In the access.conf file, there is also a new variable "PERMIT_CLIENT_TIMEOUT" to allow each SOURCE stanza to allow client-defined timeouts or not.
  • (SPAPICT Group) Submitted patches to include support for the SHA1 digest algorithm for SPA packet replay attack detection. I modified these patches for maximum configurability (see the --digest-alg argument on the fwknop command line), and the ability to use the SHA256 algorithm as well. The default path to the /var/log/fwknop/md5sums file has been changed to /var/log/fwknop/digest.cache, and the default digest algorithm is now SHA256 (but this is tunable via the DIGEST_TYPE variable in the fwknop.conf file).
  • Added the Digest::SHA perl module in support of the SHA1 and SHA256 digest algorithms for replay attack detection and SPA message integrity.
  • Added full packet hex dumps (including packet headers) to fwknopd in --debug --verbose mode. This is to help diagnose packet sniffing issues over the loopback interface on Mac OS X (first reported by Sebastien Jeanquier).
  • (Test suite) Bugfix to ensure that the FWKNOP_DIR variable is set to the local output/ directory in several of the test config files in the test/conf/ directory.
  • (Test suite) Added several tests for configurable digest algorithms in support for the SHA256, SHA1, and MD5 digest changes made by the SPAPICT Group.
  • Updated the fwknop client to always call encode_base64() with the string to encode along with a second null-string argument to force all encoded data to not include line breaks.
  • Bugfix in install.pl to not test for the iptable command on non-Linux systems, and to not test for the ipfw command on systems that are Linux.
  • (Test suite) Updated to include the /proc/config.gz file so that the kernel config can be reviewed (not all Netfilter hooks are necessarily compiled in).
]]> http://www.cipherdyne.org/blog/2008/03/thwarting-distributed-ssh-brute-force-attempts.html Thwarting Distributed SSH Brute Force Attempts 2008-03-02T21:01:30-05:00 Michael Rash Thwarting Distributed SSH Brute Force Attempts The SANS Internet Storm Center has an interesting diary entry that discusses a set of distributed brute force attempts against SSHD originating from a specific Class C subnet 58.147.10.0/24. According to the diary entry, a random selection of IP addresses within this subnet (which is owned by an organization in Thailand) are probing for listening SSH daemons on the open Internet, but each IP address is limiting the number of probes to two. Presumably this is to fly beneath the radar of any threshold-based detection mechanisms that might automatically block the attempts to communicate with SSHD.
     After rummaging around in my iptables logfiles, I noticed the following two log entries that show exactly two SYN packets from the IP 58.147.10.209 and destined for one of my systems: 
Jun  3 14:55:06 minastirith kernel: DROP IN=eth0 OUT= \
MAC=00:13:d3:38:b6:e4:00:90:1a:a0:1c:ec:08:00 SRC=58.147.10.209 \
DST=71.N.N.N LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=11651 DF PROTO=TCP \
SPT=57473 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 \
OPT (020405780402080A7CCF5BC50000000001030302)
Jun  3 14:55:09 minastirith kernel: DROP IN=eth0 OUT= \
MAC=00:13:d3:38:b6:e4:00:90:1a:a0:1c:ec:08:00 SRC=58.147.10.209 \
DST=71.N.N.N LEN=60 TOS=0x00 PREC=0x00 TTL=44 ID=11653 DF PROTO=TCP \
SPT=57473 DPT=22 WINDOW=5840 RES=0x00 SYN URGP=0 \
OPT (020405780402080A7CCF677D0000000001030302)
These two packets date back to June of last year, so the 58.147.10.0/24 network has apparently been active for quite some time. So, beyond the whois information mentioned in the ISC diary entry, can we tell anything more about our friend at 58.147.10.209? Because I always use the --log-tcp-options command line argument when building my iptables LOG rules, we have nice hex dumps included in the log messages above that represent the options portion of the TCP header. Having this information is necessary in order to passively fingerprint the remote TCP stack with p0f. The psad project re-implements the p0f algorithm over iptables log messages - p0f requires raw packet data acquired via libpcap - so what does psad have to say about the remote TCP stack? 
[minastirith]# psad -A -m /var/log/messages --restrict-ip 58.147.10.209
[+] p0f(): 58.147.10.209 len: 60, frag_bit: 1, ttl: 44, win: 5840
[+] MSS: 1400, SACK, Timestamp: 2093964229, NOP, Win Scale: 2,
    Could not match 58.147.10.209 against any p0f signature.
Oh well, that is unfortunate. So, p0f does not seem to have a fingerprint that can characterize the remote TCP stack. I also looked through the passive OS fingerprinting database included with the ettercap project, but no luck there either. This was a manual process though because psad does not yet support the format of the fingerprints in ettercap, but this will be supported in the next release.
     In the meantime, the discussion in the diary entry advocates limiting the set of IP addresses that are allowed to talk to the SSH daemon, but ends with a comment stating that this is not always possible. I agree with this comment, and this is highlighted whenever you administer a system and you need access to SSHD from unpredictable source addresses (such as when travel is involved). A robust solution is to deploy Single Packet Authorization with a default-drop packet filter. Why let arbitrary source IP addresses communicate with your SSH daemon? ]]>