Michael Rash, Security Researcher

Port Knocking and SPA    [Summary View]

Software Release: fwknop-2.6.9

The 2.6.9 release of fwknop is available for download (or via the github release tag). Here is the complete ChangeLog:

  • (Jonathan Bennett) Added support for the SHA3 "Keccak" algorithm (specifically SHA3_256 and SHA3_512) for SPA HMAC and digest checking. Enabling SHA3 from the fwknop client command line is done with the -m option for the embedded SPA digest, or with the --hmac-digest-type argument for the HMAC. On the server side, SHA3_256 or SHA3_512 can be required for the incoming SPA packet HMAC via the HMAC_DIGEST_TYPE configuration variable in access.conf stanzas. The SHA3 implementation is from, Keyak and Ketje Teams, namely, Guido Bertoni, Joan Daemen, Michael Peeters, Gilles Van Assche and Ronny Van Keer - see:
  • (Damien Stuart) Added support for libnetfilter_queue so that fwknopd can acquire SPA packets via the NFQ target. This feature is enabled with a new command line switch --enable-nfq-capture for the configure script, and libpcap is not required in this mode. In support of capturing SPA packets via the NFQ target, new configuration variables have been added to the fwknopd.conf file: ENABLE_NFQ_CAPTURE, NFQ_INTERFACE, NFQ_PORT, NFQ_TABLE, NFQ_CHAIN, NFQ_QUEUE_NUMBER, and NFQ_LOOP_SLEEP.
  • (Vlad Glagolev) Added support for deriving the source IP from the X-Forwarded-For HTTP header when SPA packets are sent over HTTP connections.
  • Bug fix in command open/close cycle feature to ensure that the first successful match on a valid incoming SPA packet finishes all access.conf stanza processing. That is, no other stanzas should be looked at after the first match, and this is consistent with other SPA modes (such as basic access requests). This bug was reported by Jonathan Bennett.
  • (Jonathan Bennett) Various fixes and enhancements to the test suite to extend code coverage to new code, ensure valgrind bytes lost detection works for amount of memory less than 10 bytes, better timing strategy for fwknop client/server interactions, and more.

Single Packet Authorization and Third Party Devices

A major new feature in fwknop has been introduced today with the 2.6.8 release (github tag) - the ability to integrate with third-party devices. This brings SPA operations easily to any device or software that offers a command line interface. By default, the fwknop daemon supports four different firewalls: iptables, firewalld, ipfw, and PF. But, suppose you want to have fwknopd leverage ipset instead? Or, suppose you have an SSH pre-shared key between a Linux system and a Cisco router, and you want fwknopd (running on the Linux box) to control the ACL on the router for the filtering portion of SPA? Finally, suppose that you want a stronger measure of protection for an SSH daemon that may have been backdoored, and that runs on a proprietary OS where fwknopd can't be deployed natively? The sky is the limit, and I would be interested in hearing about other deployment scenarios.

These scenarios and many others are now supported with a new "command open/close cycle" feature in fwknop-2.6.8. Essentially, fwknopd has the ability to execute an arbitrary command upon receiving a valid SPA packet (the "open"), and then execute a different command after a configurable timeout (the "close"). This allows fwknopd to integrate with any third-party device or software if open and close commands can be defined for how to interact. These commands are specified on a per-stanza basis in the access.conf file, and a set of variable substitutions are supported such as '$SRC', '$PORT', '$PROTO', and '$CLIENT_TIMEOUT'. Naturally, the IP address, port, and protocol are authenticated and decrypted out a valid SPA packet - i.e., SPA packet headers are not trusted.

Let's see an example on a Linux system ("spaserver"). Here, we're going to have fwknopd interface with ipset instead of iptables. First, we'll create an ipset named fwknop_allow, and we'll link it into the local iptables policy. If a packet hits the fwknop_allow ipset and there is no matching source IP, then the DROP rule at the end of the iptables policy implements the default-drop policy. No userspace daemon such as SSHD can be scanned or otherwise attacked from remote IP addresses without first producing a valid SPA packet.
[spaserver]# ipset create fwknop_allow hash:ip,port timeout 30
[spaserver]# iptables -A INPUT -m conntrack --ctstate ESTABLISHED,RELATED -j ACCEPT
[spaserver]# iptables -A INPUT -m set --match-set fwknop_allow src,dst -j ACCEPT
[spaserver]# iptables -A INPUT -j DROP
Now, we create a stanza in the fwknop /etc/fwknop/access.conf file and fire up fwknopd like this:
[spaserver]# cat /etc/fwknop/access.conf
SOURCE            ANY
KEY_BASE64        <base64 string>
HMAC_KEY_BASE64   <base64 string>
CMD_CYCLE_OPEN    ipset add fwknop_allow $SRC,$PROTO:$PORT timeout $CLIENT_TIMEOUT

[spaserver]# service fwknopd start
With fwknopd running and iptables configured to drop everything except for IP communications that match the fwknop_allow ipset, let's use the fwknop client from a remote system "spaclient" to gain access to SSHD on the server for 30 seconds (note that the iptables conntrack module will keep the connection open after the SPA client IP is removed from the ipset). We'll assume that the encryption and HMAC keys have been previous shared between the two systems, and on the client these keys have been written to the "spaserver" stanza in the ~/.fwknoprc file:
[spaclient]$ fwknop -A tcp/22 -a -f 30 -n spaserver
[spaclient]$ ssh user@spaserver
So, behind the scenes after the SPA packet has been sent above, fwknopd on the server has authenticated and decrypted the SPA packet, and has executed the following ipset command. In this case, there is no need for a corresponding close command because ipset implements the timer provided by the client itself, so the client IP is deleted from the ipset automatically. (In other scenarios, the close command can be fully specified instead of using the string 'NONE' as we have above.) Here are the syslog messages that fwknopd has generated, along with the 'ipset list' command output to show the IP as a member of the set:
[spaserver]# grep fwknopd /var/log/syslog |tail -n 2
Dec 23 15:38:06 ubuntu fwknopd[13537]: (stanza #1) SPA Packet from IP: received with access source match
Dec 23 15:38:06 ubuntu fwknopd[13537]: [] (stanza #1) Running CMD_CYCLE_OPEN command: /sbin/ipset add fwknop_allow,6:22 timeout 30

[spaserver]# ipset list
Name: fwknop_allow
Type: hash:ip,port
Revision: 5
Header: family inet hashsize 1024 maxelem 65536 timeout 30
Size in memory: 224
References: 0
Members:,tcp:22 timeout 27
In addition to the ability to swap out the existing firewall with a completely different filtering infrastructure, there are other notable features and fixes in the 2.6.8 release. The most important of these is a new feature implemented by Jonathan Bennett (and suggested by Hank Leininger in github issue #62) that allows access.conf files to be imported via a new '%include' directive. This can be advantageous in some scenarios by letting non-privledged users define their own encryption and authentication keys for SPA operations. This way, users do not need write permissions to the main /etc/fwknop/access.conf file to change keys around or define new ones.

The complete ChangeLog is available here, and the current test suite has achieved 90.7% code coverage (measured by lines).

Software Release: fwknop-2.6.7

fwknop-2.6.7 software release The 2.6.7 release of fwknop is available for download (or via the github release tag). This release adds significant support for running commands delivered by SPA packets via 'sudo' on the server, and this allows the powerful 'sudoers' syntax to filter commands that remote users are allowed to execute.

In addition, the --key-gen (key generation) mode has been added to fwknopd. This will allow better integration with Jonathan Bennett's Fwknop2 Android client - particularly when only the fwknopd server is installed on a system (as is usually the case for embedded distributions such as OpenWRT). Further, Jonathan contributed a console QR code generator, so that fwknop encryption and HMAC keys can be imported into the Fwknop2 Android client via the phone's camera. Here is an example: $ fwknopd --key-gen | ./extras/console-qr/ fwknop QR key code In other news, Jonathan and I will be giving a lengthy interview on Single Packet Authorization with fwknop for the FLOSS Weekly show organized by the venerable Randal Schwartz of perl fame. Tune in September 2nd at 11am Eastern time.

As usual, fwknop has a Coverity Scan score of zero, and the code coverage report achieved by the 2.6.7 test suite is available here. Note that the fwknop test suite is now achieving 90% code coverage counted by lines, and 100% code coverage counted by functions. This reflects the commitment the fwknop project makes towards rigorous security and testing quality.

Here is the complete ChangeLog for fwknop-2.6.7:

  • [server] When command execution is enabled with ENABLE_CMD_EXEC for an access.conf stanza, added support for running commands via sudo. This was suggested by Github user 'freegigi' (issue #159) as a means to provide command filtering using the powerful sudoers syntax. This feature is implemented by prefixing any incoming command from a valid SPA packet with the sudo command along with optional user and group requirements as defined by the following new access.conf variables: ENABLE_CMD_SUDO_EXEC, CMD_SUDO_EXEC_USER, and CMD_SUDO_EXEC_GROUP.
  • [server] Kevin Layer reported a bug to the fwknop mailing list that simultaneous NAT access for two different access.conf stanza was not functioning properly. After some diagnosis, this was a result of rule_exists() not properly detecting and differentiating existing DNAT rules from new ones with different port numbers when 'iptables -C' support is not available. This was against iptables-1.4.7, and has been fixed in this release of fwknop (tracked as issue #162).
  • [server] Added --key-gen to fwknopd. This feature was suggested by Jonathan Bennett, and will help with ease of use efforts. The first platform to take advantage of this will likely be OpenWRT thanks to Jonathan.
  • [server] By default, fwknopd will now exit if the interface that it is sniffing goes down (patch contributed by Github user 'sgh7'). If this happens, it is expected that the native process monitoring feature in things like systemd or upstart will restart fwknopd. However, if fwknopd is not being monitored by systemd, upstart, or anything else, this behavior can be disabled with the EXIT_AT_INTF_DOWN variable in the fwknopd.conf file. If disabled, fwknopd will try to recover when a downed interface comes back up.
  • [extras] Added a script from Jonathan Bennett at extras/console-qr/ to generate QR codes from fwknopd access.conf keys.
  • [build] Added --with-firewalld to the autoconf configure script. This is a synonym for --with-firewall-cmd to avoid confusion. Some package maintainers use --with-firewalld to build fwknop.

Android Fwknop2 Client and OpenWRT

Jonathan Bennett's new Android 'Fwknop2' client is ready for prime time. It is available now in the Google Play store as well as on F-Droid, and Jonathan put together a nice video demonstration of Fwknop2 being used to access SSHD on a router running OpenWRT. Also demonstrated is the usage of NAT to transparently access SSHD running on a system behind the router. This illustrates the ability fwknopd offers for creating inbound NAT rules for external SPA clients - something that is, to my knowledge, unique to the fwknop in the world of SPA software. Finally, in an innovative twist, the Fwknop2 client can read encryption and authentication keys from a QR code via the phone's camera. This simplifies the task of getting longer keys - particularly those that are base64-encoded - to the phone for SPA operations.

New Android Single Packet Authorization Client: Fwknop2

Fwknop2 on Android After a long while without updates to the fwknop clients on Android or iOS, I'm excited to announce that Jonathan Bennett has developed an entirely new client for Android called Fwknop2. This client adds significant features such as the ability to save configurations (including both encryption and authentication keys), proper handling of base64 encoded keys, and support for manually specified IP addresses to be allowed through the remote firewall. Further, Fwknop2 integrates with JuiceSSH so that an SSH connection can seamlessly be launched right after the SPA packet is sent. Finally, in an interesting twist, Fwknop2 will be able to read encryption and HMAC keys via a QR code via the camera. The QR code itself is generated from key material in --key-gen mode (which is currently available in the standard fwknop client and will be available in the fwknopd server in the next release).

Fwknop2 will be available on both the Google Play store and via F-Droid within the next few hours, and in the meantime the APK is available on github here.

Below are a couple of screenshots of the new Android client in action - these are from an Android VM running under Parallels on Mac OS X (Yosemite): fwknop Android app fwknop Android app

NAT and Single Packet Authorization

People who use Single Packet Authorization (SPA) or its security-challenged cousin Port Knocking (PK) usually access SSHD running on the same system where the SPA/PK software is deployed. That is, a firewall running on a host has a default-drop policy against all incoming SSH connections so that SSHD cannot be scanned, but a SPA daemon reconfigures the firewall to temporarily grant access to a passively authenticated SPA client: SPA and basic SSH access This works well enough, but both port knocking and SPA work in conjunction with a firewall, and "important" firewalls are usually gateways between networks as opposed to just being deployed on standalone hosts. Further, Network Address Translation (NAT) is commonly used on such firewalls (at least for IPv4 communications) to provide Internet access to internal networks that are on RFC 1918 address space, and also to allow external hosts access to services hosted on internal systems.

The prevalence of NAT suggests that SPA should integrate strongly with it. Properly done, this would extend the notion of SPA beyond just supporting the basic feature of granting access to a local service. To drive this point home, and to see what a NAT-enabled vision for SPA would look like, consider the following two scenarios:

  1. A user out on the Internet wants to leverage SPA to access an SSH daemon that is running on a system behind a firewall. One option is to just use SPA to access SSHD on the firewall first, and then initiate a new SSH connection to the desired internal host from the firewall itself. However, this is clunky and unnecessary. The SPA system should just integrate with the NAT features of the firewall to translate a SPA-authenticated incoming SSH connection through to the internal host and bypass the firewall SSH daemon altogether: SPA and DNAT SSH access to internal host

  2. A local user population is behind a firewall that is configured to block all access by default from the internal network out to the Internet. Any user can acquire an IP on the local network via DHCP, but gaining access to the Internet is only possible after authenticating to the SPA daemon running on the firewall. So, instead of gaining access to a specific service on a single IP via SPA, the local users leverage SPA to gain access to every service on every external IP. In effect, the firewall in this configuration does not trust the local user population, and Internet access is only granted for users that can authenticate via SPA. I.e., only those users who have valid SPA encryption and HMAC keys can access the Internet: SPA and SNAT to external hosts
(A quick note on the network diagrams above - each red line represents a connection that is only possible to establish after a proper SPA packet is sent.)

Both of the above scenarios are supported with fwknop-2.6.6, which has been released today. So far, to my knowledge, fwknop is the only SPA/PK implementation with any built-in NAT support. The first scenario of gaining access to an internal service through the firewall has been supported by fwknop for a long time, but the second is new for 2.6.6. The idea for using SPA to gain access to the Internet instead of just for a specific service was proposed by "spartan1833" to the fwknop mailing list, and is a powerful extension of SPA into the world of NAT - in this case, SNAT in iptables parlance.

Before diving into the technical specifics, below is a video demonstration of the NAT capabilities of fwknop being used within Amazon's AWS cloud. This shows fwknop using NAT to turn a VPC host into a new gateway within Amazon's own border controls for the purposes of accessing SSH and RDP running on other internal VPC hosts. So, this illustrates the first scenario above. In addition, the video shows the usage of SPA ghost services to NAT both SSH and RDP connections through port 80 instead of their respective default ports. This shows yet another twist that strong NAT integration makes possible in the SPA world.

Now, let's see a practical example. This is for the second scenario above where a system with the fwknop client installed is on a network behind a default-drop firewall running the fwknop daemon, and the new SNAT capabilities are used to grant access to the Internet.

First, we fire up fwknopd on the firewall after showing the access.conf and fwknopd.conf files (note that some lines have been abbreviated for space):
[firewall]# cat /etc/fwknop/access.conf
SOURCE                      ANY
KEY_BASE64                  wzNP62oPPgEc+k...XDPQLHPuNbYUTPP+QrErNDmg=
HMAC_KEY_BASE64             d6F/uWTZmjqYor...eT7K0G5B2W9CDn6pAqqg6Oek=
FW_ACCESS_TIMEOUT           1000
DISABLE_DNAT                Y
FORWARD_ALL                 Y

[firewall]# cat /etc/fwknop/fwknopd.conf
ENABLE_IPT_SNAT             Y;

[firewall]# fwknopd -i eth0 -f
Starting fwknopd
Added jump rule from chain: FORWARD to chain: FWKNOP_FORWARD
Added jump rule from chain: POSTROUTING to chain: FWKNOP_POSTROUTING
iptables 'comment' match is available
Sniffing interface: eth3
PCAP filter is: 'udp port 62201'
Starting fwknopd main event loop.
With the fwknopd daemon up and running on the firewall/SPA gateway system, we now run the client to gain access to the Internet after showing the "[internet]" stanza in the ~/.fwknoprc file:
[spaclient]$ cat ~/.fwknoprc
KEY_BASE64                  wzNP62oPPgEc+k...XDPQLHPuNbYUTPP+QrErNDmg=
HMAC_KEY_BASE64             d6F/uWTZmjqYor...eT7K0G5B2W9CDn6pAqqg6Oek=
ACCESS                      tcp/22  ### ignored by FORWARD_ALL
USE_HMAC                    Y
ALLOW_IP                    source

[spaclient]$ nc -v 80
### nothing comes back here because the SPA packet hasn't been sent yet
### and therefore everything is blocked (except for DNS in this case)
[spaclient]$ fwknop -n internet
[spaclient]$ nc -v 80
Connection to 80 port [tcp/http] succeeded!
Back on the server, below are the rules that are added to grant Internet access to the spaclient system. Note that all ports/protocols are accepted for forwarding through the firewall, and an SNAT rule has been applied to the spaclient source IP of
(stanza #1) SPA Packet from IP: received with access source match
Added FORWARD ALL rule to FWKNOP_FORWARD for -> */*, expires at 1429387561
Added SNAT ALL rule to FWKNOP_POSTROUTING for -> */*, expires at 1429387561
Most users think of port knocking and Single Packet Authorization as a means to passively gain access to a service like SSHD running on the same system as the PK/SPA software itself. This notion can be greatly extended through strong integration with NAT features in a firewall. If the SPA daemon integrates with SNAT and DNAT (in the iptables world), then both internal services can be accessed from outside the firewall, and Internet access can be gated by SPA too. The latest release of fwknop supports both of these scenarios, and please email me or send me a message on Twitter @michaelrash if you have any questions or comments.

Single Packet Authorization Threat Modeling

fwknop Threat Modeling Last week there was an interesting question posed by Peter Smith to the fwknop mailing list that focused on running fwknop in UDP listener mode vs. using libpcap. I thought it would be useful to turn this into a blog post, so here is Peter's original question along with my response:

"...I want to deploy fwknop on my server, but I'm not sure If I should use the UDP listener mode or libpcap. At first UDP listener mode seems to be the choice, because I don't have to compile libpcap. However, I then have to open a port in the firewall. Thinking about this, I get the feeling that I'm defeating the purpose of using SPA, by allowing Internet access to a privileged processe.

If an exploitable security issue is found, even though fwknop remains passive and undiscoverable, an attacker could blindly send his exploit to random ports on servers he suspects running fwknopd, and after maximum 65535 tries he would have root access. I'm not a programmer, so I can't review the code of fwknop or SSH daemon, but if both is equally likely of having security issues, I might as well just allow direct access to the SSH daemon and skip using SPA.

Is my point correct?..."

First, let me acknowledge your point as an important issue to bring up. If someone finds a vulnerability in fwknopd, it doesn't matter whether fwknopd is collecting packets via UDP sockets or from libpcap (assuming we're talking about a vulnerability in fwknopd itself). The mere processing of network traffic is the problem.

So, why run fwknopd at all?

This is something of a long-winded answer, but I want to be thorough. I'll likely not settle the issue with this response, but it's good to start the discussion.

Starting with your example above with SSHD open to the world, suppose there is a vulnerability in SSHD. What does the exploit model look like? Well, an attacker armed with an exploit can trivially find the SSH daemon with any scanner, and from there a single TCP connection is sufficient for exploitation. I would argue that a primary enabling factor benefiting the attacker in this scenario is that vulnerable targets that listen on TCP sockets are so easy to find. The combination of scannable services + zero-day exploits is just too nice for an attacker. Several years ago I personally had a system running SSHD compromised because of this. (At the time, it was my fault for not patching SSHD before I put it out on the Internet, but still - the system was compromised within about 12 hours of being online.)

Now, in the fwknopd world, assuming a vulnerability, what would exploitation look like? Well, targets aren't scannable as you say, so an attacker would have to guess that a target is running fwknopd and the corresponding port on which it watches/listens for SPA packets [1]. Forcing an attacker to send thousands of packets to different ports (assuming a custom non-default port instead of UDP port 62201 that fwknopd normally watches) is likely a security benefit in contrast to an obviously targetable service. That is, sending tons of SPA exploit packets to different ports is a common pattern that is quite noisy and is frequently flagged by IDS's, firewalls, SIEM's, and flow analysis engines. How many systems could an attacker target with such heavyweight scans before the attacker's own ISP would notice? Or before the attacker's IP is included within one of the Emerging Threats compromised hosts lists? Or within DShield as a known bad actor? 10 million? How many of these are actually running fwknopd? I know there are some spectacular scanning results out there, so it's really hard to quantify this, but either way there is a big difference between sending thousands of > 100-byte UDP packets to each target vs. a port sweep for TCP/22.

Further, when a system is literally blocking every incoming packet [2], an attacker can't even necessarily be sure there is any target connected to the network at all. Many attackers would likely move on fairly quickly from a system that is simply returning zero data to some other system.

In contrast, for a service that advertises itself via TCP, an attacker immediately knows - with 100% certainty - there is a userspace daemon with a set of functions that they can interact with. This presents a risk. In my view, this risk is greater than the risk of running fwknopd where an attacker gets no data.

Another fair question to ask is about the architecture of fwknopd. When an HMAC is used (this will be the default in a future release), fwknopd reads SPA packet data, computes an HMAC, and does nothing else if the HMAC does not match. This is an effort to try to stay faithful to a simplistic design, and to minimize the functions an attacker can interact with - even for packets that are blindly sent to the correct port where fwknopd is watching. Thus, most functions in fwknopd, including those that parse user-supplied data and hence where bugs are more likely to crop up, are not even accessible without first passing the HMAC which is a cryptographically strong test. When libpcap is also eliminated through the use of UDP, not even libpcap functions are in the mix [3]. In other words, the security of fwknop does not rely on not being discoverable or scannable - it is merely a nice side effect of not using TCP to gather data from the network.

To me, another way to think of fwknopd in UDP mode is that it provides a lightweight generic UDP authenticator for TCP-based services. Suppose that SSHD had a design built-in where it would bind to a UDP socket, wait for a similar authentication packet as SPA provides, and then switch over to TCP. In this case, there would not be a need for SPA because you would get the best of both worlds - non-scannability [4] plus everything that TCP provides for reliable data delivery at the same time. SPA in UDP listening mode is an effort to bridge these worlds. Further, there is nothing that ties fwknopd to SSH. I know people who expose RDP to the Internet for example. Personally, I'm quite confident there are fewer vulnerabilities in fwknopd than something like RDP. Even if there isn't a benefit in terms of the arguments above in terms of service concealment and forcing an attacker to be noisy, my bet is that fwknopd - even if it advertised itself via TCP - would provide better security than RDP or other services derived from massive code bases.

Now, beyond all of the above, there are some additional blog posts and other material that may interest some readers:

[1] If an attacker is in the position to watch legitimate SPA packets from an existing client then this guesswork can be largely eliminated. However, even in this case, if fwknopd is sniffing via libpcap (so not using the UDP only mode), there is no requirement for fwknopd to be running on the system where access is actually granted. It only needs to be able to sniff packets somewhere on the routing path of the destination IP chosen by the client. I mention this to illustrate that it may not be obvious to an attacker where a targetable fwknopd instance runs even when SPA packets can be monitored. Also, there aren't too many attackers who are in this position. At least, the number of attackers in this position is _far_ lower than those people (everyone) who are in a position to discover a vulnerable service from any system worldwide with a single TCP connection.

[2] As you point out, in UDP-only mode, the firewall must allow incoming packets to the UDP port where fwknopd is listening. But, given that fwknopd never sends anything back over this port, it remains indistinguishable to every other filtered port.

[3] There are more things built into the development process that may be worth noting too that heighten security such as the use of Coverity, AFL fuzzing, valgrind, etc., but these probably takes us too far afield from the topic at hand. Also, there are some roadmap items that have not been implemented yet (such as privilege separation) that will make the architecture even stronger.

[4] Assuming a strong firewall stance against incoming UDP packets to other ports so one cannot infer service existence because UDP/22 doesn't respond to a scan but other ports respond with an ICMP port unreachable, etc.

RAM Disks and Saving Your SSD From AFL Fuzzing

The American Fuzzy Lop fuzzer has become a critical tool for finding security vulnerabilities in all sorts of software. It has the ability to send fuzzing data through programs on the order of hundreds of millions of executions per day on a capable system, and can certainly put strain on your hardware and OS. If you are fuzzing a target program with the AFL mode where a file is written and the target binary reads from this file, then AFL is going to conduct a huge number of writes to the local disk. For a solid-state drive this can reduce its life expectancy because of write amplification.

My main workstation is a Mac running OS X Yosemite, and I run a lot of Linux, FreeBSD, and OpenBSD virtual machines under Parallels for development purposes. The drive on this system is an SSD which keeps everything quite fast, but I don't want to prematurely shorten its life through huge AFL fuzzing runs. Normally, I'm using AFL to fuzz fwknop from an Ubuntu-14.10 VM, and what is needed is a way to keep disk writes down. The solution is to use a RAM disk from within the VM.

First, from the Ubuntu VM, let's get set up for AFL fuzzing and show what the disk writes look like without using a RAM disk from the perspective of the host OS X system. This assumes AFL 0.89 has been installed already and is in the current path:
$ git clone fwknop.git
$ cd fwknop.git
$ ./
$ cd test/afl/
$ ./compile/
We're not running AFL yet. Now, from the Mac, launch the Activity Monitor (under Applications > Utilities) and look at current disk utilization: AFL not running disk writes So, not terrible - currently 31 writes per second at the time the snapshot was taken, and that includes OS X itself and the VM at the same time. But, now let's fire up AFL using the digest cache wrapper on the Ubuntu VM (the main AFL text UI is not shown for brevity):
$ ./fuzzing-wrappers/
[+] All right - fork server is up.
[+] All test cases processed.

[+] Here are some useful stats:

    Test case count : 1 favored, 0 variable, 1 total
           Bitmap range : 727 to 727 bits (average: 727.00 bits)
                   Exec timing : 477 to 477 us (average: 477 us)

[+] All set and ready to roll!
And now let's take a look at disk writes again from OS X: AFL no RAM disk writes Whoa, that's a massive difference - nearly two orders of magnitude. AFL has caused disk writes to spike to over 2,700 per second with total data written averaging at 19.5MB/sec. Long term fuzzing at this level of disk writes would clearly present a problem for the SSD - AFL frequently needs to be left running for days on end in order to be thorough. So, let's switch everything over to use a RAM disk on the Ubuntu VM instead and see if that reduces disk writes:
# mkdir /tmp/afl-ramdisk && chmod 777 /tmp/afl-ramdisk
# mount -t tmpfs -o size=512M tmpfs /tmp/afl-ramdisk
$ mv fwknop.git /tmp/afl-ramdisk
$ cd /tmp/afl-ramdisk/fwknop.git/test/afl/
$ ./fuzzing-wrappers/
Here is disk utilization once again from the Mac: AFL RAM disk writes We're back to less than 10 writes per second to the SSD even though AFL is going strong on the Ubuntu VM (not shown). The writes for the previous fuzzing run are still shown to the left of the graph (since they haven't quite aged out yet when the screenshot was taken), and new writes are so low they don't even make it above the X-axis at this scale. Although the total execs per second - about 2,000 - achieved by AFL is not appreciably faster under the RAM disk, the main benefit is that my SSD will last a lot longer. For those that don't run AFL underneath a VM, a similar strategy should still apply on the main OS. Assuming enough RAM is available for whatever software you want to fuzz, just create a RAM disk and run everything from it and extend the life of your hard drive in the process.

Software Release: fwknop-2.6.4

fwknop-2.6.4 software release The 2.6.4 release of fwknop is available for download. New functionality has been developed for 2.6.4, including a new UDP listener mode to remove libpcap as a dependency for fwknopd, support for firewalld on recent versions of Fedora, RHEL, and Centos (contributed by Gerry Reno), and support for Michal Zalewski's 'American Fuzzy Lop' fuzzer. Further, on systems where execvpe() is available, all system() and popen() calls have been replaced so that the shell is not invoked and no environment is used. As usual, fwknop has a Coverity Scan score of zero, and the code coverage report achieved by the 2.6.4 test suite is available here.

Here is the complete ChangeLog for fwknop-2.6.4:

  • [server] Added a UDP server mode so that SPA packets can be acquired via UDP directly without having to use libpcap. This is an optional feature since it opens a UDP port (and therefore requires the local firewall be opened for communications to this port), but fwknopd is careful to never send anything back to a client that sends data to this port. So, from the perspective of an attacker or scanner, fwknopd remains invisible. This feature is enabled in fwknopd either with a new command line argument --udp-server or in the fwknopd.conf file with the ENABLE_UDP_SERVER variable. When deployed in this mode, it is advisable to recompile fwknop beforehand with './configure --enable-udp-server' so that fwknopd does not link against libpcap.
  • [server] Replaced all popen() and system() calls with execvpe() with no usage of the environment. This is a defensive measure to not make use of the shell for firewall command execution, and is supported on systems where execvpe() is available.
  • (Gerry Reno) Added support for firewalld to the fwknopd daemon on RHEL 7 and CentOS 7. This is implemented using the current firewalld '--direct --passthrough' capability which accepts raw iptables commands. More information on firewalld can be found here:
  • [server] Added support for the 'American Fuzzy Lop' (AFL) fuzzer from Michal Zalewski. This requires that fwknop is compiled with the '--enable-afl-fuzzing' argument to the configure script as this allows encryption/digest short circuiting in a manner necessary for AFL to function properly. The benefit of this strategy is that AFL can fuzz the SPA packet decoding routines implemented by libfko. See the test/afl/ directory for some automation around AFL fuzzing.
  • (Bill Stubbs) submitted a patch to fix a bug where fwknopd could not handle Ethernet frames that include the Frame Check Sequence (FCS) header. This header is four bytes long, and is placed at the end of each Ethernet frame. Normally the FCS header is not visible to libpcap, but some card/driver combinations result in it being included. Bill noticed this on the following platform: BeagleBone Black rev C running 3.8.13-bone50 #1 SMP Tue May 13 13:24:52 UTC 2014 armv7l GNU/Linux
  • [client] Bug fix to ensure that a User-Agent string can be specified when the fwknop client uses wget via SSL to resolve the external IP address. This closes issue #134 on github reported by Barry Allard. The fwknop client now uses the wget '-U' option to specify the User-Agent string with a default of "Fwknop/<version>". In addition, a new command line argument "--use-wget-user-agent" to allow the default wget User-Agent string to apply instead.
  • [python module] When an HMAC key is passed to spa_data_final() then default to HMAC SHA256 if no HMAC mode was specified.

Software Release: fwknop-2.6.3

fwknop-2.6.3 software release The 2.6.3 release of fwknop is available for download. The emphasis in this release is maximizing code coverage through a new python SPA packet fuzzer, and also on fault injection testing with the excellent fault injection library libfiu developed by Alberto Bertogli. Another important change in 2.6.3 is all IP resolution lookups in '-R' mode now happen over SSL to make it harder for an adversary to mount a MITM attack on the resolution lookup. As always, manually specifying the IP to allow through the remote firewall is safer than relying on any network communication - even when SSL would be involved.

Here is the complete ChangeLog for fwknop-2.6.3:

  • [client] External IP resolution via '-R' (or '--resolve-ip-http') is now done via SSL by default. The IP resolution URL is now '', and a warning is generated in '-R' mode whenever a non-HTTPS URL is specified (it is safer just to use the default). The fwknop client leverages 'wget' for this operation since that is cleaner than having fwknop link against an SSL library.
  • Integrated the 'libfiu' fault injection library available from This feature is disabled by default, and requires the --enable-libfiu-support argument to the 'configure' script in order to enable it. With fwknop compiled against libfiu, fault injections are done at various locations within the fwknop sources and the test suite verifies that the faults are properly handled at run time via test/fko-wrapper/fko_fault_injection.c. In addition, the libfiu tool 'fiu-run' is used against the fwknop binaries to ensure they handle faults that libfiu introduces into libc functions. For example, fiu-run can force malloc() to fail even without huge memory pressure on the local system, and the test suite ensures the fwknop binaries properly handle this.
  • [test suite] Integrated a new python fuzzer for fwknop SPA packets (see test/ This greatly extends the ability of the test suite to validate libfko operations since SPA fuzzing packets are sent through libfko routines directly (independently of encryption and authentication) with a special 'configure' option --enable-fuzzing-interfaces. The python fuzzer generates over 300K SPA packets, and when used by the test suite consumes about 400MB of disk. For reference, to use both the libfiu fault injection feature mentioned above and the python fuzzer, use the --enable-complete option to the test suite.
  • [test suite] With the libfiu fault injection support and the new python fuzzer, automated testing of fwknop achieves 99.7% function coverage and 90.2% line coverage as determined by 'gcov'. The full report may be viewed here:
  • [server] Add a new GPG_FINGERPRINT_ID variable to the access.conf file so that full GnuPG fingerprints can be required for incoming SPA packets in addition to the abbreviated GnuPG signatures listed in GPG_REMOTE_ID. From the test suite, an example fingerprint is:
    GPG_FINGERPRINT_ID     00CC95F05BC146B6AC4038C9E36F443C6A3FAD56
  • [server] When validating access.conf stanzas make sure that one of GPG_REMOTE_ID or GPG_FINGERPRINT_ID is specified whenever GnuPG signatures are to be verified for incoming SPA packets. Signature verification is the default, and can only be disabled with GPG_DISABLE_SIG but this is NOT recommended.
  • [server] Bug fix for PF firewalls without ALTQ support on FreeBSD. With this fix it doesn't matter whether ALTQ support is available or not. Thanks to Barry Allard for discovering and reporting this issue. Closes issue #121 on github.
  • [server] Bug fix discovered with the libfiu fault injection tag "fko_get_username_init" combined with valgrind analysis. This bug is only triggered after a valid authenticated and decrypted SPA packet is sniffed by fwknopd:
    ==11181== Conditional jump or move depends on uninitialised value(s)
    ==11181==    at 0x113B6D: incoming_spa (incoming_spa.c:707)
    ==11181==    by 0x11559F: process_packet (process_packet.c:211)
    ==11181==    by 0x5270857: ??? (in /usr/lib/x86_64-linux-gnu/
    ==11181==    by 0x114BCC: pcap_capture (pcap_capture.c:270)
    ==11181==    by 0x10F32C: main (fwknopd.c:195)
    ==11181==  Uninitialised value was created by a stack allocation
    ==11181==    at 0x113476: incoming_spa (incoming_spa.c:294)
  • [server] Bug fix to handle SPA packets over HTTP by making sure to honor the ENABLE_SPA_OVER_HTTP fwknopd.conf variable and to properly account for SPA packet lengths when delivered via HTTP.
  • [server] Add --test mode to instruct fwknopd to acquire and process SPA packets, but not manipulate firewall rules or execute commands that are provided by SPA clients. This option is mostly useful for the fuzzing tests in the test suite to ensure broad code coverage under adverse conditions.