cipherdyne.org

07 March, 2012

After a widely publicized hack of github (now fixed), I thought it would be a good idea to ensure that the cipherdyne.org git repositories remain secure on both github and on the cipherdyne.org webserver. The techniques in the blog post may not be well suited to large git repositories with a lot of different people able to commit code, but for my repositories these checks provide a fairly high level of confidence that no malicious code has been introduced. First, the github hack was made possible through a "mass assignment" vulnerability in Ruby on Rails, and would have permitted an attacker to gain admin privileges to any project on github. Once admin access is acquired, an attacker would be in a position to do anything to the underlying code base - including adding new code that implements "undocumented features".

Now, in order to add a backdoor into a code base on github, what would an attacker need to do?

Altering the code base for a project would need to be done through standard git operations as a new commit - i.e., make code changes to a local copy, git add ..., git commit ... - as opposed to manually editing previously committed code in the git repository itself. This is because every commit must match a corresponding SHA1 hash according to git's object model, and a SHA1 collision such that the bogus data is also working code would be "computationally difficult" to say the least (attacks against SHA1 not withstanding). As a basic check, one can create a git repository for testing purposes, write a random byte to a random position within the .git/objects/pack/*.pack file and then try to clone it to see what happens:

error: packfile ./objects/pack/pack-9c886ed427d9a7538093f09edf516a0a718201ac.pack does not match index
error: packfile ./objects/pack/pack-9c886ed427d9a7538093f09edf516a0a718201ac.pack cannot be accessed
fatal: git upload-pack: cannot find object 7e8e48412ff985461095a09874059e955145d513:
fatal: The remote end hung up unexpectedly

The repository has essentially been corrupted and the clone operation fails.

Ok, so a malicious code modification would most likely need to be done via an entirely new commit. This could certainly be done by an attacker, but anyone who has cloned the repository would be able to see the change. For a large, highly active project without rigorous code review and committer hierarchy, it is conceivable that such a change might just get lost within the noise of lots of commits. After all, a human would need to review the change and recognize it as being malicious.

Could such a malicious code change affect the cipherdyne.org projects? In a word, "no", and here's why: all commits to the cipherdyne.org code bases are pushed to github from private git repositories on a dedicated system that is not generally accessible, and git pull ... is only done occasionally and every change comes from a known source and is reviewed. The accessible non-private git repositories on cipherdyne.org are mirrors of the github repositories, so if a malicious change were introduced into github, then they would have this change too. The private repositories would still be safe however. So, given this work flow, what I need is a way to verify that there are no commits in either the github repositories or cipherdyne.org mirrors that are not in private repositories. Further, I would like to be able to verify this without having to push or pull code into the private repositories (so I can regularly check at any time without any modifications coming through). There are probably many ways to do this in the git world - for example, one could just use git fetch to bring in changes into remote tracking branches and compare these against local branches (any malicious code would not be merged into a local branch after it is discovered), but here is an alternate solution:

• On my system where the private repositories live, create two directories GH/ and CD/, and clone all of the github repositories into the GH directory and all cipherdyne.org repository mirrors into the CD directory. We'll assume that the private repositories live in a directory called private/
• For each repository pair in the GH and CD directories, diff the output of git rev-list --all. There should be zero differences here.
• If the step above checks out, then diff the git rev-list --all output across the GH/<repo> and private/<repo> pairs. For this step we expect that the private repositories will have local commits that are not necessarily pushed upstream - what we're concerned about is any commit in a GH/<repo> that is not in a private/<repo>.

Here are a few commands to accomplish the above (we'll assume that the git clone --bare <repo> steps have already been done for brevity):

$ for r in fwknop psad gpgdir fwsnort IPTables-Parse IPTables-ChainMgr
> do
> echo "[+] Checking $r...";
> diff -u <(cd ~/CD/$r.git && git rev-list --all ) <( cd ~/GH/$r.git && git rev-list --all)
> done
[+] Checking fwknop...
[+] Checking psad...
[+] Checking gpgdir...
[+] Checking fwsnort...
[+] Checking IPTables-Parse...
[+] Checking IPTables-ChainMgr...

The output above indicates there are identical git commits in the github repositories vs. the cipherdyne.org mirrors. Good. Now, let's compare the github repositories vs the private ones - we grep on "+" which would indicate new commits in github that are not in the private repositories:

$ for r in fwknop psad gpgdir fwsnort IPTables-Parse IPTables-ChainMgr
> do
> echo "[+] Checking $r...";
> diff -u <(cd ~/private/$r.git && git rev-list --all ) <( cd ~/GH/$r.git && git rev-list --all) | egrep "^\+" | grep -v @
> done
[+] Checking fwknop...
+++ /dev/fd/62  2012-03-07 22:50:48.004281002 -0500
[+] Checking psad...
+++ /dev/fd/62  2012-03-07 22:50:48.054281002 -0500
[+] Checking gpgdir...
+++ /dev/fd/62  2012-03-07 22:50:48.164281002 -0500
[+] Checking fwsnort...
+++ /dev/fd/62  2012-03-07 22:50:48.194281002 -0500
[+] Checking IPTables-Parse...
[+] Checking IPTables-ChainMgr...

Again, good, no commits in github that are not in the private repositories. If there had been a line like the following I would have been concerned:

+fff688f5b4275152636d8959f67bbcd46839fbbb

Rather than modifying code and committing it to a git repository on github, it would have been far more damaging for an attacker to just alter the github website to serve up drive by exploits for a popular web browser. Either way, I'm glad they fixed the vulnerability.