Adventures in Forensic File Recovery
Last updated: Wed, 27 Apr 2005 12:01:00 GMT
Picture the scene...
Picture the scene: a young system administrator, renowned for his sparkling wit and dashing good looks, decides that it's about time he revisits PGP, because it's plain that client support is abundant enough that it can actually be useful, and he has some secret stuff to do or say. He decides that he'd like to maintain the separation of personal and professional identity, and creates two separate key pairs, reasoning that this way he doesn't have to leave his personal private key lying around on public servers.
So he creates his personal key pair and stores it on his laptop, and on his fileserver at home, just in case. He thinks about copying them to his USB keyring, but FreeBSD is having trouble with the USB hardware on his laptop and, well, he'll get round to it.
Some time later...
Picture a slightly haggard system administrator. The same guy, but he's looking older, and fatter, and there's a murderous glint in his eye. His glare is directed towards his laptop, now refusing to boot FreeBSD. Not quite true -- it gets so far, but it refuses point blank to fsck his /usr partition. Quite rightly as it happens, because his drive appears to have suffered a fairly major media failure. It's Sunday morning, he has work to do. He's already ordered a shiny new iBook to replace his crappy laptop and he needs a quick fix to get what work he can get done, done.
So he force-mounts the partition, tars up what he can of his home directory before the IDE controller starts honking, takes special care to tar up his work and his ssh keys, and plays his last-ditch recovery card:
dd if=/dev/ad2s1e of=/dev/ad2s1e
in the hope that the controller will map out the bad blocks scattered across his platters. It doesn't work. In desperation he reaches for the Windows XP rescue CDs that came with his laptop.
This is just the quick fix he needs. Half an hour later, he's squirted an image of XP all over his disc and he's working again. XP cares not for holey discs.
What he forgot, though he wouldn't realise this until some two weeks later, was his PGP keys. Not suprising, given that he rarely uses them; this is the nature of such security -- it must be set up far in advance of us actually using it. And he has the security that the keys are on his fileserver, and on his USB keyring.
They are, aren't they?
Well, he never quite got round to copying them onto his USB keyring. And let's not forget the disc failure he suffered on his fileserver last month.
Oh dear.
The scene is set. Where do I go from here? I have a laptop with a flaky disc containing an image of XP that's essentially been unzipped and dd'd over the old drive. Hopefully in a sparse fashion, only writing what wasn't empty. Time for some so-called forensic file recovery.
It's not something I've done before, but I have plenty of impetus to learn, and anything's quicker than trying to brute the other half of my key pair.
A quick chat with our backup and security advisor left me with a recommendation for Autopsy, which is a webtastic front-end for The Sleuth Kit. I need an operating system to run this from, and I decide that a purpose-built Linux distribution is probably my most convenient option. Google recommends, or at least provides highly ranked links for, F.I.R.E.
F.I.R.E. is currently at version 0.4a. This was a sign. The CD I burned now lies in my drawer, bearing the epithet "DO NOT USE: SUCKS ASS!"
I was greeted at boot time by a lot of green and black, and lovely chatty startup messages: "Sweet!!!" and "Now the [something with an 'X' in it] has you!!!" And then we were ready to roll. Ah, multiple xterms with green-on-black courier text. Clearly, this tool is meant for the elite. I was further impressed by vi's constant segfaulting, and the inability to run, well, anything that relied on libresolv, for instance. Aw3some. It was a good job that so many xterms were opened by default, because they did seem prone to hanging.
The rage came.
Now that I've calmed down, I note that the F.I.R.E. web page does say something about a directory full of statically linked binaries. Perhaps I'd have had more luck there. Perhaps this /statbin directory is a common Linux thing, it's not something I've come across before, unless we count IRIX's /stand.
But, you know what? If I knew that all of the dynamically linked executables in my distro were screwed, I think I'd have done something about it. Something really hardcore like, I dunno, changing PATH, or filling /sbin with functional statically linked binaries. I used to think that that was what /sbin was for.
So, what of Autopsy? I gave it a whirl. Five minutes later, I'd defined a case name, and added some users to that case, and some hosts. I was all set to delegate the hell out of this thing and manage my case managerially. Unfortunately, Autopsy didn't manage to recognise my disc partition.
Perhaps I'm being unfair, but it seemed confused by the disc partition I presented it with, which it would only accept as an NTFS partition. Of course that's exactly what it was, on the surface, but I wanted to get at the remains of the UFS partitions that lay beneath, and I had no joy.
This, I've got to be honest, is probably less of a failing in Autopsy than a failing on my part. I was flapping, and I was getting angry. My prejudice against all things Linux had been reinforced by the crapitude of F.I.R.E. and I was kicking myself for getting myself into such a n00b mess.
At this point, I decided that I'd write my own tool. What was I looking for? Well, the secret keyring I had in my work account contained only one secret key, it was short -- little more than 2KB -- and to my eye it looked like random binary data with my name and email address tucked away in plain text.
I needed nothing more than to seek over 30GB of data, finding references to my email address and then excerpting, say, 8KB before and after each occurence. Trivial stuff. A friend suggested that using file's magic numbers might further help identify candidates, and I knew I was onto a winner. I could decide on whether to search for my email address, (which would be a slow comparison but produce relatively few results,) and narrow the field with magic numbers or search for those two bytes of magic number first (which would be cheap, but produce a lot of candidates) and narrow the field with a search for my email address. Either way, I'd probably end up writing my own strcmp(). All I had to do was get an image of the drive onto something with a working operating system.
I tired rapidly of trying to do this from F.I.R.E. and of messing around trying boot the FreeBSD 5.3 live installation CD on my laptop, and in the end settled for tearing the drive out and jury-rigging it into a desktop PC running FreeBSD 5.4. It will come as no surprise to the observant reader that I am a man of short temper, and am prone to irrationality. Sometimes I lack finesse.
Five minutes later I was sucking the data from the broken disc with a simple
dd if=/dev/ad2 of=./ad2.dmp conv=noerror,sync
It sprinted for the line until it hit 22GB, and then the media errors started. It slowed to a crawl, I started thinking about my home-brewed file discovery tool, and googled the magic numbers I needed to identify a GPG secring.
Meanwhile, back at the ranch, one of those young turks I've mentioned before had been doing a little looking around, (by which I mean a grep over the package description files in /usr/ports,) and had struck gold in the form of foremost.
Foremost is the general solution to the problem I wanted to solve with my very specific tool. Five minutes with a man page, and she was off, in quick mode, while the dd was still ripping data from the disc. I had asked only for files starting with the two magic numbers associated with GPG secrings, but foolishly left the excerpt size at the default of 100KB. Configured ike so:
...
#---------------------------------------------------------------------
# PGP (PRETTY GOOD PRIVACY)
#---------------------------------------------------------------------
#
# PGP Disk Files
# pgd y 500000 \x50\x47\x50\x64\x4d\x41\x49\x4e\x60\x01
#
# Public Key Ring
# pgp y 100000 \x99\x00
# Security Ring
pgp y 100000 \x95\x01
pgp y 100000 \x95\x00
# Encrypted Data or ASCII armored keys
# pgp y 100000 \xa6\x00
# (there should be a trailer for this...)
# txt y 100000 -----BEGIN\040PGP
...
Issue:
foremost -q -o secrings/ -c gpg.conf ./ad2.dmp
It took about two hours for foremost to do its stuff. It found me 6000 candidates, and grep told me that some 30 of those contained my email address. I could hardly wait.
Sadly, most of those files seemed to be fragments of old web pages, or emails saved. My vCard was there and, tantalisingly, an old copy of my crontab that contained the sh I use to build my .sig, which contained a link to the public half of my key pair. A lot of the combined hits were simply down to the size of the excerpt that foremost produced. My key would be less than 8KB in size, but here I had 100KB of disc garbage.
I was despondent. I started casting around for other places I might find my key. But I was kidding myself; I'd copied to my fileserver and that was it. Maybe I'd give foremost a go over my fileserver's broken disc.
I thought I'd have one last go, for luck, and this time do a proper job.
foremost -o secrings/ -c gpg.conf ./ad2.dmp
Without quick mode, foremost finds a lot more crap, and I didn't have the space to store an arbitrary number of 100KB excerpts along with the 30GB image. I tried tuning the excerpt size down to 8192 bytes and foremost segfaulted. Up to 16384 bytes and segfault again. At 20000 bytes it seemed happy. I've no idea why. I kicked it off and went to bed.
I got to work this morning to find that foremost had exited uncleanly about 19GB through my image, complaining "file too large." Perhaps the audit file it generates had grown beyond some internal limit. It was 70MB in size, listing some 1006700 candidates. Perhaps the directory listing was too large for the fileserver I was storing the files on.
Like the Solaris flunkly that I am, I set a find -exec grep {} running over the candidates to find something containing my email address. I forget that GNU grep recurses. This was the weakest link; find and grep took their time, but stil they were quicker than trying to brute my lost key. A slow trickle of candidates came through, every one a disappointment until, after four hours, the jackpot -- a 20KB file containing what was obviously my GPG secret keyring followed by random disc trash.
I was saved some work, trimming the garbage from the file, by my famous sense of humour. A few months earlier, I had been discussing PGP with a friend whose ubertarded admins had told him that he wasn't allowed to use PGP because "[they] wouldn't pay for the keys." For a laugh, I generated him a key pair and emailed it to him with a note to the effect that I'd undercut any key seller they could get a quote from by 10%, and that I offered discount on bulk orders.
GnuPG was quite happy to run over the corrupted file and export all keys except the last, which just happened to be that gag key.
Bingo.
Twenty four hours wasted, but an amusing interlude with a happy ending. What more can I ask for?
I'm trying to think of a list of lessons learned, but struggling. The worst of Linux still sucks. FreeBSD's documentation is perhaps not what it could be, but it's still Better Than Linux. Take time to look for a wheel before putting your inventing hat on? I don't know about that one. I could have made some fundamental error and left my home-brew solution churning all night and not found what I wanted, or it might have found my keyring in two hours flat. Whatever, foremost did the trick.
Oh, I've copied my keys to my USB keyring. Before I go home I might print out a hard copy to store in the gun cabinet.