For Firefox' Sake!
Last updated: Wed, 08 Jul 2009 08:57:07 GMT
What's the big deal about SSL? It's been around for yonks; we must have it taped by now, right? Wrong, sadly.
Sadder still, it could all be plain sailing, but for the numerous issues around "bad" (but otherwise good) certificates. Be they self-signed, issued by your own internal CA, or issued by some reputable free CA like CAcert, they're still considered bad by "99% of browsers."
Perhaps the saddest part of all, though, is that it still apparently costs money to get a "good" certificate. Even in a world Freedom reigns, security, it seems, is not free. I suspect, with some evidence, that this is a matter of money changing hands.
That makes me a sad panda.
I've heard some compelling arguments recently that X509 SSL certificate handling should be more like SSH's handling of host keys. Our clients should, without too much fuss, allow us to accept the certificate the first time, as long as it's valid in this time period. Screw the chain, just accept it. Let me know when it changes.
It's an interesting idea. Of course, the purpose of that traceable route from a known, trusted signatory to your server's certificate is to validate identity. SSL aims to provide security and authentication. But an interesting idea none the less. I reserve judgement.
Of course, the client's behaviour plays a large part in how SSL certs are used. Web browsers, for instance, go to some lengths to check that the certificate's CN matches the hostname part or the URL, that the dates are right and the certificate is current, and that a chain of signature can be traced back to a trusted CA. Lots of browsers will pop up a warning asking you to confirm that you'd like to proceed, and may give you other visual clues that there's something not quite right. A minor inconvenience.
Until Firefox 3.
Firefox 3 goes to great lengths to make acceptance of a "bad" certificate painful. This wouldn't smart so much if getting a "good" certificate weren't effectively a giant scam, in which the aforementioned 99% of browsers appear to be complicit. But it is, in my opinion, and it does smart. And, consequently, it's a bloody great pain in the arse.
A little while ago, I stumbled onto a whitepaper about the Map-Reduce system that Google use to process massive data sets over large clusters. I stumbled there from a blog entry about how great tech employers find, interview and select new employees. I was looking for work at the time, and I was about to be interviewed for a great job, so it struck a chord.
The author explained that they asked candidates to solve a particular problem, on the whiteboard, using whatever form of code or notation they felt comfortable with. Once this was done, they'd then ask "So, how would you scale this out to, say, 1TB of data? What about 10TB?"
I was asked similarly scaled questions at my own interview, but as you can imagine there was no pseudocode involved. It wasn't until I got here that the problem of scale really hit me. It keeps coming up again and again, and one day I hope that it'll be in the forefront of my mind, as it is in my colleagues'. Every time I think of a solution, someone says "But how are we going to scale that across n-thousand hosts?"
Back to SSL, and Firefox.
This is not a rant about HP, but HP's blade management tools are broken as far as SSL goes. Every enclosure has two Onboard Administrators, administered over HTTPS. Every enclosure has two switches, administered over HTTPS. Every enclosure has 16 or 32 blades, each with its own iLO, all administered over HTTPS.
None of these devices will allow me to upload a key and certificate. So I can't use the wildcard cert I just paid for. They'll all generate a CSR, but signed with their own, self-generated key, so I'm forced to get that CSR and, say, have it signed by our own internal CA. That might work, except that the iLOs cannot be made to generate a CSR with a correct, well-formed CN (which has to match the hostname in the URL, remember.) The OAs are a little better, except that they insist on redirecting us to the iLOs by IP address, not hostname.
So, basically, we have no choice but to accept that our browsers will give us some minimally annoying warning when we connect to a device for the first time.
Oh. Firefox 3.
Well, yeah, it's annoying, but how annoying can it really be?
And now for the punchline... "That's fine; but how would you scale this over 100 devices? How about 1000 devices? 10000 devices?"
With that many devices, how great are your chances that one is failing at any one time? How often are you going to need console access? How likely is it that you'll ever visit the same device on two different days? Is your list of acceptably-bad certificates every going to be complete?
We're boned.
Like I said, this isn't a rant about HP. Their OAs' and iLOs' handling of SSL is broken. But I've only just opened a case with them and we'll see what they've got to say. And it's not a rant about Firefox 3, either. I'm sure that they think that they're doing the right thing. In my humble opinion, the right thing would begin with finding a way to start including CAcert's root certificates, but that's just me.
We're still left with a problem. I looked for a number of solutions, but the only vaguely sensible one seemed to be to find a way to automate the "accept this certificate as an exception" process.
The follow links proved useful:
It all sounds like an hours' scripting, apart from that mention of "A special encoding of the allowed cert's serial number and the issuer name". That's the kicker. That's why I had to start poking about in the source code.
Enter ffsak3
This is an ugly, first-hack, never-to-be-improved, dirty little script, with some associated nastiness built on bits of NSS. You'll need to be comfortable downloading and compiling Firefox 3 yourself, but if you're in a position where using this makes sense, your sk177z are probably ph33rsUm already. You have been warned.
Here goes...
Download and unarchive ffsak3. Download a current copy of Firefox, unarchive it into the directory created when you unarchived ffsak3:
$ gzcat ffsak3.tar.gz | tar xf - $ cd ffsak3 $ bzcat ../firefox-3.5rc3-source.tar.bz2 | tar xf -
We'll be exposing a previously hidden part of Firefox' supporting libraries. The easiest way to do this, at least while I was still hacking away and didn't know where I'd be poking about, was to hardwire Firefox' configure script not to use the hidden options provided by gcc:
$ cd mozilla-1.9.1/ $ echo ac_cv_visibility_hidden=no > .mozconfig
Now run configure, in a manner that it'll want to build Firefox for us. Any problems at this point are your own:
$ ./configure --enable-application=browser creating cache ./config.cache checking host system type... x86_64-unknown-linux-gnu checking target system type... x86_64-unknown-linux-gnu checking build system type... x86_64-unknown-linux-gnu checking for gawk... gawk checking for gcc... gcc ... checking for visibility(hidden) attribute... (cached) no ... creating Makefile creating config/Makefile creating config/mkdepend/Makefile updating cache ../.././config.cache creating ./config.status creating config/autoconf.mk creating js-config.h invoking make to create js-config script rm -f js-config.tmp sed < js-config.in > js-config.tmp \ -e 's|@prefix@|/usr/local|' \ -e 's|@exec_prefix@|/usr/local|' \ -e 's|@includedir@|/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/include|' \ -e 's|@libdir@|/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/lib|' \ -e 's|@MOZILLA_VERSION@||' \ -e 's|@LIBRARY_NAME@|mozjs|' \ -e 's|@NSPR_CFLAGS@|-I/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/include/nspr|' \ -e 's|@JS_CONFIG_LIBS@|-L/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/lib -lplds4 -lplc4 -lnspr4 -lpthread -ldl -ldl -lm -lm -ldl |' \ -e 's|@MOZ_JS_LIBS@|-L/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/lib -lmozjs|' \ && mv js-config.tmp js-config && chmod +x js-config
Now build Firefox:
$ make rm -f -rf ./dist/sdk rm -f -rf ./dist/include ... gmake[4]: Leaving directory `/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/testing/xpcshell/example' gmake[3]: Leaving directory `/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/testing/xpcshell' gmake[2]: Leaving directory `/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1' gmake[1]: Leaving directory `/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1'
Copy the getdbkey.cpp source file into the security manager subdirectory, use the Makefile's rules to compile an object:
$ cp src/getdbkey.cpp mozilla-1.9.1/security/manager/ssl/src $ cd mozilla-1.9.1/security/manager/ssl/src $ make getdbkey.o getdbkey.cpp c++ -o getdbkey.o -c -DNSS_ENABLE_ECC -DXPCOM_TRANSLATE_NSGM_ENTRY_POINT=1 -DMOZILLA_INTERNAL_API -D_IMPL_NS_COM -DEXPORT_XPT_API -DEXPORT_XPTC_API -D_IMPL_NS_COM_OBSOLETE -D_IMPL_NS_GFX -D_IMPL_NS_WIDGET -DIMPL_XREAPI -DIMPL_NS_NET -DIMPL_THEBES -DZLIB_INTERNAL -DOSTYPE=\"Linux2.6.25\" -DOSARCH=Linux -I/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/include/nss -I. -I. -I../../../../dist/include/nspr -I../../../../dist/include/xpcom -I../../../../dist/include/string -I../../../../dist/include/necko -I../../../../dist/include/uriloader -I../../../../dist/include/pref -I../../../../dist/include/docshell -I../../../../dist/include/caps -I../../../../dist/include/dom -I../../../../dist/include/intl -I../../../../dist/include/locale -I../../../../dist/include/profile -I../../../../dist/include/windowwatcher -I../../../../dist/include/js -I../../../../dist/include/widget -I../../../../dist/include/layout -I../../../../dist/include/content -I../../../../dist/include/xpconnect -I../../../../dist/include/unicharutil -I../../../../dist/include/pipboot -I../../../../dist/include -I../../../../dist/include/pipnss -I/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/include/nspr -I/usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/dist/sdk/include -fPIC -fno-rtti -fno-exceptions -Wall -Wpointer-arith -Woverloaded-virtual -Wsynth -Wno-ctor-dtor-privacy -Wno-non-virtual-dtor -Wcast-align -Wno-invalid-offsetof -Wno-long-long -pedantic -fno-strict-aliasing -fshort-wchar -pthread -pipe -DNDEBUG -DTRIMMED -Os -freorder-blocks -fno-reorder-functions -DMOZILLA_CLIENT -include ../../../../mozilla-config.h -Wp,-MD,.deps/getdbkey.pp getdbkey.cpp
Just so that we can link, set LD_LIBRARY_PATH to point at our recently built libraries. Create a getdbkey executable:
$ setenv LD_LIBRARY_PATH ../../../../dist/lib
$ gcc -o getdbkey -L../../../../dist/lib -lnss3 -lsmime3 -lxul \
-lnssutil3 -lmozjs -lsqlite getdbkey.o
Just testing, here, using a DER format key I've prepared earlier. Note that we need to point getdbkey at directory containing the NSS certificate and key DB files that Firefox expects to see. You can use certutil to create an empty one here for testing:
$ cat cert.der | ./getdbkey . ./getdbkey: could not initialise NSS $ certutil -N -d . Enter a password which will be used to encrypt your keys. The password should be at least 8 characters long, and should contain at least one non-alphabetic character. Enter new password: Re-enter password:
Now, does getdbkey get us the all-important secret-squirrel key we need?
$ cat cert.der | ./getdbkey . AAAAAAAAAAAAAAAEAAAAewAwdzELMAkGA1UEBhMCVVMxDWlMTjAMBgNVBAgTBVRl eGFzMRAKExdwDgYHEwdIb3VzdG9uMSAwHgYDVQQIZXDVQQdsZXR0LVBhY2thcmQg Q29ueTEMMAoGA1UECxMDtcGFSVNTMRYwFAYDVQQDEw1IxMDmb2trZXFhaWxv
Yep. Copy the executable back upto the root of the ffsak3 directory:
$ pwd /usr/home/car/MuhJob/src/ffsak3/mozilla-1.9.1/security/manager/ssl/src $ cp getdbkey ../../../../../ $ cd ../../../../../
The ffsak3.sh script contains some information you may need to change, like LD_LIBRARY_PATH. Barring that, it's ready to go. It requires one argument, a directory, as we've seen. It accepts host:port entries, one per line, on standard input. It fetches the certificate from the host and port specified, using the s_client SSL client tool provided by openssl, inserts the certificate into the cert8.db file and adds an exception entry to the cert_override.txt file. This cannot be done while your firefox client is running! But you could create a db and override file and copy them into place at your leisure.
Here's me testing it with a single entry:
$ echo dingo101ilo.wall.muhjob.co.nz:443 | MuhJob/src/ffsak3/ffsak3.sh ~car/.mozilla/firefox/mm5lhbef.default depth=0 /C=US/ST=Texas/L=Houston/O=Hewlett-Packard Company/OU=ISS/CN=dingo101ilo verify error:num=18:self signed certificate verify return:1 depth=0 /C=US/ST=Texas/L=Houston/O=Hewlett-Packard Company/OU=ISS/CN=dingo101ilo verify return:1 DONE read: 1 added: 1
Success. In practice, I generated a list of all of the HP management devices that were up, and from that generated two lines for each: one FQDN:port, one IPv4:port. Then I set the script to chew on that and went and did something else. Three hours later, several thousand certificates accepted.
Works a treat. Takes a damned long time to load Firefox' certificate management Preference windows, but I can live with that.
You're welcome to take this or leave it. I'm not presenting it as a nicely wrapped little tool, and it needs plenty of improvement. There are, of course, a dozen more graceful and elegant solutions to this problem, but this one was at least open to me at the time of writing.
Enjoy.