The Death of SKS PGP Keyservers, and How First Look Media is Handling It

As big users of PGP encrypted email, First Look Media has heavily relied on the decentralized network of PGP keyservers powered by free software called SKS. Each day, on every staffer’s computer, a program called GPG Sync fetches each PGP key on our centrally-managed keylist from the SKS keyservers, ensuring that everyone always has the latest version of everyone else’s public key.

But the SKS keyserver network is dying. This has been a long time coming. Over six years ago, I gave a talk called Trolling the Web of Trust that outlined a number of vulnerabilities with SKS keyserver network, but none of the issues were ever addressed, nor will they ever be addressed because the SKS open source project itself is dead. The last release was over three years ago, while issues like this critical vulnerability that was reported over a year ago remain unpatched, and even pull requests that partially fix some of the issues remain unmerged.

Here is the crux of the issue: anyone can add information to anyone else’s public key and upload it to an SKS keyserver, including provably malicious information like fake user IDs or revocation certificates, or 30mb worth of signatures. SKS trustingly appends this information to the end of the key without doing any validation, and the servers in the SKS network synchronize this information amongst each other. Basically, the system has no defenses against abuse.

Fake Signatures IRL

The nail in SKS’s coffin came in late June when someone abused some actual public keys that people rely on (including mine!), breaking them so that they could no longer be downloaded from SKS keyservers. Here’s what happens when you try importing my PGP key, which currently has 32mb of fake signatures (84k of them) on it, from an SKS keyserver:

$ gpg --keyserver hkp://pool.sks-keyservers.net --recv-keys 927F419D7EC82C2F149C1BD1403C2657CD994F73
gpg: key 403C2657CD994F73: 58 duplicate signatures removed
gpg: key 403C2657CD994F73: 84562 signatures not checked due to missing keys
gpg: key 403C2657CD994F73: 1 signature reordered
gpg: error writing keyring '/home/user/.gnupg/pubring.kbx': Provided object is too large
gpg: key 403C2657CD994F73: public key "[User ID not found]" imported
gpg: Total number processed: 1
gpg:               imported: 1
gpg:           not imported: 1

There’s nothing I can do to fix my key. I could always switch to a new PGP key, but I honestly don’t see the point when anyone could break it again if they chose to. I’m actually surprised so few PGP keys seem to have been abused so far.

Hagrid, Keeper of Keys

But don’t fret too much. There’s a new keyserver in town: Hagrid (“keeper of keys”) is an open source verifying keyserver software project, implemented in Rust, which is designed to be resistant to the abuse and privacy issues that plague SKS. Three prominent software projects in the PGP ecosystem (Sequoia-PGP, OpenKeychain, and Enigmail) collectively run the Hagrid server keys.openpgp.org.

The keys.openpgp.org keyserver has a few fundamental differences when compared to SKS keyservers:

• It’s a central service that isn’t federated and doesn’t sync with other keyservers. They plan to decentralize it at some point, but the federated Hagrid servers will never be badly-run hobby projects like SKS keyservers sometimes are.
• Anyone can upload public keys, but Hagrid strips all user IDs, signatures, and everything else from them. Only the cryptographic key material, like the numbers required to encrypt a message to the public key, or a revocation certificate, is freely distributed.
• If a user wants their user ID (their name and email address) to be published, the user needs to prove ownership over the email address using double opt-in validation -- Hagrid will send a verification email, and the user must click a link in the email.
• Because signatures are stripped, Hagrid cannot be used to facilitate the web of trust. If you sign someone’s key and want others to know that you signed it, you’ll need to find some other way to distribute that signed public key to those people.

When a new employee starts at FLM, they get a laptop, an email address, a PGP key, and GPG Sync configured to subscribe to our keylist of over 200 PGP keys that we use internally. But if a new employee downloads all the keys on our keylist from keys.openpgp.org, only the ones who have opted in to publishing their email addresses will be useful for encrypted email -- the others won’t contain a user ID, so their email program won’t know which PGP key to use for them. So for FLM to be able to switch to keys.openpgp.org, first we must get everyone to opt in to publishing their email address.

To opt in to publishing your email address on keys.openpgp.org, you can upload your PGP public key using this form and click the “Send Verification Mail” button next to each email address you want to verify. Hagrid will send you an email with a link, and after you click it, your PGP key will include this user ID when people download it.

GPG Sync v0.3.4 and hagrid-verifier

Getting hundreds of people to manually opt-in to publishing their email address on this keyserver -- who had IT staff set up encrypted email on their computers for them to begin with -- is challenging. We solved this problem in three ways.

• First, when we onboard new employees, or when we help an employee revoke their old PGP key and generate a new one, we help them opt-in to keys.openpgp.org as part of the process.
• Second, I wrote a script called hagrid-verifier to automate submitting all PGP keys on a keylist to keys.openpgp.org, and to opt-in to publishing their email addresses. We still need to get everyone to click the verification link in their email, but that’s a much simpler problem.
• And finally, we released GPG Sync 0.3.4, which uses keys.openpgp.org by default. When you add a keylist to sync, there is an advanced setting, which is enabled by default, called “Use modern keyserver.” If checked, GPG Sync downloads public keys from keys.openpgp.org using their API (which is much nicer to work with than subprocessing out to `gpg`). You can always uncheck it if you want to use legacy SKS keyservers.

RIP SKS

It’s been pretty good, SKS keyserver network. Rest in peace.

Links to Code

• GPG Sync - Ensure users always have up-to-date public keys for other members of their organization
• hagrid-verifier - Submit all the keys on a keylist to keys.openpgp.org

---

Padlock photo by Alexander Schimmeck on Unsplash