There’s a bit of a kerfuffle in the web hosting community just at the moment over an expired web security certificate from a certificate authority called Sectigo, formerly Comodo Certificate Authority.
Expired certificates are a problem because they cause the web server that relies on them to show up as “invalid” to any program that tries to do the right thing and verify the validity of the site it’s connecting to.
But this problem isn’t Sectigo’s fault – indeed, the company has had a warning about the impending problem available for a while now, explaining what was about to happen and why.
The problem comes from what’s known as backwards compatibility, which is a jargon way of saying “trying to support old software reliably even though it really ought to have been upgraded to a newer and more reliable version”.
When your browser visits a website, it’s almost certain to be using HTTPS, short for secure HTTP, which means using the Transport Layer Security protocol (or TLS for short) to encrypt and validate the connection.
As you probably know, TLS doesn’t vouch for the content that’s ultimately served up by a web server – crooks can use TLS to deliver malware “securely” if they like – but it is nevertheless a vital part of everyday browsing.
Not only does it shroud your traffic from surveillance and snooping, it stops someone in between you and the server you’re visiting from tampering with the content on the way out or back. (Because rogues of this sort can be anywhere along the network path, it’s known colloquially as a MiTM attack, short for man-in-the-middle.)
Of course, if crooks could trivially issue certificates in the names of other websites, MiTM attacks would still be easy, even with TLS, because the crooks could put a fake site half way along the network path to the real one, and you would be unable to tell it from the real deal.
So, to make it harder for crooks to mint a web certificate in your name, you need to get your certificate vouched for by someone else, known as a certificate authority.
You then present your certificate and their certificate, and they vouch for you; if their certificate is, in turn, vouched for by your browser itself (i.e. is in a list of already-trusted-certificates-that-can-sign-other-certificates), then your browser will automatically accept your certificate because it’d been signed by someone that the browser already trusts.
This forms a chain of trust.
What this means is that every browser (or every operating system on behalf of the browsers you might use) needs to have access to an up-to-date list of what are called root certificates, which is the name given to certificates that aren’t vouched for by anyone else, but that are explicitly trusted to vouch for others.
Intermediate certificates
Obviously, the part of a root certificate that’s called the private key, which is used for signing purposes, needs to be kept extra-super-secure, because replacing or re-issuing root certificates is a much trickier exercise than updating or issuing so-called leaf certificates – the ones that go with your website and typically only last anywhere for 3 months to 2 years anyway.
To make it easier and safer to sign and distribute new keys, most leaf certificates use a chain of three links, not just two, to “prove” their validity.
There’s the leaf certificate that vouches for your website; there’s an intermediate certificate that vouches for your leaf; and then the intermediate certificate is vouched for by a root certificate that is itself magically imbued with vouching power because it is trusted directly by your browser or your operating system.
Root certificates therefore often have long lifetimes, typically 10 or 20 years, and the assumption is that everyone will have plenty of time to stop relying on old root certificates long before they expire.
But old software programs, and old operating systems, have long shelf-lives too, and old software programs, tied to an old database of trusted root certificates, often end up relying on ageing root certificates in their so-called “chain of trust” long after they should.
So, even if you do the right thing and ask your certificate authority – the company that’s vouching for you – to use their latest intermediates and their latest root certificates every time you renew your certificates, which is usually at least once a year, you might end up confusing customers with old software (possibly even with old software of your own manufacture).
That’s because old software that hasn’t yet been taught about the latest and greatest root certificates that are available – because it’s not getting reliably updated, for example – will keep on trusting the old root certificates you are keen to move away from, even as they edge towards expiry, yet will keep on rejecting the new ones as “untrusted” even though the new ones have years of life left in them.
Ironically, then, the newer and fresher your chain of trust, the less reliable your certificates will seem to old-timer programs out there.
Cross-signing
What many companies do, to support both ends of the equation, is what’s called cross-signing, where they denote two different intermediate certificates to vouch for your leaf certificate, one signed by an old root; the other by a new one.
The idea is to please most of the people most of the time.
Of course, that can make your security situation seem better than it is.
Old and possibly insecure web clients – which will include all sorts of software tools other than browsers, notably including autoupdate programs and licence-checking tools that are supposed to keep the software running correctly – will give you a false sense of being “up to scratch”.
When the tired old root certificate expires, software that has never heard of the all-new root certificate that replaced it will simply stop working. (Unless it isn’t checking the validity of your web certificate at all, but that’s increasingly rare because it’s easy for researchers to detect and will guarantee bad publicity if they do.)
It’s worse than that
But, as Andrew Ayer of SSLMate explains, the situation is worse than that.
Technically speaking, certificate chains where there’s a choice of cross-signed intermediate certificates, can be “resolved” more than one way.
You can follow the old-style intermediate certificate to the now-expired root certificate, or you can try the other way home, validating with the new-style intermediate and correctly determining that it is signed by a new and valid root.
Ideally, newer certificates should trump older ones, so that as long as one of the certificate chains checks out, the leaf certificate should be accepted.
But, as Ayer explains, some older TLS software (or some older versions of current TLS libraries) fail if the first certificate chain they try has expired, even though trying again with fresher data would find that the HTTPS connection was valid.
That’s the trouble here – even though one of Sectigo’s backwards-compatible root certificates has now expired, some web software is still relying on that old root certificate, which expired on 30 May 2020, even though it already knows about the new root certificate and should be verifying the certificate chain as valid.
What to do?
If you are getting web connection errors on software that was working fine until the end of last month, where the error lists an invalid certificate called AddTrust External CA Root, you need to take action.
You may need to update the software that’s trying to make the connection, or its root certificate “trust store”, or both.
If you’re stuck, consult your vendor – and if you are the vendor because it’s your own software, you may need to consider upgrading to a more modern TLS programming library that handles web certificate verification in a more future-proof way.
Ayer has some advice in his blog article – notably, if you are using a TLS library that ought to validate Sectigo certificates but isn’t doing so, you may be able to fix the problem simply by deleting the now-expired AddTrust External CA Root certificate – which is no use anyway but may nevertheless get in the way – from the certificate database on your computer.
The expired certificate was replaced a decade ago (!) by one denoted USERTrust RSA Certification Authority, so many TLS libraries do known about the “new” root certificate perfectly well; the problem is that they still know about the old one too, and get hung up on it even though it serves no purpose any more.
