A world of hurt after GoDaddy, Apple, and Google misissue >1 million certificates

A major operational error by GoDaddy, Apple, and Google has resulted in the issuance of at least 1 million browser-trusted digital certificates that don’t comply with binding industry mandates. The number of non-compliant certificates may be double that number, and other browser-trusted authorities are also likely to be affected.

The snafu is the result of the companies' misconfiguration of the open source EJBCA software package that many browser-trusted authorities use to generate certificates that secure websites, encrypt email, and digitally sign code. By default, EJBCA generated certificates with 64-bit serial numbers, in keeping, it seemed, with an industry mandate that serial numbers contain 64 bits of output from a secure pseudo-random number generator. Upon further scrutiny, engineers discovered that one of the 64 bits must be a fixed value to ensure the serial number is a positive integer. As a result, the EJBCA default produced a serial number with 63 bits of entropy.

The 63 bits is far off the mark of the required 64 bits and, as such, poses a theoretically unacceptable risk to the entire ecosystem. (Practically speaking, there’s almost no chance of the certificates being maliciously exploited. More about that later.) Adam Caudill, the security researcher who blogged about the mass misissuance last weekend, pointed out that it’s easy to think that a difference of 1 single bit would be largely inconsequential when considering numbers this big. In fact, he said, the difference between 2⁶³ and 2⁶⁴ is more than 9 quintillion.

Almost no chance of exploitation

With all that said, despite the shortcomings of the misissued certificates, there is very little chance their non-compliant entropy can be exploited. Certificates are now generated using SHA256, a modern algorithm that doesn’t have the known vulnerabilities of MD5. The 64-bit requirement, rather, is more a matter of insuring against new attacks that will likely be discovered in the coming decades.

What that means is that, while the revocation and reissuance of between 1 million and 2 million certificates (at the time this post went live, researchers were still debating the number) is a major undertaking, there is virtually no security threat posed by the error.

“This is a big deal for CAs and their customers,” Caudill told Ars. “The impact of replacing large numbers of certificates is substantial. From a threat perspective though, this isn’t exploitable. It would require a major breakthrough in cryptography, and even then, 63 bits of entropy provides a huge safety margin. This is a problem because of impact to people and companies; hackers aren’t going to start forging certificates because of this.”

In online forums discussing the problem, a GoDaddy official initially said his company issued more than 1.8 million certificates that didn’t comply with the 64-bit requirement. Under industry rules, GoDaddy had five days to revoke the certificates, but GoDaddy said it wouldn’t be able to make that deadline for all the certificates identified.

“Within the next 30 days”

“Our goal is to reissue all the certificates within the next 30 days,” wrote Daymion Reynolds, who is senior director of SSL/PKI security products at GoDaddy. “We have started the revocation process. We have a significant number of customers that use manual methods for managing their certificates, so being agile for them is difficult. We want to keep our customers using https through the entire revocation period. Due to the large number of certificates and the benign nature of the issue, our plan is to revoke in a responsible way.”

In an update posted Tuesday, Reynolds revised the estimate of misissued live certificates to about 12,000 and another 273,784 certificates that were “orphaned,” meaning they were stopped in mid-issuance for reasons including requestor cancellation and system errors. Reynolds said that the original estimate of more than 1.8 million certificates was based on a “more aggressive criteria than necessary.” Caudill and other researchers asked Reynolds to provide additional details before accepting the revised number.

An Apple official said here that the total number of non-compliant certificates his company issued was about 878,000, although the number of certificates that were still valid, and not expired and not revoked as of last Thursday, was about 558,000. A Google official, meanwhile, estimated the company had issued more than 100,000 non-complying certificates since 2016, but that as of late last month, only about 7,100 of them remained valid.

Both Apple and Google use their publicly trusted authorities to issue certificates for use internally and by affiliated organizations. Caudill said additional certificate authorities may also be affected.

An Apple representative told Ars the company has taken the following steps:

• Stopped issuance of certificates with non-compliant serial numbers, and is continuing to work with users to revoke impacted certificates
• Configured the software to generate serial numbers with 16 octets, ensuring entropy greater than 64 bits
• Reinstated alerts for detecting serial numbers suspected to be insufficient in length
• Enhanced validator software that checks certificates for SSL Baseline compliance to evaluate collections of certificates instead of individual certificates. These enhancements are expected to be implemented by April 30, 2019.

A Google representative provided this link as comment for this post.

Update on 3/14/2019, 6:56 PM: GoDaddy now says that the 63-bit issue affects zero certificates it has issued.