Aaron Peters

EV Certificates Make The Web Slow and Unreliable

  • 15 min read

Extended Validation (EV) certificates are a bad choice for web performance. They have a much bigger negative impact on website speed and reliability than Domain Validation (DV) certificates. The EV certificate significantly increases the time it takes to secure the connection between browser and server and therefore extends how long users stare at a blank screen, waiting for the page to start rendering. Perhaps more importantly, using an EV certificate means the reliability of your website depends on your Certificate Authority's infrastructure: if the CA's server is down, your site is down.

Want to see just one image to know EV certificates are bad for performance?

EV Certificate with OCSP Staple - Recovation Check Fail in Chrome

That is what Chrome users see when visiting a website that uses an OCSP stapled EV certificate and the Certificate Authority's server (the OCSP responder) is down. That's right, OCSP stapling is pretty much useless with EV certificates.

OCSP stapling allows the presenter of the certificate (the server) to check with the CA if the certificate has been revoked and then add ("staple") this information to the certificate. Consequently, the browser may skip the revocation status check.

Let's dive deep into the world of DV and EV certificates, online revocation status checks and OCSP stapling and find out how Chrome and Firefox behave in case the CA's server responds quickly, or not at all.

Table of contents:

Testing Online Revocation Status Checks in WebPageTest

Most of the data for this 'research' was collected using WebPageTest. WebPageTest in general is great for web performance analysis, but an especially good fit here because it exposes the online revocation status checks (Firefox/Chome Dev Tools do not!) and you can easily simulate the failure of OCSP responders. Read the next section to learn what I did in WebPageTest, or skip to DV Certificates and Web Performance.

WebPageTest makes it easy to see how the loading of a webpage is impacted by a failing (third party) domain. Here, that failing domain is the CA's OCSP responder, so let's get this first.

Get the Domain of Your Certiticate's OCSP Responder

In Chrome, navigate to the site you want to test, for example www.cloudflare.com. Click the lock icon in the address bar and then click Certificate. In the new modal window, scroll down until you see Online Certificate Status Protocol. The value of the URI, without the http:// is the domain of the CA's OCSP responder. In the screenshot below, the domain is ocsp.digicert.com.

See OCSP Responder Domain in Chrome

Run SPOF test on WebPageTest

On the WebPageTest homepage, first enter the URL you want to test and select a test location and browser.

Before you start the test, open the SPOF tab and enter the domain of the OCSP responder:

WebPageTest SPOF Example

WebPageTest will now run a normal test and a test with the domain(s) blackholed, to then show a video comparison. You can also access the waterfall chart and all details for each test.

In WebPageTest, blackholing means the browser tries to connect to an IP address and never gets a response. Browsers are very patient at this stage and will typically wait ~60 seconds for response from the server.

Does the Certificate Have an OCSP Staple?

Before interpreting the WebPageTest results, you will want to know if the website's certificate contains a stapled OCSP response. You can't see this in the browser, but it's trivial to spot with OpenSSL:

echo QUIT | openssl s_client -servername www.cloudflare.com -connect www.cloudflare.com:443 -status 2> /dev/null | grep -A 17 'OCSP response:' | grep -B 17 'Next Update'

If the certificate is not stapled, the command will show no output.

DV Certificates and Web Performance

The 2019 Web Almanac provides a list of the top ten Certificate Authorities. All of these CAs are issuers for DV certificates, meaning most websites use a Domain Validation certificate.

Based on my data, using an OCSP stapled DV certificate is a great choice for web performance :

  • Chrome never checks the revocation status with the CA for any DV certificate
  • Firefox always checks a DV certificate's revocation status with the CA, unless the certificate is OCSP stapled (then it never checks with the CA)

With the TL;DR out of the way, let's take a closer look at Chrome and Firefox behaviour, starting with websites that serve a certificate that has not been stapled.

DV Certificate - No OCSP Staple

T-mobile.nl uses a non-OCSP stapled DV certificate, so browsers can't know just from the certificate whether or not the certificate has been revoked.

Chrome

Below are two (truncated) WebPageTest waterfall charts.

If Chrome checks the revocation status of T-mobile's certificate online, the very first request in the WebPageTest waterfall chart will be for ocsp2.globalsign.com.

Let's first look at a normal test run (load the URL in new browser instance with empty cache):

DV Certificate Without OCSP Staple - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

Chrome did not send a request to the OCSP responder. To confirm Chrome's behaviour, I ran tests that blackhole ocsp2.globalsign.com.

DV Certificate Without OCSP Staple - Responder Blackholed - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

Again, no request to ocsp2.globalsign.com.

Apparently, when Chrome is presented a DV certificate without an OCSP staple, it simply assumes the certificate has not been revoked and does not check online with the CA. This is great for performance.

Firefox

The same tests give very different results in Firefox.

Result for the normal test:

DV Certificate Without OCSP Staple - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

Aha! Firefox sends a request for http://ocsp2.globalsign.com/gsorganizationvalsha2g2. This is the browser checking revocation status of the certificate with GlobalSign.

FYI, WebPageTest renders this as the top request in the waterfall chart but it's actually the second request because - obviously - the browser first needs to receive the certificate from the www.t-mobile.nl server.

The fetch from ocsp2.globalsign.com took 106 ms incl. DNS lookup. That's a good, low number but still: this request makes the T-mobile page load 0.1 second slower.

The request to GlobalSign's OCSP responder is handled by Fastly CDN (could also have been Cloudflare, as GlobalSign uses multiple CDNs).

Inspecting the Fastly response headers provides an interesting insight:

X-Served-By: cache-sin18045-SIN, cache-ams21035-AMS
X-Cache: MISS, HIT
X-Cache-Hits: 0, 4

Fastly docs help interpret these headers:

  • the response was a cache hit served from an edge server in Amsterdam
  • if Amsterdam would not have the response in cache, that edge server would first fetch it from the Fastly datacenter in Singapore and then send to the browser

Amsterdam <=> Singapore is probably ~100 ms, so let's hope Fastly has a very high cache hit rate at the edge for OCSP responses.

Now, what will happen in Firefox after re-opening the browser without emptying the cache and then loading the same page?

DV Certificate Without OCSP Staple - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

Hooray! The request to ocsp2.globalsign.com did not go out. Apparently Firefox cached the OCSP response for T-mobile's certificate and used that.

To my surprise, Firefox again sends OCSP requests for http://ocsp.pki.goog/gts1o1 and http://status.geotrust.com/. This is odd because just earlier Firefox received OCSP responses with explicit 'do cache' headers: Cache-Control: public, max-age=86400 and Cache-Control: max-age=128876 respectively. I don't have an explanation for this, so let's move on.

The next question to answer is: what happens if GlobalSign's OCSP responder does not respond at all?

DV Certificate Without OCSP Staple - Responder Blackholed - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

Two things stand out in this Firefox waterfall chart:

  1. there is no request to ocsp2.globalsign.com ๐Ÿค”
  2. the first request to https://www-t-mobile.nl/ has a TLS handshake time of 2 seconds ๐Ÿ˜ฆ

I don't know why WebPageTest does not show the request to ocsp2.globalsign.com, but I'm 100% sure the request did go out because of the big increase in TLS handshake time for https://www-t-mobile.nl/ and the awesome Ryan Sleevi confirming to me Firefox has a soft-fail-after-2-seconds policy for online revocation status checks for non-EV certificates.

Knowing Firefox did not receive anything from the CA's OCSP responder, will the browser behave the same when re-visiting the page after a short time?

DV Certificate Without OCSP Staple - Responder Blackholed - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

How about that? TLS handshake time is low and no request to ocsp2.globalsign.com.

Ryan Sleevi explained to me Firefox will store the certificate in local cache even if the revocation status check was aborted after 2 seconds and Firefox will then use that cached certificate until expired without checking again with the CA.

DV Certificate With OCSP Staple

We already know that Chrome doesn't bother checking the revocation status online for DV certificates that do not have an OCSP staple. Surely the browser does also not do that check for DV certs with the staple, right? That would make no sense at all, but always good to verify.

And Firefox ... does stapling the certificate effectively get rid of the slowness introduced by that online revocation status check?

I tested with the KLM website - https://www.klm.com/ - and got all the answers.

Chrome

As expected, Chrome doesn't bother to check the revocation status of the OCSP-stapled DV certificate. Below is the waterfall chart for a test run with the OCSP responder blackholed:

DV Certificate With OCSP Staple - Responder Blackholed - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

Firefox

I'm only showing the waterfall charts for the tests with ocsp.comodoca.com blackholed because that's enough to know Firefox does not check revocation status online for OCSP stapled DV certificates ๐Ÿ˜ƒ !

DV Certificate With OCSP Staple - Responder Blackholed - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

Close browser, open browser and load https://www.klm.com/ again:

DV Certificate With OCSP Staple - Responder Blackholed - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

Again no OCSP request goes out for KLM's certificate and this is expected.

If someone at Akamai or KLM is reading this: do take a close look at the full WebPageTest results because weird things are happening. Browser and server first do HTTP/1.1 and later switch to H/2 (in the first run, first view, the switch happens after request 81). Also, why does Firefox load all those images from network in the repeat view instead of from cache?

EV Certificates and Web Performance

Read all of the above? OK, you now have a good view on how browsers behave when being presented with a DV certificate, but what about EV certificates?

Do Chrome and Firefox behave differently when processing EV vs DV/OV certificates? For example, will Chrome check revocation status online? How long will these browsers wait for the CA to respond and if no response arrives in time, what happens?

TL;DR

  • Chrome always checks revocation status with the CA regardless of the EV certificate being OCSP stapled or not and regardless of Chrome having the certificate and/or a "not revoked" OCSP response in its local cache
  • Firefox also performs the online status check for EV certificates even if it has the staple, with one exception: Firefox will re-use a "not revoked" OCSP response from its local cache, but only if the certificate contains a "not revoked" OCSP response (weird!)
  • Chrome patiently waits for the CA's response until the request times out (30+ seconds) and unless it receives a "not revoked" response in time, Chrome will hard-fail and present the user an error page
  • Firefox is more forgiving but less patient: the browser only aborts securing the connection and loading the page if the CA sent an explicit "not revoked" response within 10 seconds; in all other scenarios Firefox happily continues

Let's look at the data that backs all this up, starting with non-stapled EV certificates.

EV Certificate - No OCSP Staple

KPN, the #2 telecom provider in The Netherlands, uses an EV certificate on their main website https://www.kpn.com and it's not stapled.

Chrome

EV Certificate Without OCSP Staple - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

Chrome checks the revocation status not only of the server/leaf certificate but also (and first) of the intermediate certificate. One DNS lookup, one TCP connect and two sequential HTTP requests later, these two requests have added a big 0.3 seconds to the time the user is waiting for something to appear on the screen. Ugh.

What happens when Chrome navigates to the same site again after a short while? Sadly, the same as before: two blocking OCSP requests:

EV Certificate Without OCSP Staple - Chrome - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

Chrome does not re-use revocation check responses from cache for EV certificates. ๐Ÿ˜ฎ

The next scenario I tested is the blackholed OCSP responder:

EV Certificate Without OCSP Staple - Responder Blackholed - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

What is going on here?

Ryan Sleevi was so kind to explain:

  • this definitely is failure due to the online revocation status check
  • Chrome waited for the OCSP response until Chrome's connection timeout threshold was hit (30 seconds on Windows, 60s on Linux/ChromeOS), and then aborted the page load process
  • the request to http://www.gstatic.com/generate_204 is the "captive portal check chrome does on startup"

Chrome aborts the connection to a domain that serves an EV certificate in case the online revocation status check fails (the CA's OCSP responder is down or super slow).

The repeat view paints the same picture so I'm not showing the same waterfall chart twice.

Firefox

Launch a fresh Firefox instance and navigate to https://www.kpn.com:

EV Certificate With OCSP Staple - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

Firefox shows the same behaviour as Chrome: check status of the server/leaf certificate and of the intermediate certificate.

Close and re-open the browser and load https://www.kpn.com/ again:

EV Certificate With OCSP Staple - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

Why does Firefox do the two OCSP requests three times? I have no clue.

Next test is the blackholed OCSP responder:

EV Certificate With OCSP Staple - Responder Blackholed - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

Firefox is less patient than Chrome: after waiting for ~ 12 seconds for the CA's server to respond, Firefox moved on and loaded the page. This means Firefox is more forgiving than Chrome when it comes to checking revocation status of EV certificates online.

Close browser, open browser and load https://www.kpn.com/ again:

EV Certificate With OCSP Staple - Responder Blackholed - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

If the certificate status check failed before, Firefox will try again. Makes sense.

EV Certificate - With OCSP Staple

Vodafone is one of the bigger mobile ISPs in The Netherlands and active in the premium consumer and business market segments. The https://www.vodafone.nl/ uses an EV certificate that contains a stapled OCSP response.

I repeated all the same tests with the objective to answer the following question:

Does OCSP stapling help mitigate the web performance problems for EV certificates?

This can also be phrased as:

Does stapling the OCSP response to an EV certificate have a positive impact on website speed and reliability?

Let's first take a look at Chrome.

Chrome

EV Certificate With OCSP Staple - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

The EV certificate is stapled, but Chrome still checks the revocation status of the server/leaf certificate online.

Interestingly, the status of the intermediate certificate is not checked online. I expected the browser to send an OCSP request to the root CA (here: DigiCert High Assurance EV Root CA) but it did not. Maybe the root certificate is in the local machine's trust store and Chrome got a positive response from that.

The repeat view waterfall looks the same so it's clear Chrome does re-use revocation check responses from cache for stapled EV certificates:

EV Certificate With OCSP Staple - Chrome - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

The next scenario I tested is the blackholed OCSP responder:

EV Certificate With OCSP Staple - Responder Blackholed - Chrome - Waterfall Chart

Click image to view the full WebPageTest results

This looks familiar ๐Ÿ˜ž

Overall, Chrome behaves the same for stapled and non-stapled EV certificates.

Let's move on and find out if Firefox has different behaviour for EV certificates.

Firefox

New browser instance with empty cache:

EV Certificate With OCSP Staple - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

This looks the same as the waterfall chart for Chrome.

But things get more interesting when we examine the repeat view test results:

EV Certificate With OCSP Staple - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

No OCSP request! Apparently, Firefox does store "not revoked" OCSP responses in a local cache and re-uses these for some time. That's good for performance, so +1 for Firefox.

Now let's see what happens in Firefox when we blackhole the OCSP responder:

EV Certificate With OCSP Staple - Responder Blocked - Firefox - Waterfall Chart

Click image to view the full WebPageTest results

This is the same behaviour as when the certificate does not have the OCSP staple.

Repeat view:

EV Certificate With OCSP Staple - Responder Blocked - Firefox - Repeat View - Waterfall Chart

Click image to view the full WebPageTest results

Here too Firefox behaves the same as for EV certificates sans OCSP staple.

Key Insights

Do you want your website to be fast and reliable? Don't use an EV certificate.

Even if the certificate has an OCSP staple ... Chrome always sends a blocking request to the Certificate Authority's server when connecting to a website that uses an EV certificate and this request can take hundreds of milliseconds. To make things worse, if the CA's server is down, your users see an error page instead of your website.

Firefox also sends that blocking request every time the browser is establishing a new connection to your server, except when your EV certificate has a valid OCSP staple and the browser has recently checked the cert's revocation status online with the CA self (this is a bit weird). In other words: some returning visitors will benefit from the OCSP staple in your EV certificate but not new visitors for whom your site is slower and less reliable than it can be.

For optimal web performance, serve an OCSP stapled DV certificate: Chrome never checks the revocation status with the CA for any DV certificate and Firefox won't either if the certificate is stapled.

Besides the performance benefits, DV certificates are also much cheaper than EV certificates and less hassle to acquire, so why even consider an EV certificate?

Better security because site visitors have more trust in a website with an EV certificate? That argument is no longer valid as browsers don't show that 'green bar' anymore for EV certificates.

Closing Remarks

A big thanks to Ryan Sleevi for taking the time to provide me with a wealth of information about certificates, OCSP and performance. You rock!

For those interested in reading another lengthy article about SSL/TLS, OCSP and performance: Matt Hobbs published an awesome article yesterday with the title "The impact of SSL certificate revocation on web performance". I highly recommend reading it. It's complementary to my article and provides a lot of information about topics like Chain of Trust and Certificate Revocation List.

A few days after first publishing this article, I posted some stats about the certifcates the big sites in The Netherlands are using in The Sorry State of SSL/TLS Certificates in NL.

Webmentions

4 Replies

Matt Hobbs Matt Hobbs
Superb post Aaron! I completely forgot about the SPoF functionality in WPT ๐Ÿ˜ฌ. I'll crosslink it with mine later today ๐Ÿ’ช
Christian Schaefer Christian Schaefer
Wow, totally not my usual webperf ground but super interesting learnings. Thank you!!
Aaron Peters Aaron Peters
Blackhole on local machine: add an entry in /etc/hosts pointing the OCSP responder domain to the IP of blackhole.webpagetest.org
Secret Hero Secret Hero
That is exactly what I was looking for, and it's another really well written post! ๐Ÿ‘ I've also just realised that I asked you how to black hole a connection, when I wanted to ask what would happen if you did, so thanks for reading my mind there ๐Ÿ˜‚