The Performance Cost Of Spectre / Meltdown / Foreshadow Mitigations On Linux 4.19

One of the most frequent test requests recently has been to look at the overall performance cost of Meltdown/Spectre mitigations on the latest Linux kernel and now with L1TF/Foreshadow work tossed into the mix. With the Linux 4.19 kernel that just kicked off development this month has been continued churn in the Spectre/Meltdown space, just not for x86_64 but also for POWER/s390/ARM where applicable. For getting an overall look at the performance impact of these mitigation techniques I tested three Intel Xeon systems and two AMD EPYC systems as well as a virtual machine on each side for seeing how the default Linux 4.19 kernel performance -- with relevant mitigations applied -- to that of an unmitigated kernel.

The Linux 4.19-rc1 development kernel released this past weekend was used as the basis of the benchmarking across the systems. On the Intel side the relevant mitigations include page table isolation (PTI/KPTI) for Meltdown and then the various Spectre speculative execution mitigations including __user pointer sanitization, full generic Retpolines via IBPB IBRS_FW, speculative store bypass disable via prctl and seccomp, and for L1TF/Foreshadow is PTE inversion and conditional cache flushes for VMs. By default the Linux kernel doesn't enforce the "full" mitigation of disabling Intel HT/SMT support, so keep that in mind if you are running VMs and whether untrusted code/users have access to the VM that if you opt for the full mitigation where SMT is disabled the performance impact will be a lot more noticeable due to halving the number of threads available. Public cloud providers seem to just be adjusting their scheduler to ensure no SMT threads go across users to avoid the obvious huge toll of having to disable Hyper Threading. For this comparison today is just the stock/default kernel mitigations.

On the AMD EPYC side the default mitigations are just for their relevant vulnerabilities with __user pointer sanitization for Spectre V1, AMD Retpoline IBPB for Spectre V2, and speculative store bypass disable (SSBD) for Spectre V4.

After testing all of the configurations on the stock Linux 4.19-rc1 kernel, tests were repeated after using the various run-time switches for disabling mitigations. All of the systems were tested with Ubuntu 18.04.1 LTS x86_64 with the Linux 4.19-rc1 kernel via the Ubuntu Mainline Kernel PPA, up-to-date microcodes/BIOS, GCC 7.3, and the EXT4 file-system.

The system configurations tested for this comparison included:

- The Intel Xeon E3-1280 v5 Skylake processor on an MSI Z170A SLI PLUS motherboard, 16GB DDR4, and 256GB Toshiba RD400 NVMe SSD.

- The Intel Xeon E5-2687W v3 Haswell processor on an MSI X299 SLI PLUS motherboard, 32GB DDR4, and a 80GB Intel 530 SATA 3.0 SSD.

- A dual Intel Xeon Gold 6138 setup running within a Tyan 1U chassis and having 96GB of RAM and backed by a Samsung 970 EVO NVMe SSD 256GB.

- A single KVM-based virtual machine running on the above-mentioned dual Xeon Gold server. This VM was the only active process on the system and configured to access 80% of the CPU cores/threads (64 threads), 48GB of RAM, and a 118GB virtual disk. During the VM testing when running mitigations disabled both the guest and host kernels had the mitigation switches disabled.

- The AMD EPYC 7601 processor on a Tyan 2U server with 128GB of RAM and a 280GB Intel Optane 900p NVMe SSD.

- A single KVM-based virtual machine running on the above-mentioned AMD EPYC 7601 server. The VM had access to 80% of the CPU cores/threads (52 threads), 48GB of RAM, and 120GB virtual disk.

- An AMD EPYC 7551 server using a Gigabyte MZ31-AR0 motherboard, 32GB of RAM, and a Samsung 960 EVO 256GB NVMe SSD.

The hardware is obviously quite different and not intended to be compared in raw performance between these systems but rather looking at a diverse spectrum of systems to see the overall impact of these CPU vulnerability mitigations as of the Linux 4.19 kernel. As such, for the results all of the data was normalized against each system's unmitigated performance for an easy look at the relative cost across the board. All of these Linux benchmarks were carried out using the Phoronix Test Suite.

As always, if you enjoy my daily Linux benchmarking and open-source news coverage, consider showing your support by joining Phoronix Premium. If there is sufficient interest and support I will carry out a similar Spectre/Meltdown mitigation test on the desktop CPUs with Linux 4.19.