posted by Matthew Flatt

With the upcoming version 8.0 release, Racket on Chez Scheme (Racket CS) will become the default implementation of Racket. This report discusses the implications of that change, continuing a series that was originally about Racket CS; the most recent previous report was the February 2020 report. The original January 2018 report explains the motivation behind throwing out around 200k lines of C code to replace it with around 150k lines of Scheme and Racket code.

Switching to Racket CS

For most users, the differences between Racket CS and Racket BC (“before Chez”) may be too small to notice. The installer and executables will be larger, because they contain machine code instead of bytecode, and some programs may run a little faster. Otherwise, Racket programs are supposed to run the same.

To test this, we’ve built and tested every package available from the package server on Racket CS over the past several months and compared the results with those from Racket BC version 7.9. Out of more than 1800 packages, currently only 12 now fail to compile, and 5 others fail some of their tests. The new build failures, as well as 2 of the 5 new test failures, are the result of changes at the C API level. Several packages already have pull requests to fix them, and we hope others will be fixed soon, too. The remaining 3 test failures are differences in thread scheduling that provoked existing bugs in the packages.

Although Racket CS will be the default for v8.0, Racket BC will remain available through a “More Installers” link on the download page for the foreseeable future. So, Racket users will still have the option of falling back to the old runtime system and compiler if something goes wrong, or if they need a package that does not yet work on Racket CS.

Progress in the Past Year

During 2020, Racket CS became a little faster and a little smaller, and many corners of the implementation were repaired or made more compatible with the BC implementation. The planned changes that were on the previous report’s roadmap all happened! In addition, we introduced an AArch64 backend and ported to run natively on M1 Macs. Finally, we made the Chez Scheme garbage collector run in parallel, which brought the performance of places-based parallelism in Racket CS about on par with BC.

Building Racket CS no longer involves first building Racket BC as a way to bootstrap Chez Scheme. Instead, Chez Scheme bootstraps itself on any supported platform using a new portable bytecode (pb) backend. Bootfiles for pb mode are checked into a separate Git repository that is used in a submodule-like way, but bootfiles can always be built from source by using an existing Racket build; the bootstrap implementation is now detangled so that much older Racket versions in the v7 series can work for bootstrapping.

Benchmark Performance

For traditional Scheme benchmarks, not much has changed in the results. A plot is practically indistinguishable from the one in the previous report.

For the shootout benchmarks, which measure more Racket-specific functionality and libraries, there’s some improvement, especially toward the end of the table. (Shorter is better.) The light-blue line shows the results from the previous report (which corresponds to version 7.6), so a shorter “CS” bar means improvement there. The improvements mostly reflect unboxed floating-point arithmetic.

Below is another set of microbenchmarks from the racket-benchmarks package, comparing Racket CS to BC. Some points to note:

• CS is usually faster than BC. (Shorter is better.)

• The fastest relative CS runs are much faster than the fastest relative BC runs; that is, the shortest blue bars are much shorter than the shortest red bars. Although it’s difficult to extract from these plots, the difference is really that CS performs more consistently, so it’s slowest relative runs are not as slow as the slowest relative BC runs.

• These benchmarks mostly compare subsystems that are implemented in C for BC to subsystems implemented in Scheme and Racket for CS. That’s the key and motivating point — the maintenance advantage, more than the performance advantage.

For the rest of this report, the plots will show CS slower or bigger, but gradually approaching BC performance. You should not read that as “CS is slower than BC.” This report is about the parts of the implementation where we spent time last year, and we don’t dwell on parts where no improvement was needed.

About the measurements: Benchmarks are in the "racket-benchmark" package in the "shootout", "control", "hash", and "chaperone" directories. We dropped some of the "control" benchmarks because they take to long to run on BC, which means that the plots here understate how much more consistently CS performs compared to BC.

Startup and Memory Use

Load times improved for Racket CS:

That’s mostly due to reduced code size:

On a different scale and measuring peak memory use instead of final memory use for DrRacket start up and exit:

The large drop in peak memory use for DrRacket CS is due to garbage-collection improvements for old generations in large heaps, where old-generation objects tend to be marked in place instead of copied.

About the measurements: These results were gathered by running racket with the arguments -l racket or -l drracket. The command further included -W "debug@GC" -e ’(collect-garbage)’ -e ’(collect-garbage)’, and reported sizes are based on the logged memory use before the second collection. For the “BC” line, the reported memory use includes the first number that is printed by logging in square brackets, which is the memory occupied by code outside of the garbage collector’s directly managed space. Baseline memory use was measured by setting the PLT_GCS_ON_EXIT environment variable and running with -n, which is has the same effect as -e '(collect-garbage)' -e '(collect-garbage)'. DrRacket was initialized with racket/base as the default language; also, background expansion was disabled, because measuring memory use is tricky on Racket BC.

Build Times

Compile times improved for Racket CS:

This improvement, combined with others, makes the CS distribution now build a little faster from source than a BC build, and it uses about 20% less memory than the BC build. The following two plots use the same scale, where the foreground blue or red line shows memory use (vertical) plotted over time (horizontal) as recorded on each major collection (see this page for a detailed description):

A shorter build time in less space represents a big milestone for Racket CS. Even though most Racket users start with a pre-built distribution, build time measures end-to-end performance across many parts of the implementation, and build performance correlates well with end-to-end performance in many Racket applications.

The build plots above are for a sequential (i.e., single-threaded) build. More typically, Racket is built on machines with more CPUs, and build times easily benefit from process-like parallelism. Over the past year, garbage-collector improvements for parallel collection have made build times with place-based parallelism dramatically shorter on a machine with 4 hyperthreaded cores (although build dependencies ultimately limit parallelism). Since parallel collection debuted in Racket CS v7.9 but has improved since, we show v7.9 as an intermediate point:

As the plot shows, as of the previous report (before parallel garbage collection), trying to build with multiple places was counterproductive. The speedup from places-based parallelism in v8.0 is finally close to BC’s speedup, which is about the same as using process-based parallelism (but with the advantage of staying within a single Racket process). The parallelism available from BC remains a little higher, because its implementation of places uses completely separate allocation heaps, while parallelism in CS use a single heap.

About the measurements: The compile-time results were gathered by using time in a shell a few times and taking the median. The build plots were generated using the "plt-build-plot" package, which drives a build from source and plots the results. The parallelism results were generated by starting with an in-place installation, using raco setup –fast-clean and rm doc/docindex.sqlite to start from a clean state, and getting elapsed real time with time racro setup –jobs 8, and dividing by elapsed time from build plots (which is not exactly the same build job, but is close).

Reflection and Outlook

We tried to declare success and move on a year ago, but that didn’t work. This time really is different, because Racket CS is ready to be the default Racket implementation. Time will tell whether that difference is enough to allow a different focus in the coming year. Given how much the CS compilation strategy (ahead-of-time) differs from the BC strategy (bytecode compilation plus a JIT), it’s not clear that the compile time, load time, and memory footprint differences between BC and CS can be reduced further.

In terms of maintainability, one possible direction for improvement is to shift even more Racket support into the Chez Scheme layer. Gustavo’s decision to implement type reconstruction as a Chez Scheme pass was good call. Contributor yjqww6’s implementation of Racket-style lifting as a Chez Scheme pass was a clear improvement for Racket CS, because it eliminated a larger pass on the “schemify” side and implemented at a better layer of abstraction with less interference on local optimizations. Similarly, extending Chez Scheme’s support for weak hash tables, which was not difficult, allowed us to discard a complex and slower implementation in the “rumble” Racket-adapter layer. More direct support for left-to-right evaluation guarantees and cross-linklet inlining could further reduce the schemify layer, and maybe one day eliminate it.

We’ve mostly kept BC and CS in sync, so far, but moving forward, Racket BC will not necessarily get new functionality that is added to CS. After all, the point of CS is to eventually shed technical debt in BC. This divergence has already started: Racket CS fully supports M1 Macs, but Racket BC runs only in interpreted mode or as x86_64 (i.e., we did not add an AArch64 JIT to Racket BC).

Thanks to Sam, Ryan, Robby, Matthias, and John for improving this report. Special thanks to Sam, Paulo, and Robby for moving CS testing forward. Many contributors helped track down and repair CS bugs, and besides others mentioned, thanks to Shu-Hung, Sorawee, and Bogdan for tackling some substantial repairs.