A Future for Rust Debugging

Recently, there has been a lot of progress in the Rust logging & tracing ecosystem: projects like tracing make our lives much simpler, allowing us to track down bugs even in complex asynchronous environments in production. However, they still can’t replace interactive debuggers like gdb or lldb, which provide so much more than just stack traces and variables printing.

In an ideal world, you can use your debugger as a read-eval-print loop, writing, rewriting, and testing entire functions in an interactive session. Techniques like post-mortem debugging make it possible to prod dead processes, providing immeasurable help in understanding the state of a program just before it crashed. But the harsh reality is that debuggers can be counter-intuituve, hard to use, or just suck, and it’s doubly challenging for Rust.

Considering the complexity of modern Rust projects, it’s sometimes outright impossible to debug certain cases because the commonly used debug tools like gdb or lldb are tailored mostly for C or C++, and they make certain assumptions about the way your code is written. Needless to say, these assumptions sometimes can be drastically wrong because C++ and Rust are very different languages. For example, here’s a relatively simple program that uses thread-local variables:

use std::cell::RefCell;
thread_local! {
  pub static XYZ: RefCell<u64> = RefCell::new(0);
}
fn main() {
  XYZ.with(|val| {
    *val.borrow_mut() = 42;
  });
}

And here’s what happens if I want to print the current value of XYZ in lldb:

The first intuitive attempt to print out the variable using the standard Rust syntax fails miserably because lldb doesn’t understand Rust at all; in fact, it uses a C++ expression parser instead. But even if we peek into the Rust thread-local vars implementation and use a C++ expression to print them, it wouldn’t work because lldb is made to think it works with C++, and it doesn’t know that Rust has a new name mangling scheme which has diverted from C++.

And thread-local variables isn’t a very contrived case: for example, they are used in the Tokio runtime implementation to store information about tasks, and it would be very helpful to inspect them for understanding how your async code is executed (or, at the very least, this kind of debug info can be helpful for developers of Tokio). Speaking of Tokio and the async ecosystem, it is impossible to print out async backtraces without also having to plod through async runtime internals. Sometimes a runtime can reschedule tasks, losing the original stack context in the process, and if you want to track the origin of an async failure in this case, you are left with only one option – logs and traces (if there’s an alternative way, I’d be happy to learn about it!).

All of these problems have the same underlying cause: Rust has to live in the C/C++-centric world. So what can we do about it (other than Rewriting-It-In-Rust, of course)? I think that the ideal systems programming language needs to have ideal, modern debugging tools.

Luckily for us, a debugger like lldb does not insist on using C/C++ for everything. It has a modular core, and languages support can be added as plugins – and that’s exactly what lldb does for C++, which is not really a special case, but a plugin that’s built on top of Clang, an LLVM-based C++ compiler. There has been an effort to implement a similar language plugin for Rust – which, however, is a tricky enterprise, because the lldb plugin API is private^[1] and it’s written – you guessed it – in C++. Implementing a parser for a subset of Rust in C++ is hard enough, but keeping up with all the changes in the language is barely possible.

But now, with an effort to “library-ify” Rust, I hope that this project can be reinvigorated. And while we are at it, we can try to apply the idea of compiler as a library in another interesting way: debugger as a library.

In this example, the private lldb plugin API, which implements the language support, calls into the shared Rust debugger library to parse debug expressions. The debugger library can then delegate parsing to Rust compiler libraries, and in turn use debugger APIs provided by lldb when needed^[2].

It should be noted that this hypothetical debugger library will work entirely in the Rust domain, so it will have the access to knowledge and power of mid-level internal representation. Of course, this doesn’t have to work for lldb only: the abstract interface can be ported to other debuggers. And if we make this library itself modular, it opens up a lot of interesting possibilities, like a more complete Rust REPL, custom visualizations, or domain-specific debuggers.

This is not a new idea and there are numerous good examples we can learn from. mdb, or a Modular Debugger, was extended to inspect NodeJS programs. The .NET Core runtime diagnostics tool has a debugger abstraction library with adapters for lldb and Windows Debugger. I’m sure there are many others, but most of these examples boil down to the same idea of extending the debugger core with language-specific capabilities.

Conclusion

Obviously, this is a huge effort, which might (and most certainly will) require lots of changes across different projects like LLVM, the Rust compiler, or even the DWARF debug format specification^[3]. Instability of private APIs is definitely one of the challenges here, but it can be partially solved by having stable forks – which is how it’s done in a large number of cases anyway^[4]. Supporting abstractions like traits can be another difficult task because debuggers are not particularly suited to work with generic code (which is compiled down to concrete types). But I believe that even a basic Rust-aware debugger can make a huge difference in users experience, and I deem it a worthy goal to pursue.

Thank you for reading this!

References

[1] [lldb-dev] Rust language support question.

[2] There is even a bindings crate for the public part of the lldb API.

[3] Tom Tromey discusses debugging support in rustc (video).

[4] E.g., see an Apple’s fork of lldb to add support for Swift.