cargo-semver-checks v0.25 squashes nearly all bugs related to doc(hidden) items — its most common source of false-positives. What does doc(hidden) mean in Rust, and why was handling it correctly so hard?

cargo-semver-checks is a linter that helps prevent accidental breaking changes in the public APIs of Rust libraries.

Seems simple enough, no? And yet, almost every word in that sentence is unexpectedly load-bearing!

In today's post, we dig into how "public API" isn't the same thing as "all pub items," and why that caused headaches for cargo-semver-checks users prior to v0.25. Resolving those headaches is now as simple as upgrading to the latest cargo-semver-checks!

Discuss on r/rust or lobste.rs.

TL;DR + table of contents

The #[doc(hidden)] attribute is a way to mark pub items exported by a crate as "not public API" and therefore exempt from semver obligations. It's commonly used — any Rust project likely has at least one dependency containing public doc(hidden) items.

Prior to v0.25, doc(hidden) could be a huge problem for cargo-semver-checks users. Our study of semver compliance in the top 1000 Rust crates showed it was the source of over 60% of our false-positives. The study ended up discarding nearly half (46.6%) of all issues reported by cargo-semver-checks as doc(hidden) false-positives alone. To put it mildly: not great. Thankfully, these false-positives were not evenly distributed: most crates had zero or very few and were able to use cargo-semver-checks without issue, while a minority of crates were doc(hidden) Georg and cargo-semver-checks was practically unusable for them. The next section offers intuition for why this is the case. For crates that heavily rely on doc(hidden), using cargo-semver-checks was a frustrating experience whenever the hidden items changed.

cargo-semver-checks v0.25 addresses this problem. A tiny handful of rare edge cases remains unsolved since it first requires another feature: the ability to detect whether, and to what degree, a trait is sealed. As readers of this blog know, that's also more complex than it may seem at a glance. Fortunately, real-world data shows those cases are extremely rare in practice — and we have a path toward handling them correctly in the future. More on this in future posts! It's not every day that one eliminates over 60% of outstanding bugs! 🎉

This was a long-term team effort that stands on the shoulders of giants. David Hewitt, a maintainer of PyO3, helped flesh out the initial idea and prototypes. I consulted the rustdoc maintainers about the subtleties of doc(hidden). All of us benefit from shockingly good tooling in the Rust ecosystem, from rustc's diagnostics onward. And there's no way this would have shipped today without the sponsors that fund my work. If you or your employer use cargo-semver-checks, please consider supporting my work! If you aren't sure how to approach your employer about this — let's chat!

Contents

The doc(hidden) attribute

If you are already familiar with doc(hidden), feel free to skip ahead.

Here are a few examples of using doc(hidden):

#[doc(hidden)]
pub struct Example;

#[doc(hidden)]
pub fn frobnicate() {
    // implementation here
}

#[doc(hidden)]
pub mod hidden {
    pub struct AnotherExample;

    pub fn frobnicate_more() {
        // more implementation
    }
}

All items in this snippet are public in the sense of "accessible outside their crate." None of them are part of the crate's public API. If this crate had a docs.rs page, none of these items would show up on it. We can verify this in the real world: the impl_ module in PyO3 is public but hidden — can you find it in the module listing on its docs.rs page?

The most common use case for doc(hidden) is in crates that export macros. Code generated by macros is part of the consumer's crate, so it can only call pub code from the macro definition's crate. In practice, that requires large portions of macros' implementation details to be pub even though they are never intended to be used directly by end users. Changes to internal implementation details shouldn't require a major version bump, whether in macros or not! Aside from "feeling right," there's also a pragmatic argument here: cargo will not automatically update libraries to new major versions. We don't want projects to end up with multiple major versions of macro crates for no reason — that just adds bloat, wastes compile time, and worsens maintainability. At scale, even small amounts of friction can drastically change outcomes.

Prior to v0.25, cargo-semver-checks was oblivious to doc(hidden) and treated all public items as public API. This worked just fine for many crates! However, crates like PyO3 that define many complex macros often found that cargo-semver-checks reported too many false-positives to be practical. As of v0.25 — not anymore!

Why did handling one Rust attribute take well over a year?

Because none of the obvious approaches would have worked.

Why the obvious solutions don't work

"Easy," one might say, "just pretend #[doc(hidden)] items don't exist at all."

And here I thought we were trying to fix false-positives, not cause them 😂 Watch this!

pub enum Example {
    Regular,

    #[doc(hidden)]
    Sneaky,
}

Say we make Example::Sneaky no longer doc(hidden). Is this a breaking change? No — we've only expanded the public API surface area. Any code that worked before will continue to work. One can argue that clippy should raise a warning lint here, since it's probably a good idea to make the enum non-exhaustive if we're hiding its variants. I'm sympathetic to that argument! But as of Rust 1.74, this is legal, lint-free Rust code.

Recall how cargo-semver-checks lints work: they look for differences between a baseline ("already published") version and a "current" version that is being checked. One example of such a difference is when an exhaustive enum gains a new variant: Here's the implementation for that lint. 

// Exhaustive enum -- no `#[non_exhaustive]` attribute.
pub enum SomeEnum {
    First,

    // Added in the new version.
    // Breaking change!
    Second,
}

Adding SomeEnum::Second is a major breaking change! SomeEnum's public API supports exhaustive pattern-matching, which will now require a case for SomeEnum::Second as well.

Now let's go back to the previous snippet:

pub enum Example {
    Regular,

    #[doc(hidden)]  // removed in new version
    Sneaky,
}

If cargo-semver-checks pretended that doc(hidden) items don't exist, then Example::Sneaky looks just like SomeEnum::Second — a brand-new variant in an exhaustive enum, an apparent breaking change!

Behold, a false-positive!

And far from the only one! Pretending doc(hidden) items don't exist would also have failed us in a dozen other ways. Another failure mode is a variant of the re-exports issue that I discussed in a previous post, where knowledge of private items is required to determine which names are publicly visible in a module. There are more — let me know if you'd like me to write about them!

I spent months coming up with ideas, then finding counterexamples. This is how a few hundred lines of code can take over a year to write.

"Lints are queries" to the rescue!

In the end, the query-based design of cargo-semver-checks once again played a key role in the solution.

Each lint in cargo-semver-checks is implemented as a query over an abstract data model implemented as an adapter for the Trustfall query engine. I've written up how those queries work in a prior post, and you can try them out by querying Rust crates' data in our query playground. For example: "Find the enums and their variants defined in the itertools crate."

The data backing that abstract model is derived from rustdoc JSON, but the additional layer of abstraction buys us a lot of flexibility. We already rely on that flexibility in two ways I've touched on in previous posts:

We handle doc(hidden) by leaning on that flexibility just a bit more. We add two new "synthetic" fields to our data model: "Synthetic" because they don't per se exist in rustdoc JSON, and are instead computed on-demand by the Trustfall adapter for rustdoc. 

interface Item {
  # <... existing properties and edges ...>

  """
  Whether this item is eligible to be in the public API.

  # <... more docs ...>
  """
  public_api_eligible: Boolean!
}

type ImportablePath {
  # <... existing properties and edges ...>

  """
  This path should be treated as public API.

  # <... more docs ...>
  """
  public_api: Boolean!
}

These two properties capture the two ways that public items can be suppressed from the public API: by marking the item itself doc(hidden), or by making sure all paths under which that item is visible are themselves doc(hidden).

In other words, these two properties are necessary to handle the following edge case:

#[doc(hidden)]
pub mod foo {
    pub struct Bar;
}

// `this_crate::Bar` is public API here!
//
// `this_crate::foo::Bar` is pub-and-hidden,
// but this re-export is not hidden
// and neither is `Bar` itself.
pub use foo::Bar;

In this case, cargo-semver-checks detects that Bar is importable from this crate in two ways: this_crate::Bar and this_crate::foo::Bar. The former is public API, and the latter is not since it passes through the hidden this_crate::foo module.

In our data model:

Equipped with these new additions, making cargo-semver-checks lints aware of doc(hidden) was just a matter of updating their queries to use these new properties appropriately.

Updating 50+ queries sounds like a lot of work, but in reality it wasn't that bad: it was just a matter of adding an extra clause or two to queries that are type-checked by the Trustfall engine, then checked for correctness against a substantial test suite. If our lints were specified imperatively, the change could have been a hundred times harder to get right.

Declarative lints for the win! 🚀

What about deprecated items?

Our semver study of the top Rust crates helped contribute many excellent edge cases that guided the handling of doc(hidden) items in cargo-semver-checks. It was great to find these edge cases proactively — before launching this new functionality — rather than retroactively by having annoyed users open bug reports post-launch 😅 In my recent talk at P99 CONF, I argued that having high performance is what made it viable to find such issues proactively. If cargo-semver-checks were slower, scanning 14000 Rust crate releases would have been infeasibly expensive. In my (admittedly biased) opinion, I think building on top of Trustfall is a key reason why cargo-semver-checks was able to achieve high performance despite all the domain's challenges and without expending an overwhelming amount of effort.

Via the results of that study, I learned that maintainers sometimes use doc(hidden) when deprecating public API items. This allows them to suppress the deprecated items from documentation without breaking code that still depends on them.

#[deprecated = "Use crate::Other instead."]
#[doc(hidden)]
pub struct LegacyCode;

It turns out this is reasonably common! As such, cargo-semver-checks must accommodate this use case as well.

Our answer: while normally doc(hidden) items are not public_api_eligible, items that are both deprecated and hidden are considered public_api_eligible and remain in the public API. This allows us to ensure that public APIs remain unchanged both during and after their deprecation.

This approach also introduces a small quirk: if an item is both hidden and deprecated, removing the deprecation is considered a breaking change — it makes the item no longer public API. Whether this is correct or not is debatable, and likely best decided on a case-by-case basis. Multiple reasons make me feel this is unlikely to be a problem in practice:

I'd like to hear from you!

That's a wrap for the biggest feature of cargo-semver-checks v0.25!

What do you think? Are you using cargo-semver-checks in your project? Please let me know!

Whether your opinion is positive, negative, or just plain "meh," I'd like to hear about it. Feedback helps drive the project forward: positive feedback is motivation, and constructive negative feedback helps improve the tool in places where it needs improvement the most.

I'd love to make cargo-semver-checks my full-time job that pays my rent, and I'd love your help with that as well.

Is there a feature or improvement that your team or company really wants? Do you want to lint your project's codebase for other things beyond just semver? Is your organization interested in a private talk about the tech behind cargo-semver-checks and how it can best be used and scaled up? I can do all this and more — give me a ping!

Discuss on r/rust or lobste.rs.