Testing guide

Who this is for: anyone writing a test, or trying to work out why a test that passes in CI fails on their laptop. What you'll have at the end: the commands, the conventions, the fixtures that already exist, and a diagnosis list for the local failures that are not your code's fault. Time: twenty minutes.

NativeLink has two test runners over the same tests. Cargo is faster to iterate with; Bazel is what CI runs and what gates a merge. Both matter, and the vocabulary differs between them in a way that trips people up.

The commands

# Cargo: fastest loop. `smol` keeps target/ around 1GB instead of ~12GB.
cargo test --all --profile=smol
cargo test -p nativelink-store

# Bazel: what CI runs.
bazel test //...              # everything, plus rustfmt and clippy aspects
bazel test //:unit_tests      # in-file `mod tests` across all crates
bazel test doctests           # doc examples
bazel test //nativelink-store:integration   # one crate's tests/ directory

bazel test also attaches rustfmt and clippy aspects, which is why a formatting mistake shows up as a test failure rather than a separate lint step.

Where tests live

The convention is a separate tests/ directory per crate, not inline modules; only nine mod tests blocks exist in the whole tree.

A new test file has to be added to its crate's rust_test_suite in BUILD.bazel. Cargo picks it up automatically; Bazel does not.

#[nativelink_test]

Around 780 tests use it. It is defined in

and it is not a thin wrapper; it expands to five things:

• #[tokio::test], so the test body can be async
• #[::tracing_test::traced_test], capturing logs for assertions
• an error_span! named after the test function
• reseed_rng_for_test(), so randomised tests are reproducible
• a log-redaction assertion that fails the test if any captured log line contains " data: b" (with one carve-out for an aws_runtime line that legitimately does)

That last one is the surprising one. It exists to catch blob contents leaking into debug output. If your test starts failing with Non-redacted data in "..." and you did not change any assertion, something you touched is now logging a Debug of a buffer.

The forms:

#[nativelink_test]                          // the usual case
#[nativelink_test("crate")]                 // inside nativelink-util itself
#[nativelink_test(flavor = "multi_thread")] // args forwarded to tokio::test
#[nativelink_test(start_paused = true)]     // virtual clock, see below

Fixtures that already exist

Reach for these before writing your own.

Redis: nativelink-redis-tester provides a fake Redis, a MockPubSub, and a read-only Redis wrapper. Note that it is a regular dependency of nativelink-store, not a dev-dependency.

Time: MockInstantWrapped plus mock_instant::MockClock::advance lets you move a store's clock forward without sleeping. Combine with #[nativelink_test(start_paused = true)] for Tokio's virtual time.

Scheduler helpers: nativelink-scheduler/tests/utils/scheduler_utils.rs, pulled in with an inline mod utils { pub(crate) mod scheduler_utils; } block rather than as a crate.

Mongo: nativelink-store/tests/mongo_runner/ downloads and starts a real mongod. This is why the store crate's Bazel timeout is moderate rather than short, and why that suite needs network on a cold cache.

Temp paths: make_temp_path honours TEST_TMPDIR, so tests behave under Bazel's sandbox.

The end-to-end suites

./run_integration_tests.sh

It refuses to run as root, requires Bazel, Docker and sudo, and waits for ports 50051, 50052 and 50071 before starting. It drives simple_cache_test.sh, simple_remote_execution_test.sh, simple_tls_test.sh and chunking_cache_test.sh. You can pass a pattern to run one of them.

Beyond those, integration_tests/ also holds BuildStream, Buck2 and Mongo suites driven from Nix, and nativelink-test/fuzz holds a single fuzz target (cas_config.rs) run by ClusterFuzzLite.

Coverage

nix build .#nativelinkCoverageForHost

The result symlink contains an HTML report. No coverage threshold gate; no PR is blocked on a percentage. Coverage is a tool for finding untested paths, not a target to hit.

When it fails locally but not in CI

Most of these are environment, not code.

Common questions