Codebase internals

Who this is for: anyone about to write NativeLink code rather than read it. What you'll have at the end: the four things that are enforced rather than suggested: the store trait split, the error model, the spawning rules, and which metric system to use. Time: thirty minutes.

Most of NativeLink's review feedback is about conventions, not logic. The conventions are strong, mostly mechanically enforced, and each one exists for a reason you can point at in the source. This page is the set of them that a first contribution reliably trips over.

The repository targets Rust edition 2024 with rust-version = "1.93.1".

The three store traits, and which one you implement

Three of them exist, and they are not interchangeable.

StoreDriver (StoreDriver) is the one you implement. Its supertraits are Sync + Send + Unpin + MetricsComponent + HealthStatusIndicator + 'static, so a new store is observable and health-checkable by construction, not by remembering to add it.

Store is a #[repr(transparent)] newtype over Arc<dyn StoreDriver>. It is what everything else holds.

StoreLike is the caller-facing surface: two required methods, every convenience built on top of them as a default, and blanket impls so that both Store and impl StoreDriver satisfy it. You call StoreLike; you implement StoreDriver.

The StoreDriver methods you must supply are post_init, has_with_results, update, get_part, inner_store, as_any, as_any_arc and register_remove_callback.

StoreKey is a two-variant enum, Str or Digest, with a variant-salted Hash so that a string key and a digest key cannot collide.

Register health with the default_health_status_indicator!(YourType); macro as the last line of the file, unless the store has something real to report. Health checks across all components run concurrently (buffer_unordered(usize::MAX)), with a comment in

explaining that a serial sweep would blow the kubelet's probe timeout.

The scheduler traits are split by caller

No ActionScheduler trait exists. The state layer is split into ClientStateManager, WorkerStateManager and MatchingEngineStateManager in operation_state_manager.rs, plus WorkerScheduler, KnownPlatformPropertyProvider and AwaitedActionDb. The split by caller is what allows the in-memory and Redis-backed implementations to be swapped wholesale. See scheduler internals.

The error model

Code is tonic::Code, re-exported from Error. No separate internal code enum and no mapping layer exist, which is why an error raised deep in a store arrives at a client with a sensible gRPC status without anyone writing a conversion.

An Error carries a code, a Vec<String> of messages, and a context. The messages are a stack, joined with " : " when displayed. This is the whole point of the model: each layer appends what it was trying to do, so the final message reads outside-in.

some_store
    .get_part_unchunked(key, 0, None)
    .await
    .err_tip(|| "while loading the action result")?;

Three macros do most of the work: make_err!, make_input_err! and error_if!. ResultExt, and therefore err_tip, is implemented for both Result and Option, so an Option can become a well-described error without an intermediate ok_or_else.

Around twenty-five From<…> for Error impls exist for the common foreign error types, so ? usually just works.

clippy::todo is denied. clippy::unwrap_used is currently allow, with a TODO in the manifest recording the intent to flip it; write new code as though it were already denied.

Async: tokio::spawn is banned

The Tokio runtime is built by hand in main(), not with #[tokio::main]: the builder installs an on_thread_start hook (macOS QoS), and the process signal handlers and the shutdown broadcast are spawned onto it before inner_main runs. Each of those calls carries an #[expect(clippy::disallowed_methods, reason = …)], because the same lint that bans tokio::spawn everywhere else bans the runtime builder too.

Fourteen entries in

make the standard spawning API a lint error. Use the three macros in

instead:

The first argument to every one of them is a task name, which becomes a tracing::error_span!. That name is how a task is identified in logs and traces, so make it specific. OpenTelemetry context propagates into spawned tasks automatically.

File I/O goes through nativelink_util::fs, which serialises open file descriptors behind a global semaphore (DEFAULT_OPEN_FILE_LIMIT is 24576). Opening a file with std::fs or tokio::fs directly bypasses the limit and will eventually exhaust the process's descriptors under load.

Shutdown is coordinated by

with three priorities: P0, P1, LeastImportant. Awaiting wait_for at a given priority blocks until everything less important has drained. SIGTERM drains and then exits 143; SIGINT is a hard exit 130.

Two metric systems

They are not redundant, and picking the wrong one is a common review comment.

System A: nativelink-metric. A #[derive(MetricsComponent)] plus #[metric(help = …, kind = …, group = …, handler = …)] on fields, with RootMetricsComponent marking a tree root. Roughly 190 #[metric] attributes exist. Publishing a tree walks the live object graph and emits one tracing span per component and one event per field, on the target nativelink_metric, so what it describes is the structure of a running process. Nothing in the v1.6.5 binary serves that tree over HTTP; the old metrics collector went away with the OpenTelemetry migration, so today the derive is what keeps every store and scheduler describable, and the trait bound on StoreDriver is what forces new ones to be.

System B: nativelink-util/src/metrics.rs. OpenTelemetry instruments in eight LazyLock bundles (CACHE_METRICS, EXECUTION_METRICS, WORKER_METRICS, RPC_METRICS, SCHEDULER_METRICS, STORE_TIER_METRICS, HEALTH_METRICS, CONNECTION_METRICS), wired up in telemetry.rs::init_tracing and exported over OTLP (NL_OTEL_ENDPOINT overrides the endpoint). Attribute sets for the hot paths are pre-computed into CacheMetricAttrs and ExecutionMetricAttrs structs specifically to bound cardinality.

Choose A when you are exposing the structure of a component: how many stores, which config, what state. Choose B when you want a number someone will graph or alert on.

Instrument names are dotted OpenTelemetry style, with .with_description and a .with_unit that is either a UCUM unit ("By", "s", "ms") or a braced count such as "{entry}", "{action}" or "{worker}".

Lints, formatting and the small stuff

clippy::all, clippy::nursery and clippy::pedantic are all denied at the workspace level. std_instead_of_core is denied too, which is why you will see use core::… throughout for anything that does not need std.

Suppress a lint with #[expect(lint, reason = "…")] rather than a bare #[allow]; expect fails the build once the suppression is no longer needed, which is how they get cleaned up. #[allow] still appears where the lint only fires on some targets or feature sets, and it should carry a reason too. Nothing enforces this mechanically; reviewers do.

.rustfmt.toml uses group_imports = "StdExternalCrate" and imports_granularity = "Module", both nightly-only options. Do not run stable cargo fmt and expect it to match; run the Bazel target instead:

bazel run --config=rustfmt @rules_rust//:rustfmt

unsafe is rare: a few dozen blocks in the whole tree, mostly FFI, a few measured optimizations, and environment-variable writes in tests. Most carry a // SAFETY: comment, and a new one without a justification will be asked for one.

Common questions