Architecture
The four roles, who actually writes to the action cache, and which parts of the system hold state you can lose.
Who this is for: anyone deciding how to deploy NativeLink, or trying to predict which component's failure costs them what. What you'll have at the end: the four roles, the path an action takes on a miss and on a hit, and an accurate answer to "what do I have to back up". Time: fifteen minutes.
NativeLink implements the Remote Execution API, and that API describes four roles: a content-addressable store, an action cache, a scheduler, and workers. It is worth being precise about what "four roles" means here, because it is the source of the most common deployment mistake.
Four roles is the shape of the protocol, not the shape of the program.
One nativelink binary exists. Which roles it plays is decided by the
config file you hand it; there are no per-role images and no build features.
A single-developer setup runs all four in one process; a large cluster runs
several deployments of the same image with different configs. See
the architecture deep dive for what
that binary actually does at startup.
The four roles
CAS: content-addressable storage
Every byte NativeLink touches lives in the CAS, keyed by a digest of its content. Source files, object files, binaries, stdout, all of it. Two identical files anywhere in your organisation collapse into one stored blob, because they hash to the same key.
The digest function is SHA-256 or BLAKE3, negotiated per request rather than fixed at build time. Blobs stored under different digest functions do not share a namespace: switching the function is equivalent to starting with an empty cache.
The CAS is the only role holding data you would miss. Lose it and every cached artifact is gone. It is also not one thing: a production CAS is a stack of stores composed in config. The store model is the page for that.
AC: action cache
The action cache maps action digest → ActionResult. The action digest
covers the command line, every input file's digest, the environment
variables, and the platform properties, so an AC hit means this exact
computation has been performed before, and here is where its outputs are.
The AC does not hold the outputs. It holds digests pointing into the CAS, which is why an AC entry whose CAS blobs have been evicted is a real failure mode rather than a theoretical one; see correctness and hermeticity.
Losing the AC is survivable: clients re-execute, workers refill it.
Scheduler
The scheduler receives Execute RPCs, decides which queued action goes next,
finds a worker whose platform properties satisfy the action's, and assigns
the work. It also merges duplicate requests, so two developers building the
same commit wait on one execution rather than two.
In configuration "the scheduler" can be a stack rather than a single
component: a cache_lookup scheduler can be placed in front of the real
scheduler to answer Execute from the action cache, so an action that hits
the cache never appears in queue depth. None of the shipped example configs
do this, because Bazel already calls GetActionResult before it calls
Execute. Scheduler internals covers
the merge rule, the ordering rule and the retry rule.
Workers
A worker fetches the action's inputs from the CAS, materialises them into a directory, runs the command, uploads the outputs, and reports back.
Workers hold no durable state; a worker that dies mid-action costs you that action, not your cache. What they do hold is a local filesystem that is deliberately not clean between actions; worker execution explains why, and what it means for hermeticity.
The worker writes the action cache, not the scheduler
It is natural to assume the scheduler records the result, since the
scheduler is what answered the client. It does not. The worker writes the
ActionResult into its configured upload_action_result.ac_store itself
(as an UpdateActionResult call when that store is a gRPC proxy), from
upload_ac_results.
The operational consequence: the worker's ac_store is what decides
whether anything gets cached. A worker with no ac_store refuses to
start unless upload_ac_results_strategy is never; a worker with
never runs every action and caches nothing, with builds still
succeeding and no errors anywhere in the logs. The symptom is a cache hit
rate pinned near zero.
One build, twice
The concrete version. A developer runs bazel build //app:server, which
expands to roughly two thousand actions. Take one of them: compiling
main.cc.
Cold, nothing in the cache:
Bazel builds the action digest. The compiler's own digest, the command line,
main.ccand every header it includes, the platform properties. All of it hashes into one digest.GetActionResult(digest)→NOT_FOUND. A single round trip, a few milliseconds.Bazel uploads what's missing.
FindMissingBlobsfirst, thenBatchUpdateBlobsfor the ones the CAS does not have. On a cold cache this is most of the wall time.Execute(action)to the scheduler. The scheduler either merges this into an identical in-flight action or queues it.A worker is assigned. The action's platform properties (
OSFamily: Linux,container-image: ...) have to be satisfiable by a worker's declared properties, or the action queues indefinitely rather than failing.The worker runs it. Fetch inputs, materialise the input root, execute the command under a timeout, capture stdout, stderr and the declared output paths.
The worker uploads. Output blobs to the CAS, then the
ActionResultto the AC.Bazel downloads what it needs. With
--remote_download_minimalit may download nothing at all and keep only the digests.
Warm, the same build on a colleague's machine:
Step 2 returns an ActionResult instead of NOT_FOUND. Steps 3 through 7
do not happen. The scheduler is never contacted; no worker is involved; no
compiler runs. The cost of that action is one gRPC round trip plus whatever
output bytes the client actually wants.
That gap, one round trip against a compile, is the whole value proposition, and it is why the cache hit rate is the number worth watching rather than worker count.
Why the second build can still be slow
A high hit rate with disappointing wall time usually means the client is
downloading outputs it does not need. --remote_download_minimal (Bazel)
keeps intermediate artifacts remote. The other common cause is a
mismatched instance_name between client and server, which produces a
permanently empty cache with no error.
What holds state
Deployment planning comes down to this table, and one widespread claim about it is wrong.
| Role | Durable state | If you lose it |
|---|---|---|
| CAS | Yes, the blobs | Every cached artifact. Rebuild from scratch. |
| AC | Yes, but cheap | Clients re-execute and refill it. |
| Scheduler | Yes, by default | In-flight actions. See below. |
| Worker | No | The action it was running, which is retried. |
The scheduler is frequently described as stateless. It is not, by default: the awaited-action state (what is queued, what is executing, who is waiting on it) lives in memory in that process. Two scheduler replicas behind a load balancer with the default configuration are two independent schedulers that cannot merge each other's actions or answer each other's clients.
Making the scheduler horizontally scalable means moving that state out, which is what the Redis-backed state manager is for. Scheduler internals covers what that changes and what it costs.
Why Rust, why this shape
Three properties drove the design, and all three are operational rather than aesthetic.
No garbage collector. A build's tail latency is very visible: a p99 stall in the scheduler shows up as developers watching a progress bar. Removing the GC removes an entire category of latency spike that is difficult to tune away in a managed runtime.
Memory safety without a runtime. A cache serving a wrong artifact is a worse incident than a cache serving no artifact, because it is silent and it propagates. Rust removes the memory-corruption class of that bug at compile time while keeping predictable resource use.
Content addressing all the way down. No cache invalidation exists logic, because there is nothing to invalidate; a blob's key is its content. Eviction is a size-and-age policy rather than a correctness mechanism. Most of the hard bugs in a build cache come from invalidation, and this design does not have the concept.
Common questions
The shape of the program rather than the protocol: the crate graph, the startup ordering, and where an action's path forks inside the binary.
Troubleshooting
A symptom-to-cause-to-fix index for NativeLink, anchored on the error strings the system actually emits, including the ones that say nothing useful, and the failures that produce no error at all.
Architecture deep dive
The crate graph, the startup sequence, and the path an action actually takes through the binary.