# NativeLink: full documentation corpus

> A remote build cache and remote execution service in one Rust binary, for
> build systems that speak the Remote Execution API: Bazel, Buck2, Siso,
> Pants, BuildStream, and CMake via recc. The cache, the scheduler and the
> worker are the same executable started with different configuration.

These docs are ordered as a path, not a catalogue: get a cache serving hits,
then put workers behind the misses, then own the config file, then reach for
the how-to guides. Each section assumes the previous one is working, and every
page on the path opens by stating what it assumes. An agent reading this file
in order learns the same sequence a human learns from the sidebar.

Getting started, then Remote execution, then Configuration, then How-to guides.
Why NativeLink comes before that path, and Operate comes after it. Concepts,
Reference and Contribute are outside the ordering: enter them from a search or
a link at any point.

This file is generated from the navigation (`meta.json`) and page frontmatter
by `web/apps/docs/scripts/gen-llms.mjs`, so it cannot drift from the sidebar.

Every page below is delimited by a heading of the form `# https://docs.nativelink.com/<path>`,
giving that page's canonical URL. The delimiter is the absolute URL rather
than the path because a bare `# /path` line is indistinguishable from a shell
comment naming a file, and several pages contain exactly that.

Page bodies are reproduced verbatim, MDX components included, so nothing is
lost in transcription; links inside them are site-relative and resolve against
https://docs.nativelink.com. The generated reference pages (the configuration
reference, the metrics reference and the changelog) are linked rather than
inlined: they are long, they change on a different cadence from the prose, and
they are better fetched on demand.


# https://docs.nativelink.com/

**Introduction**: NativeLink is a remote build cache and execution service in one Rust binary. This is the documentation for running it yourself.

NativeLink is a **remote build cache** and a **remote execution** service in
one program. Your build tool asks it "have you already built this?" and, when
the answer is no, "can you build it for me over there?" Your source code
doesn't change and your build files don't change: you point your build tool
at a URL, and work that used to happen on your machine stops happening on your
machine.

It's written in Rust and ships as a single binary that plays every role in the
system: cache, scheduler, and worker are the same executable started with
different configuration. It speaks the
[Remote Execution API](https://github.com/bazelbuild/remote-apis), an open
protocol rather than a NativeLink invention, which is why **Bazel, Buck2,
Siso, Pants** and **BuildStream** connect to it without a plugin, and why
plain CMake projects can via [recc](/getting-started/connect-your-build/cmake-recc).

This site documents the open-source distribution: setting it up, configuring
it, deploying it, operating it when other people depend on it, and
contributing to it.

<Callout type="info" title="Working with agents">
  If you're pointing a coding agent at NativeLink, at these docs or at the
  codebase, [For agents](/agents) has the three entry points and what each
  guarantees. If you're here because your *agents* are the ones waiting on
  builds, start with [AI coding agents](/use-cases/ai-coding-agents).
</Callout>

## The reading path [#the-ladder]

The first four sections are ordered, and each one assumes the last. You can
skip ahead, but the pages will tell you what they expected you to have.

| Section | What you have at the end |
| --- | --- |
| [Getting started](/getting-started) | A cache your build is talking to, and hits you can point at |
| [Remote execution](/remote-execution) | Cache misses executing on the farm instead of on your laptop |
| [Configuration](/configuration) | The ability to write a config from an empty file instead of copying one |
| [How-to guides](/how-to) | The storage backend, transport and upgrade path you actually want, instead of the ones an example gave you |

Two sections are ordered against the reading path without being part of it.
[Why NativeLink](/use-cases) comes before Getting started: what a remote cache and
remote execution actually do, and a short check for whether your build is one
that benefits. It's worth ten minutes if you haven't decided yet.
[Operate](/operate) comes after How-to guides, for when other people
depend on your cluster and "it works on my machine" stops being the bar.

Everything else sits outside the ordering: reference material you look things
up in, explanations you read when you want to know why, and the contribution
guides. Those are entered from a search or a link, at any point.

## Start where you are [#start-where-you-are]

### I'm deciding whether this is for us [#im-deciding-whether-this-is-for-us]

- [Why NativeLink](/use-cases): the mental model, and the 30-second test for
  whether your build benefits.
- [AI coding agents](/use-cases/ai-coding-agents): why an agent's build loop
  makes cache hit rate a token-cost question.
- [Silicon and large builds](/use-cases/silicon-and-large-builds): EDA flows,
  large C++ and LLVM trees, ML graphs.
- [Autonomous operation](/use-cases/autonomous-operation): what it takes to
  run a farm nobody watches.
- [Open source and Enterprise](/reference/oss-and-enterprise): what's licensed how, and
  what the paid tiers are actually for.

### I want to get something running [#i-want-to-get-something-running]

- [Quickstart](/getting-started/quickstart): a local cache in under ten
  minutes.
- [Connect your build tool](/getting-started/connect-your-build): Buck2,
  Siso, Chromium, Pants, BuildStream, CMake.
- [Shared cache](/getting-started/shared-cache): the single-server deployment
  that survives a team-wide rollout.
- [Your first remote action](/remote-execution/first-remote-action): once
  caching works and you want the misses to move too.

### I want to configure it [#i-want-to-configure-it]

- [The config model](/configuration): how a NativeLink config is put
  together, in one page.
- [Stores](/configuration/stores): declaring, naming and composing the
  storage a cache is built from.
- [Scheduler and workers](/configuration/scheduler-and-workers): how work
  gets matched to machines.
- [Your first full config](/configuration/your-first-full-config): the
  capstone: a complete cluster written from an empty file.
- [How-to guides](/how-to): TLS and auth, multiple workers, cloud object
  stores, upgrades, migrating an existing cache.

### I want to run it in production [#i-want-to-run-it-in-production]

- [Production configuration](/operate/production-config): the three-process
  split a real cluster runs in.
- [Deploy with Docker Compose](/operate/deploy-docker-compose) or
  [on bare metal](/operate/deploy-bare-metal): the two shapes, and where state
  lives in each.
- [Observability](/operate/observability): the OTLP to collector to
  Prometheus to Grafana pipeline, and the one detail that silently breaks
  every dashboard.
- [Security hardening](/operate/security-hardening): what's unauthenticated
  by default, and what to do about it.
- [Runbooks](/operate/runbooks) and
  [Troubleshooting](/operate/troubleshooting): the procedures that end an
  incident, and the symptom index that tells you which one you're in.

### I want to understand how it works [#i-want-to-understand-how-it-works]

- [Architecture](/explanations/architecture): scheduler, CAS, action cache,
  workers, and what each is responsible for.
- [Architecture deep dive](/explanations/architecture-deep-dive): the same
  system at the level of crates, startup order, and where an action's path
  forks.
- [The store model](/explanations/store-model),
  [scheduler internals](/explanations/scheduler-internals) and
  [worker execution](/explanations/worker-execution): one page per subsystem,
  written against the source.
- [Correctness and hermeticity](/explanations/correctness-hermeticity): what
  a cache hit guarantees, what it doesn't, and which gaps are yours to close.
- [Local Remote Execution](/explanations/lre): making your toolchain and your
  workers' toolchain the same toolchain, so both share one cache.
- [History](/explanations/history): why NativeLink exists and what it learned
  from what came before.

### I need to look something up [#i-need-to-look-something-up]

- [Configuration reference](/reference/nativelink-config): every field,
  generated from the source.
- [Protocol and API surface](/reference/protocol-api): every gRPC service,
  which RPCs are implemented, and what `Capabilities` advertises.
- [Metrics](/reference/metrics): every instrument, its type, and whether it's
  actually emitted.
- [CLI and environment](/reference/cli-and-env): the complete flag and
  variable surface.
- [Glossary](/reference/glossary) and [Changelog](/reference/changelog).

### I want to contribute [#i-want-to-contribute]

- [Contribution guidelines](/contribute/guidelines): what gets accepted and
  how review works.
- Build from source with [Bazel](/contribute/bazel),
  [Cargo](/contribute/cargo), or [Nix](/contribute/nix).
- [Writing docs](/contribute/docs): the archetypes and conventions this site
  is built on.

## FAQ [#faq]

Every page here ends with an FAQ for its own topic. These are the questions we
get most overall.

<Accordions type="multiple">
  <Accordion title="Is NativeLink free?" id="is-nativelink-free">
    For permitted use, yes. Most of the repository is `FSL-1.1-Apache-2.0`,
    which allows internal use, modification, and redistribution for
    non-competing purposes, and converts to Apache 2.0 two years after each
    version ships. Two modules, metrics and persistent workers, carry a
    Business Source License instead, and using those in shared, production or
    commercial settings needs a separate agreement. Nothing in the binary
    enforces any of that, which is exactly why
    [Open source and Enterprise](/reference/oss-and-enterprise) exists.
  </Accordion>
  <Accordion title="Which build tools work with NativeLink?" id="which-build-tools-work-with-nativelink">
    Any client that speaks the Remote Execution API. The ones with a page here
    are [Bazel](/getting-started/quickstart),
    [Buck2](/getting-started/connect-your-build/buck2),
    [Siso](/getting-started/connect-your-build/siso),
    [Chromium's build](/getting-started/connect-your-build/chromium),
    [Pants](/getting-started/connect-your-build/pants),
    [BuildStream](/getting-started/connect-your-build/buildstream), and
    [CMake via recc](/getting-started/connect-your-build/cmake-recc). None of
    them needs build-file rewrites.
  </Accordion>
  <Accordion title="What platforms does the server run on?" id="what-platforms-does-the-server-run-on">
    Linux and Windows on x86_64 and ARM64 via the multi-arch Docker image
    (`ghcr.io/tracemachina/nativelink`, v1.6.0+), and macOS on Apple Silicon
    and Intel via the Docker image or a from-source Nix build. See
    [Quickstart](/getting-started/quickstart).
  </Accordion>
  <Accordion title="Do I need remote execution to get value from this?" id="do-i-need-remote-execution-to-get-value-from-this">
    No, and most teams shouldn't start there. A shared cache is the smaller
    change and usually the larger win: it needs no toolchain work, no
    hermeticity audit, and no worker fleet. Remote execution is the second
    section for a reason: it's what you add once your cache is hitting and the
    misses are what's slow. [Why NativeLink](/use-cases) works through which of
    the two your build actually needs.
  </Accordion>
</Accordions>

# https://docs.nativelink.com/use-cases

**Why NativeLink**: What NativeLink is, what a remote cache and remote execution actually do, and a 30-second check for whether your build is one that benefits.

**Who this is for:** anyone deciding whether to put a remote cache and
execution layer in front of their build, including people meeting the idea for
the first time. **What you'll have at the end:** a working mental model of what
NativeLink does, and a clear yes or no for your own situation. **Time:** about
ten minutes.

## What NativeLink is [#what-nativelink-is]

NativeLink is a **remote build cache** and a **remote execution** service in one
program. Your build tool asks it "have you already built this?" and, when the
answer is no, "can you build it for me over there?" Nothing about your source
code changes. You point your build tool at a URL, and work that used to happen
on your machine stops happening on your machine.

It's written in Rust, ships as a single binary that plays every role, and
speaks the [Remote Execution API](https://github.com/bazelbuild/remote-apis),
an open protocol, not a NativeLink invention, which is why the build tool you
already use can talk to it without a plugin.

## The problem it exists to solve [#the-problem-it-exists-to-solve]

Every build system computes the same thing twice. Once on your laptop, once in
CI. Once for you, once for the colleague who pulled the same commit. Once
before lunch, once after you reverted the change that broke it. The work is
deterministic, the inputs are identical, and the machine does it again anyway.

That waste has two different shapes, and they need two different fixes.

Some work is *redundant*: somebody, somewhere, already produced exactly this
output from exactly these inputs. The fix is a cache.

Some work is *genuinely new* but badly placed: it has to run, but it doesn't
have to run on the laptop you're typing on, one action at a time. The fix is to
run it somewhere else, in parallel, on hardware that suits it.

NativeLink is both fixes.

## What a remote cache is [#what-a-remote-cache-is]

A build is a graph of actions. Each action has inputs (source files, compiler
flags, the compiler itself) and produces outputs. A **remote cache** stores
those outputs on a server keyed by a hash of all of the inputs.

Before running an action, your build tool hashes the inputs and asks the server
whether it has a result for that hash. A hit means the output is downloaded
instead of computed. A miss means the action runs and the result is uploaded so
that nobody else has to run it.

The key property is that the key *is* the content. This is called
**content-addressed storage**, and it has a consequence worth pausing on:
entries can never be stale. A cache keyed on file paths and timestamps needs
invalidation logic, and invalidation logic is where build caches go wrong. A
cache keyed on the hash of the inputs cannot return the wrong answer for a
question, because a different question has a different key.

<Callout type="info" title="Two caches, one idea">
  NativeLink stores these separately. The **CAS** (content-addressable storage)
  holds the bytes of files. The **action cache** maps "this action, with these
  inputs" to "these output files." Most sentences about "the cache" mean both
  at once.
</Callout>

The direct consequence: the more people and machines share one cache, the more
often any given action has already been run by somebody. A cache used by one
laptop is a modest optimization. A cache shared by a team and its CI is a
different kind of thing.

## What remote execution is [#what-remote-execution-is]

Caching only helps with work that has already been done once. Remote execution
is what happens on the miss.

Instead of running the action locally, your build tool sends the action's
description and its inputs to a **scheduler**, which hands it to a **worker**,
a machine in a pool that does the work and returns the outputs. Because the
pool has many machines, hundreds of actions can be in flight at once, which is
the part your laptop cannot do no matter how new it is.

<Mermaid>
{`flowchart LR
  C["your build tool"] -->|"1 · already built?"| AC["action cache"]
  AC -->|"hit: outputs"| C
  AC -->|"miss"| S["scheduler"]
  S -->|"2 · run this"| W1["worker"]
  S --> W2["worker"]
  S --> W3["worker"]
  W1 -->|"3 · outputs"| CAS["CAS"]
  CAS --> C
  W1 -.->|"inputs"| CAS
  W2 -.-> CAS
  W3 -.-> CAS`}
</Mermaid>

Four roles, CAS, action cache, scheduler and worker, and in NativeLink all four
are the same binary started with different configuration.

Remote execution asks something of you that caching does not: the action has to
be **hermetic**. Everything it needs must be declared as an input, because a
worker in another datacenter does not have your `/usr/local`, your environment
variables, or your home directory. Builds that quietly depend on the machine
they run on work locally and fail remotely, and that is the honest cost of the
feature. [Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
is the page about earning it.

## Why NativeLink specifically [#why-nativelink-specifically]

The protocol is open and there are other implementations. What's particular
here:

**It's one binary.** CAS, action cache, scheduler and worker are the same
executable with different config. No separate control plane needs operating,
and the thing you run locally is the thing you run in production.

**It's Rust, with no garbage collector.** Build caches are latency-sensitive at
the tail: a lookup that usually takes a millisecond and occasionally takes two
hundred is felt by every action in the graph. Predictable memory behaviour is
the reason for the language choice.

**Storage is composable rather than fixed.** A store is declared, named, and
wrapped by other stores: a fast local tier in front of a durable object store, a
compression layer, a shard across several backends. You describe the shape you
want instead of choosing from a menu.
[Storage backends](/how-to/stores) is that vocabulary.

**You can read all of it.** The source is public, and the code that runs your
builds is code you can audit. On licensing, use the precise words rather than
the loose ones: most of the repository is under the Functional Source License
(`FSL-1.1-Apache-2.0`), which is source-available and converts to Apache 2.0
over time, and two modules, metrics and persistent workers, are under the
Business Source License and require an enterprise agreement for production
use.

## Is this for you? [#is-this-for-you]

You will get something out of NativeLink if any of these are true:

- Your build takes long enough that people context-switch while waiting.
- CI rebuilds artifacts that a developer already built minutes earlier.
- Your team shares a codebase, so you share cache hits by construction.
- Some of your actions want hardware your laptop is not: many cores, much
  memory, a GPU, a specific OS.
- You run coding agents against the repo, and their edit-build-test loop is
  bounded by build time rather than by model latency.

You will get less out of it if your build is already seconds long, if every
action is genuinely unique (some ML training and fuzzing workloads are), or if
your build tool doesn't speak the Remote Execution API and you aren't willing to
change that.

Build tools with a real setup path in these docs are Bazel, Buck2, Siso, Pants,
BuildStream and CMake via recc.
[Connect your build](/getting-started/connect-your-build) has the specifics,
including which of them are documented for caching only.

## The three cases worth their own page [#the-three-cases-worth-their-own-page]

Most people arrive with one of three problems. Each of these is the long-form
version of a bullet above; read the one that's yours and skip the others.

**[AI coding agents](/use-cases/ai-coding-agents)**: an agent's loop is
edit, build, test, repeat, and it runs that loop far more often than a person
does. Build time is the loop's wall clock and a large share of its token cost.

**[Autonomous operation](/use-cases/autonomous-operation)**: the case for a
build farm that grows and shrinks with demand instead of being sized for the
worst Tuesday of the quarter, and an honest account of how much of that
NativeLink hands you versus how much you assemble.

**[Silicon and large builds](/use-cases/silicon-and-large-builds)**: EDA
flows, big C++ and LLVM trees, and ML graphs: thousands of actions, hardware
that differs from action to action, and a long tail that decides the wall
clock. This is the workload remote execution was invented for.

## Common questions [#common-questions]

<Accordions>

<Accordion title="Is this the same thing as ccache or sccache?" id="is-this-the-same-thing-as-ccache-or-sccache">

Same instinct, different scope. `ccache` caches compiler invocations on one
machine, and `sccache` extends that to a shared bucket. NativeLink caches every
action in the build graph rather than just compiles, and it can also *run* the
actions that miss. If a shared compile cache is all you need, you may not need
this.

</Accordion>

<Accordion title="Do I have to adopt remote execution to get value?" id="do-i-have-to-adopt-remote-execution-to-get-value">

No, and you shouldn't start there. Caching is the whole of Getting started and it works
without hermetic toolchains, without a worker pool, and without changing how
your actions are written. Remote execution is the next section and it is a bigger
commitment. Most of the win, for most teams, is in the cache.

</Accordion>

<Accordion title="Does my build have to be Bazel?" id="does-my-build-have-to-be-bazel">

No. Anything speaking the Remote Execution API can use NativeLink, and the
docs cover Buck2, Siso, Pants, BuildStream and CMake via recc alongside Bazel.
What matters is whether your build tool can describe an action's inputs
precisely enough to hash them.

</Accordion>

<Accordion title="What does it cost to run?" id="what-does-it-cost-to-run">

The software is free to run and the bill is infrastructure: a machine for the
cache, storage for the blobs, and worker machines if you adopt remote
execution. The trade to weigh is that storage and machines are cheaper than
engineer-hours spent waiting, which is a real argument but not an automatic
one: it depends on how much waiting there actually is.

</Accordion>

<Accordion title="Is it open source?" id="is-it-open-source">

"Source-available" is the accurate phrase. Most of the repository is under the
Functional Source License, which lets you read, modify and use the code with a
restriction on offering it as a competing service, and converts to Apache 2.0
on a schedule. Two modules, metrics and persistent workers, are
Business Source Licensed and need an enterprise agreement for production use.
Everything in these docs is runnable without one unless the page says otherwise.

</Accordion>

</Accordions>

<NextStep href="/getting-started" title="Getting started">
  The fastest way to find out is to run one. Getting started gets a cache serving hits
  to your build, and it's the foundation everything else here stands on.
</NextStep>

<NextStep href="/explanations/architecture" kind="aside" title="How it actually works">
  Prefer to understand the machine before you run it? The architecture page
  covers the four roles and how a request moves through them.
</NextStep>

# https://docs.nativelink.com/use-cases/ai-coding-agents

**AI coding agents**: An agent's edit-build-test loop runs far more often than a human's, so build time sets both its wall clock and a large part of its token bill, and a shared cache turns most of those iterations into downloads.

**Who this is for:** anyone running coding agents against a codebase big enough
that building it is not instant. **What you'll have at the end:** a clear model
of why build infrastructure is agent infrastructure, and what specifically to
put in place. **Time:** about ten minutes.

## The loop is the unit of work [#the-loop-is-the-unit-of-work]

A coding agent does not write a patch and stop. It edits, builds, reads the
error, edits again, builds again, and repeats until the tests pass. That loop is
the agent's entire mechanism for finding out whether it was right.

A human runs that loop a few times an hour and spends the gaps thinking. An
agent runs it as fast as the tooling allows, in a tight cycle, and frequently
runs several in parallel across candidate approaches. The economics invert: for
a person, build time is an interruption; for an agent, build time *is* the task
duration.

The arithmetic is unforgiving. A four-minute build inside a thirty-iteration
session is two hours of wall clock in which nothing is being decided. Cut the
build to forty seconds and the same session is twenty minutes.

## Why agents hit caches more than people do [#why-agents-hit-caches-more-than-people-do]

Here is the part that is rarely noticed: agents are unusually good cache clients,
and the reason is that most of what they do is repetitive in exactly the way
content-addressing rewards.

**They revisit states they've already been in.** An agent that tries an
approach, fails a test, and reverts has returned to a tree it already built. To
a content-addressed cache that is not a similar state, it is *the same* state:
same input hashes, same keys, every action a hit. The build after a revert can
be nearly free.

**They change one thing at a time.** A well-behaved agent makes a small,
targeted edit. Small edits invalidate a small part of the action graph, and
everything downstream of the untouched parts is still keyed the same. The
rebuild is proportional to the edit, not to the repository.

**They run in parallel over shared ancestry.** Three agents exploring three
approaches to the same bug share almost all of their build graph. The first to
build the common ancestry pays for it; the other two get hits. Same for an agent
and the human reviewing its work.

**They repeat each other's work across sessions.** A fresh session on the same
commit rebuilds what yesterday's session already built. With a shared cache that
is a download.

The general form: **cache hit rate rises with the number of participants who
share ancestry, and agents generate a great deal of shared ancestry.** A cache
that is a modest optimization for one developer becomes the dominant factor when
a fleet of agents is hammering the same tree.

## Determinism is the other half [#determinism-is-the-other-half]

Content-addressing gives agents something beyond speed, and it may matter more.

An agent debugging a failure needs the failure to be real. If a test fails
because of a stale artifact, a leaked environment variable, or a machine that
differs from the one that built the dependency, the agent will chase a
phantom, and it will chase it with great energy, writing plausible fixes for a
problem that does not exist. Nondeterminism doesn't just slow an agent down; it
sends it confidently in the wrong direction and burns tokens doing so.

Hermetic, content-addressed builds make "same inputs, same outputs" a property
of the system rather than a hope. When an agent reruns a build and the result
changes, that change is now information.

<Callout type="warn" title="This cuts both ways">
  Remote execution requires hermetic actions, and a build that quietly depends
  on the machine it runs on will fail remotely while working locally. That's
  real work, and it's worth doing regardless: a build an agent can trust is a
  build a person can too. See
  [Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity).
</Callout>

## The token argument [#the-token-argument]

Wall clock is the visible cost. The token bill is the hidden one, and for
teams running agents at scale it is frequently the larger.

Every iteration of the loop spends tokens: the agent reads build output,
reasons about it, and produces the next edit. Slow builds don't just make the
loop longer, they make it *wider*: a long feedback delay pushes an agent toward
batching more speculative changes per iteration, which produces bigger diffs,
noisier failures, and more context to re-read when something breaks.

Fast, trustworthy builds let an agent work in small verified increments. Small
increments mean less context re-read per iteration and fewer wasted iterations
chasing failures that were never real.

## What to actually put in place [#what-to-actually-put-in-place]

In rough order of return:

**A shared cache both agents and humans use.** Not one per agent, not one for CI
and another for developers. The value is in the sharing, and splitting it is the
most common way teams leave the win on the table.
[Shared cache](/getting-started/shared-cache) is that deployment.

**Agents pointed at it from the start.** An agent that builds locally with no
remote cache is generating exactly the redundant work this is designed to
absorb. This is one flag in the build configuration the agent already uses.

**Hermeticity treated as a bug class.** Every action that depends on unnamed
state is a source of failures your agent cannot debug and will misdiagnose.

**Remote execution once caching is steady.** Remote execution adds parallelism across a
worker pool for the actions that genuinely have to run. It matters most when
agents work in parallel, because that's when local cores become the ceiling.

**Enough cache retention to cover how agents actually work.** Eviction policy is
usually sized for a linear history of commits. Agents branch, revert and
revisit far more than that, and a cache that has already evicted the tree an
agent is returning to gives back the biggest single win on this page. Size for
the breadth of states in play, not just the depth of history.

## Common questions [#common-questions]

<Accordions>

<Accordion title="Should each agent get its own cache to avoid interference?" id="should-each-agent-get-its-own-cache-to-avoid-interference">

No: that removes the main benefit. Cache entries are keyed by the hash of their
inputs, so two agents cannot corrupt each other's results; the worst case is
that they store separate entries for genuinely different work. Isolation you
might want for other reasons is a network and access-control question, not a
cache-partitioning one.

</Accordion>

<Accordion title="Can a bad agent poison the cache for everyone?" id="can-a-bad-agent-poison-the-cache-for-everyone">

Not through ordinary use. An entry is only reachable by the hash of the inputs
that produced it, so a bad edit produces a differently-keyed entry that nobody
else asks for. What *can* cause a bad shared entry is a non-hermetic action
whose output depends on something outside its declared inputs, which is the
same hazard for humans, and another reason hermeticity is the foundation. Write
access to the cache is a trust boundary; see
[TLS and authentication](/how-to/tls-and-auth).

</Accordion>

<Accordion title="Does this help with test flakiness?" id="does-this-help-with-test-flakiness">

Indirectly, and importantly. It doesn't fix a genuinely flaky test, but it
removes the large class of failures caused by stale or inconsistent build state,
so the flakes that remain are real ones. For an agent, that distinction is the
difference between a debuggable signal and noise.

</Accordion>

<Accordion title="Our agents work on many small repos, not one monorepo." id="our-agents-work-on-many-small-repos-not-one-monorepo">

The benefit tracks shared ancestry, so it's weaker across unrelated repos and
still real within each one plus their common dependencies. If the repos share a
toolchain or a base image, those actions are shared cache entries even when the
source trees aren't.

</Accordion>

<Accordion title="How do I point my agent at NativeLink?" id="how-do-i-point-my-agent-at-nativelink">

You don't configure the agent, you configure the build tool the agent invokes,
and the agent picks it up because it runs the same commands you do. A remote
cache flag in the checked-in build configuration is inherited by every agent and
every human automatically. [Quickstart](/getting-started/quickstart) sets that
up.

</Accordion>

</Accordions>

<NextStep href="/getting-started/shared-cache" title="Shared cache">
  The deployment this page argues for: one cache, shared by everyone and
  everything that builds the tree.
</NextStep>

<NextStep href="/agents" kind="aside" title="For agents">
  The other direction: making these docs and this codebase legible to an agent
  that's reading them.
</NextStep>

# https://docs.nativelink.com/use-cases/autonomous-operation

**Autonomous operation**: What it takes for a build farm to grow and shrink with demand without anyone watching it: which properties NativeLink gives you, which pieces you assemble yourself, and where the seams are.

**Who this is for:** whoever would otherwise be paged about the build farm.
**What you'll have at the end:** an accurate picture of how close NativeLink
gets you to a hands-off farm, and exactly which parts you build. **Time:**
about fifteen minutes.

## The shape of the problem [#the-shape-of-the-problem]

Build load is spiky in a way that punishes static capacity. It is near zero at
night, ragged through the morning, and vicious in the hour before a release. A
farm sized for the peak is idle most of the time and expensive all of the time.
A farm sized for the average has a queue during every moment anybody cares
about.

The way out is a farm that changes size on its own. That is not a NativeLink
feature you switch on. It's a property that emerges when the components have
the right characteristics, and most of this page is about which characteristics
those are and which ones you have to supply.

## Why workers can be treated as disposable [#why-workers-can-be-treated-as-disposable]

Autoscaling only works if adding and removing a machine is uneventful. Three
properties of the worker make it so.

**Workers hold no authoritative state.** A worker executes actions and writes
results to the CAS. Everything durable lives in the store, not on the worker, so
a worker that vanishes takes nothing with it that isn't reproducible.

**Workers announce themselves.** A worker dials the scheduler and registers,
rather than the scheduler being told a list of workers. Starting five more is
enough; there is no membership to update. That's why
[running multiple workers](/how-to/run-multiple-workers) is mostly a matter of
starting more processes.

**Lost work is retried, not lost.** When a worker disappears mid-action, the
scheduler notices, increments an attempt counter, and requeues the action,
with its worker affinity cleared, so it goes to whichever worker is free rather
than waiting for a machine that is gone.

<SourceLink file="nativelink-scheduler/src/simple_scheduler_state_manager.rs" symbol="update_operation" />

That last property is what makes preemptible and spot instances viable, and it
is the single largest cost lever available. The retry budget is
`max_job_retries` on the scheduler, defaulting to **3**.

<Callout type="warn" title="`max_job_retries: 0` does not mean zero retries">
  A configured `0` is read as "unset" and falls back to the default of 3. There
  is no way to express "never retry" through this field.
</Callout>

A worker is declared dead when it stops keepaliving for `worker_timeout_s`,
which defaults to **5 seconds** and is checked once a second. That is a
deliberately short leash: fast detection means fast requeue, and the cost of a
false positive is one repeated action rather than a stall.

## Draining, for the shutdown you know about [#draining-for-the-shutdown-you-know-about]

Retries handle the machines that die without warning. Machines that give you
notice deserve better, and there's a specific hook for it.

The admin interface exposes a drain endpoint:

```
POST {admin_path}/scheduler/{instance_name}/set_drain_worker/{worker_id}/1
```

A draining worker accepts no new actions and finishes the ones it is running.
That is exactly the right response to a spot-termination notice or a rolling
node upgrade: call it when the notice arrives, and the in-flight work completes
instead of being thrown away and redone.

<SourceLink file="nativelink-scheduler/src/worker.rs" symbol="Worker" />

Nothing in the repository wires this to any cloud provider's interruption
signal. The handler is a short script you write, and it is the highest-value
piece of glue on this page.

## What the scheduler remembers, and what it forgets [#what-the-scheduler-remembers-and-what-it-forgets]

This is the constraint that shapes a hands-off design, so it's worth being
precise.

By default the scheduler keeps its queue **in process memory**. Restarting it
loses the in-flight queue. Clients retry and the work gets done, so this is not
data loss in any lasting sense, but it does mean the scheduler is the one
component you cannot casually cycle under load, and that a single scheduler is a
single point of interruption.

A Redis-backed alternative exists, and the configuration names it honestly:
the field is `experimental_backend`. It moves queue state out of the process so
schedulers stop being special. It also carries the rough edges the name implies:
there's a `fallback_match_interval_s` in the config specifically to paper over
eventually-consistent searches.

<Callout type="warn" title="The Redis backend does not emit the queue metric">
  `execution.active.count`, the signal an autoscaler needs, is emitted only by
  the in-memory backend. The Redis path emits none of it, so the topology that
  makes schedulers replaceable is also the topology where the scaling signal
  disappears. Know this before you plan around it.
</Callout>

## The signal you scale on [#the-signal-you-scale-on]

An autoscaler needs a number that says "more work is waiting than there is
capacity for." That number is queue depth: actions sitting in the `queued`
stage rather than `executing`.

NativeLink emits it as `execution.active.count`, an up-down counter attributed
by `execution.stage`.

<SourceLink file="nativelink-util/src/metrics.rs" symbol="EXECUTION_METRICS" />

Three things about that metric decide your architecture, and all three surprise
people:

**Telemetry is OTLP push, not Prometheus pull.** The binary has no `/metrics`
endpoint to scrape. It pushes OpenTelemetry to a collector, and the collector is
what a Prometheus-shaped world reads from.
`deployment-examples/metrics/` is the working example of that pipeline.

**Queue depth is a derived series, not a metric name.** What you scale on is a
recording rule that sums the counter filtered to the `queued` stage. The
example rules in `deployment-examples/metrics/prometheus-recording-rules.yml`
define `nativelink:queue_depth`, `nativelink:worker_utilization` and
`nativelink:actions_per_worker`. An HPA reads the recording rule through an
external-metrics adapter; there is no metric the binary emits that an HPA can
target directly.

**The metrics module is Business Source Licensed.** `nativelink-util/src/metrics.rs`
carries a header requiring an enterprise license agreement, unlike the
Functional Source License covering most of the repository. Production
observability is exactly the case that header describes, so settle the licensing
question before you design around it.

<Callout type="info" title="Workers emit no metrics of their own">
  Every utilization number is inferred scheduler-side from what it knows about
  its workers. No worker-side telemetry exists to scrape, so host-level CPU
  and memory from your existing node monitoring is the complement to the
  scheduler's view, not a duplicate of it.
</Callout>

## What the repository ships, and what it doesn't [#what-the-repository-ships-and-what-it-doesnt]

Being direct, because this is the part most likely to be assumed:

**The repository contains no autoscaling manifests for NativeLink.** The Helm
charts for Kubernetes ship with [NativeLink Enterprise](https://enterprise.nativelink.com),
not with this repository. The Kustomize building blocks under `kubernetes/`
set fixed replica counts. No HorizontalPodAutoscaler for workers, no KEDA `ScaledObject`,
no cluster-autoscaler or Karpenter configuration, and no node pools or
tolerations for build workers. The one HPA in the tree scales the OpenTelemetry
collector, not NativeLink.

What the repository does give you is the harder half: a worker that is safe to
kill, a scheduler that requeues what it loses, a drain hook for planned
shutdown, a queue-depth signal, and a working metrics pipeline to carry it.

What you assemble is the standard Kubernetes half: an HPA on the derived queue
metric via an external-metrics adapter, a cluster autoscaler or Karpenter for
the nodes underneath, spot node pools with the drain hook wired to interruption
notices, and node affinity for heterogeneous pools. These are not NativeLink
concepts, which is the point: nothing here needs a bespoke controller.

<Callout type="info" title="Where the manifests are">
  Kubernetes deployments, including the Helm charts, are part of
  [NativeLink Enterprise](https://enterprise.nativelink.com). This repository ships
  only the Kustomize building blocks under `kubernetes/` that its own examples
  use, and no KEDA adapter.
</Callout>

## What "no human in the loop" actually means [#what-no-human-in-the-loop-actually-means]

A realistic end state, in the order the pieces earn their keep:

Workers scale on queue depth and fall to a small floor overnight. Most of the
fleet is preemptible, because retries make interruption a cost rather than an
incident, and the drain hook converts warned shutdowns into clean ones. Node
capacity follows pod demand through your existing autoscaler. The CAS scales
separately and more conservatively, because it holds the state everything else
depends on. Alerts fire on the things automation cannot resolve (sustained
queue depth that scaling isn't clearing, error rates, storage approaching its
limit) rather than on capacity, which now handles itself.

The queue-depth alert is the one to get right. It's the signal that your
autoscaler has hit a ceiling it cannot pass, whether that's a quota, a node
pool's maximum, or an exhausted spot market. Everything else is noise by
comparison.

## Common questions [#common-questions]

<Accordions>

<Accordion title="Can I autoscale the CAS the same way?" id="can-i-autoscale-the-cas-the-same-way">

More carefully. Workers are stateless and cheap to churn; the CAS holds the data
everything else is asking for. Stateless CAS replicas in front of one shared
backing store scale on request rate, but a local filesystem tier is warm state
that a new replica does not have, so scaling in throws away cache locality.
Scale it slowly, and prefer bigger replicas over more of them.
[Tuning](/operate/tuning) covers the storage path.

</Accordion>

<Accordion title="What happens to an action when its worker is preempted?" id="what-happens-to-an-action-when-its-worker-is-preempted">

The scheduler notices the worker stop keepaliving within `worker_timeout_s`,
increments the action's attempt count, clears its worker affinity, and requeues
it. Any worker can pick it up. Once the attempt count passes `max_job_retries`
the action completes with an error saying it attempted to execute too many
times, so a systematically failing action does not requeue forever.

</Accordion>

<Accordion title="Is one scheduler enough?" id="is-one-scheduler-enough">

For most deployments, yes: the scheduler does bookkeeping, not the expensive
work, and it fails over by restarting. The reasons to want more are availability
across zones and not wanting a restart to drop the in-flight queue. Both point
at the Redis backend, whose name says `experimental` and which stops emitting the
queue metric. That trade is the current state of the art here.

</Accordion>

<Accordion title="How aggressively should workers scale in?" id="how-aggressively-should-workers-scale-in">

Less aggressively than they scale out. Scaling out has a bounded cost, a
machine you briefly didn't need. Scaling in during a lull that ends thirty
seconds later costs a cold start on every action that arrives next. Generous
scale-up, patient scale-down.

</Accordion>

<Accordion title="Do I need Prometheus, or can I use my existing vendor?" id="do-i-need-prometheus-or-can-i-use-my-existing-vendor">

Anything that ingests OTLP works, and the collector can fan out to several
destinations at once. Prometheus appears throughout the examples because the
recording rules and the HPA external-metrics adapter are expressed in it, but the
requirement is OTLP ingestion plus something an autoscaler can query, not
Prometheus specifically.

</Accordion>

</Accordions>

<NextStep href="/operate/tuning" title="Tuning">
  The levers underneath all of this: worker sizing, the storage path, and what
  to change when the queue is deep.
</NextStep>

<NextStep href="/operate/observability" kind="aside" title="Observability">
  Getting the telemetry out of the binary and into something that can act on
  it, which is the prerequisite for every scaling decision here.
</NextStep>

# https://docs.nativelink.com/use-cases/silicon-and-large-builds

**Silicon and large builds**: EDA flows, large C++ and LLVM trees, and ML graphs are the workloads remote execution was invented for: thousands of actions, hardware that differs from action to action, and a long tail that decides the wall clock.

**Who this is for:** anyone whose build is measured in thousands of actions and
tens of minutes, on hardware that isn't uniform. **What you'll have at the end:**
an understanding of why these builds are the hard case, and how NativeLink routes
actions to the machines that suit them. **Time:** about fifteen minutes.

## What makes a build "large" in the way that matters [#what-makes-a-build-large-in-the-way-that-matters]

Size in gigabytes is not the interesting axis. Three properties are, and a build
that has all three is the shape remote execution was designed around.

**Action count.** An EDA flow (synthesis, place-and-route, timing, DRC, LVS)
or a full LLVM tree is tens of thousands of actions. At that count the graph is
wide enough that the number of cores you can throw at it is the only thing
setting the wall clock. A laptop has sixteen. A worker pool has however many you
ask for.

**Heterogeneity.** The actions are not interchangeable. A link step wants a
machine with a great deal of memory and will use one core doing it. A thousand
compiles want a thousand cores and very little memory each. A simulation wants a
GPU. A vendor tool wants a specific OS, sometimes a specific kernel, and a
license seat. Sizing one machine for all of that means sizing it for the worst
case and wasting it on every other action.

**The long tail.** Large graphs have a critical path, and near the end of the
build most of the fleet is idle while a handful of long actions finish. The
total is set by the longest chain, not by the sum of the work. That's why "add
more cores" stops helping at a point, and why *which* machine runs the long
actions matters more than how many machines you have.

## Why caching alone runs out [#why-caching-alone-runs-out]

On a small build, the cache is most of the story. On a build like this it is
still enormously valuable (it is still the thing you deploy first, and Getting
started is still where to start), but it stops being sufficient, for a specific reason.

A cache helps with work somebody has already done. A large build has a long
list of actions that nobody has done, because a real change in a low-level
header invalidates a large subtree, and because verification runs are new by
construction. Those actions have to execute. The question stops being "can we
avoid this work" and becomes "where does this work run, and how much of it runs
at once."

That is the transition from Getting started to Remote execution, and this class of build is the
clearest case for making it.

## Routing actions to the hardware that fits [#routing-actions-to-the-hardware-that-fits]

The mechanism is **platform properties**: key-value pairs an action declares as
requirements, matched against properties a worker declares about itself. The
scheduler will not assign an action to a worker that does not satisfy them.

A worker declares its properties in its configuration, either as literal values
or as a command run at startup whose output becomes the value, which is how a
worker discovers, say, its own core count rather than having it hard-coded per
host.

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="PropertyType" />

The scheduler declares, separately, which property keys exist and how each one
is matched. Four kinds exist and the differences are load-bearing:

**`minimum`** compares numerically: the worker's value must be greater than or
equal to the action's. It is also **consumable**: when an action requesting
`cpu_count: 8` is assigned, eight is subtracted from that worker's available
count and restored when the action completes. This is how a 64-core worker takes
eight 8-core actions and not sixty-four of them.

**`exact`** requires string equality. This is the one for OS, ISA, container
image, or a GPU model, anything where "more" is meaningless.

**`priority`** does not restrict anything by value. It requires only that the
key be present. The behaviour its name suggests, preferring some workers over
others, is an unimplemented TODO in the scheduler.

**`ignore`** matches everything, and is the way to accept a property that
clients send without letting it affect placement.

<Callout type="warn" title="`priority` is not a constraint">
  A key configured as `priority` will match a worker regardless of the value on
  either side. If you use it for something that must actually be satisfied, the
  action will happily land on a worker that cannot satisfy it. Use `minimum` for
  quantities and `exact` for identities; reach for `priority` only when you mean
  "record this, don't route on it."
</Callout>

An unknown property key is refused rather than silently matched: the scheduler
logs `Unknown platform property` and the action stays queued, so a typo in a
requirement stalls visibly instead of quietly sending a GPU action to a CPU
worker.

<SourceLink file="nativelink-scheduler/src/platform_property_manager.rs" symbol="make_prop_value" />

[Platform properties](/remote-execution/platform-properties) is the reference
for the mechanism.

## Special hardware, and what NativeLink actually knows about it [#special-hardware-and-what-nativelink-actually-knows-about-it]

NativeLink has no GPU support, and that phrasing is deliberate rather
than a shortcoming. No GPU-aware code exists anywhere in the codebase: no
CUDA integration, no device discovery, no special-cased scheduling. GPUs are
routed the same way a specific kernel version or a licensed tool is: as plain
platform properties that you name and the scheduler matches.

The convention across the shipped example configurations is a pair:

```json5
"supported_platform_properties": {
  "cpu_count": "minimum",
  "memory_kb": "minimum",
  "gpu_count": "minimum",
  "gpu_model": "exact",
}
```

`gpu_count` as `minimum` so it is both a threshold and consumable: a worker
with four GPUs can hold four single-GPU actions. `gpu_model` as `exact` so an
action needing a particular device does not land on a different one.

<Callout type="warn" title="One shipped example gets this wrong">
  The Kubernetes example config in `kubernetes/nativelink/nativelink-config.json5`
  declares both `gpu_count` and `gpu_model` as `priority`, which, per the
  section above, enforces nothing. An action requesting four GPUs will schedule
  onto a worker with none. If you started from that file, change both.
</Callout>

The same pattern covers everything else that varies. An `os` and an `arch` key
as `exact` give you cross-platform pools from one scheduler. A `license_seats`
key as `minimum` makes a vendor tool's seat count a scheduling constraint rather
than a race, because `minimum` is consumable: the seats are held for the duration
of the action and released when it finishes. That last one is a genuinely useful
trick for EDA flows and is not apparent from the config reference.

## Running a heterogeneous fleet [#running-a-heterogeneous-fleet]

The pool is not one shape of machine. It's several, each a worker process with
its own configuration, all registering with the same scheduler.

<Mermaid>
{`flowchart TB
  S["scheduler<br/>declares which keys exist<br/>and how each is matched"]
  S -->|"cpu_count ≥ 4"| C["compile pool<br/>many small workers"]
  S -->|"memory_kb ≥ 64000000"| L["link pool<br/>few large-memory workers"]
  S -->|"gpu_count ≥ 1<br/>gpu_model = a100"| G["GPU pool"]
  S -->|"os = windows"| O["platform pool"]`}
</Mermaid>

Nothing about this requires more than one scheduler, or a separate deployment
per pool. Each worker announces what it is; the scheduler routes on that.
[Run multiple workers](/how-to/run-multiple-workers) is the mechanics, and
[tuning](/operate/tuning) covers sizing each pool once you're running them.

The practical advice is to start with one pool and split it only when you can
name the reason. Two pools with different properties is a real operational cost,
and it pays for itself when a specific class of action is either starved of the
hardware it needs or hogging hardware it doesn't.

## Toolchains are the hard part, not the scheduling [#toolchains-are-the-hard-part-not-the-scheduling]

For this class of build the routing is the straightforward half. The half that takes the
time is making the actions hermetic enough to run somewhere else at all.

Large old codebases accumulate assumptions: a compiler at an absolute path, a
tool installed by the platform team years ago, an environment variable set in a
shell profile, a vendor toolchain with a license daemon. All of that works
locally and none of it exists on a worker. Getting a large build onto remote
execution is mostly the work of finding those assumptions and declaring them.

NativeLink's answer is **Local Remote Execution**: building the toolchain
itself as a declared, reproducible artifact so that the environment a worker
runs in is the same one you get locally, by construction rather than by
convention. [Local Remote Execution](/explanations/lre) is the explanation, and
[Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity) is
the practical path.

This is also the reason to be honest about sequencing. Getting started (caching) costs
you a URL in a config file. Remote execution costs you a toolchain project. The payoff on
a build of this size justifies it, but it is a project, and planning it as a
config change is how these migrations stall.

## What a build like this looks like when it works [#what-a-build-like-this-looks-like-when-it-works]

One published account is worth reading rather than paraphrasing:
[Reid Kleckner's write-up](https://reidkleckner.dev/posts/llvm-recc-nativelink/)
describes building LLVM with `recc` executing remotely against NativeLink and
taking a build from roughly seventeen minutes to roughly four. It is one
person's measurement on their own hardware, with caveats the post itself spells
out (precompiled headers disabled, a mixed home fleet, a wireless link), so
treat it as an illustration of the shape rather than a number to plan against,
but the shape is right. A wide graph of independent compiles is exactly the case
where adding remote cores collapses the wall clock.

The honest general statement is that speedups on this class of build vary by an
order of magnitude between workloads, because they depend on how wide the graph
is, how long the critical path is, and how much of the build was already
cacheable. The way to find out is to measure your own.

[CMake with recc](/getting-started/connect-your-build/cmake-recc) is the path
for CMake-based C++ trees, which is what most large non-Bazel C++ projects are.

## Common questions [#common-questions]

<Accordions>

<Accordion title="My build is CMake or Make, not Bazel. Is any of this available?" id="my-build-is-cmake-or-make-not-bazel-is-any-of-this-available">

Partly. `recc` puts a CMake-based build on the remote cache by intercepting
compiler invocations, and it's the documented path for large C++ trees. The
setup in these docs is cache-only (`RECC_CACHE_ONLY=1`, so misses compile
locally); recc can also forward compiles to remote workers once you have them,
one compiler invocation at a time. Remote execution of the whole graph needs a
build tool that describes every action's complete input set, which is what
Bazel, Buck2, Siso and BuildStream do and what Make does not.

</Accordion>

<Accordion title="How do I stop huge link actions from crowding out compiles?" id="how-do-i-stop-huge-link-actions-from-crowding-out-compiles">

Give them different properties and let `minimum` do the accounting. A link
action declaring a large `memory_kb` only matches workers that have it, and
because `minimum` is consumable, one such action occupies that worker's memory
budget until it finishes rather than letting three more land on top of it. If
they are still colliding, that's the signal to split the pool.

</Accordion>

<Accordion title="Can workers run different operating systems?" id="can-workers-run-different-operating-systems">

Yes: a worker is a process, and pools are distinguished only by the properties
they declare. Declare `os` as `exact` on the scheduler, have each worker declare
its own, and one scheduler routes to all of them. The constraint is not
NativeLink's; it's that the action's toolchain has to exist for the target
platform.

</Accordion>

<Accordion title="Does a vendor tool's license server work with this?" id="does-a-vendor-tools-license-server-work-with-this">

The tool still talks to the license server as it always did, so the worker
needs network access to it. What changes is that seat exhaustion becomes a
scheduling problem you can express: a `license_seats` key matched as `minimum`
holds seats for the duration of an action and releases them after, which turns
a race into a queue.

</Accordion>

<Accordion title="Is the cache still worth it if most of my actions are unique?" id="is-the-cache-still-worth-it-if-most-of-my-actions-are-unique">

Usually yes, because "most actions are unique" is rarely true across a whole
tree: the third-party dependencies, the toolchain, and everything below the
change you made are shared. It stops being true for workloads that are
genuinely new every run, and some ML training and fuzzing jobs are exactly that.
For those, remote execution is the part that helps and the cache is close to
free anyway.

</Accordion>

</Accordions>

<NextStep href="/remote-execution/platform-properties" title="Platform properties">
  The reference for the routing mechanism: how to declare properties on workers,
  how to match them on the scheduler, and how each matching kind behaves.
</NextStep>

<NextStep href="/explanations/lre" kind="aside" title="Local Remote Execution">
  The toolchain half of the problem: making the environment a worker builds in
  reproducible rather than assumed.
</NextStep>

# https://docs.nativelink.com/getting-started

**Getting started**: The first section of the reading path: a cache running, connected to your build, proven to be hitting.

**Who this is for:** anyone who has decided to try NativeLink and wants the
shortest path to a real result. **What you'll have at the end of this
section:** a cache your build system is talking to, hits you can point at,
and, if you want it, the same cache shared with your team. **Time:** ten
minutes for the first win, an afternoon for the shared one.

<Prerequisites>
  Nothing beyond a build system that speaks the Remote Execution API.
</Prerequisites>

## Why caching first [#why-caching-first]

NativeLink does two things: it caches build outputs, and it executes build
actions remotely. It is tempting to reach for the second one immediately,
because it's the more impressive half. Do the first one first anyway.

Caching is the half with the short feedback loop. You start it, you point
your build at it, you build twice, and the second build is faster. The whole
loop is minutes and nothing about it can fail mysteriously. Remote execution
adds a scheduler, workers, a platform-matching contract, and a toolchain
hermeticity requirement, and every one of those is easier to debug when the
storage layer underneath is already known-good.

This section is the first of four on the reading path, and each section
assumes the one before it is standing:

| Section | You end with |
|---|---|
| **Getting started** (you are here) | a cache serving hits to your build |
| [Remote execution](/remote-execution) | workers running the actions that miss the cache |
| [Configuration](/configuration) | a config file you wrote and understand |
| [How-to guides](/how-to) | the backend, transport, and upgrade path you want |

## Start here, or skip ahead [#start-here-or-skip-ahead]

**Start here** if NativeLink isn't running yet. Take
[Quickstart](/getting-started/quickstart) in order; it's a guaranteed happy path and
every step tells you what output to expect.

**Skip ahead** if you already have a cache running and hitting. The next section picks
you up at [Remote execution](/remote-execution) and assumes exactly this
section's outcome and nothing more.

**Skip sideways** if your build tool isn't Bazel. The
[connect your build tool](/getting-started/connect-your-build) pages cover
Buck2, Siso, Pants, BuildStream and CMake via recc: same cache, different
flags.

## The order to read them in [#the-order-to-read-them-in]

1. **[Quickstart](/getting-started/quickstart)**: install NativeLink and get a cache
   answering on localhost.
2. **[Verify your cache](/getting-started/verify-your-cache)**: build twice and
   read the count of remote cache hits.
3. **[Connect your build tool](/getting-started/connect-your-build)**:
   the flags for Buck2, Siso, Pants, BuildStream and recc.
4. **[Shared cache](/getting-started/shared-cache)**: move it off localhost so
   the rest of the team gets your hits.

<NextStep href="/getting-started/quickstart" title="Quickstart">
  Install NativeLink and run your first cached build. Ten minutes, one
  terminal, no cluster required.
</NextStep>

# https://docs.nativelink.com/getting-started/quickstart

**Quickstart**: Install NativeLink and point your build system at it in under 10 minutes.

<Prerequisites>
  Nothing beyond a build system that speaks the Remote Execution API.
</Prerequisites>

This guide takes you from zero to your first cached build. Pick the
installation method that matches your platform, then wire your build
system to the running cluster.

<Callout type="info" title="Prerequisites">
  You need a build tool that speaks the Remote Execution API: Bazel,
  Buck2, Siso, Pants, Goma, or CMake via recc. For the Docker path,
  Docker. For the Nix path, a recent Nix install with flakes enabled.
</Callout>

## Install NativeLink [#install-nativelink]

The fastest way is the prebuilt container image. It runs anywhere Docker
runs; the configuration it needs is a single JSON5 file from the
repository.

<Tabs items={["Linux (Docker)", "macOS / Linux (Nix)", "Windows (Docker)"]}>

  <Tab value="Linux (Docker)">

```bash
# Grab a known-good basic configuration
curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/v1.6.5/nativelink-config/examples/basic_cas.json5

# Run the official image
docker run \
  -v $(pwd)/basic_cas.json5:/config \
  -p 50051:50051 \
  ghcr.io/tracemachina/nativelink:v1.6.5 config
```

The image is multi-arch: x86_64 and ARM64 both run natively (use
v1.6.0 or later; earlier tags are x86_64-only). The server is now
listening on `localhost:50051`.

  </Tab>

  <Tab value="macOS / Linux (Nix)">

```bash
curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/refs/tags/v1.6.5/nativelink-config/examples/basic_cas.json5

nix run github:TraceMachina/nativelink/v1.6.5 ./basic_cas.json5
```

Slower than the prebuilt image because it builds from source, but works on
macOS (Apple Silicon and Intel) and any Linux with Nix. Use it when you
want a native binary instead of a container; the Docker path above also
runs natively on Apple Silicon.

<Callout type="warn">
  - **Prerequisites**: Make sure your Nix installation has experimental features enabled (add `experimental-features = nix-command flakes` to your `~/.config/nix/nix.conf`).
  - **Disk Space**: The installation and build process requires downloading and compiling toolchains and compilers. Ensure you have at least **15 to 20 GB of free space** on your drive.
  - **Portability**: Executables built for macOS link dynamically against libraries from the
  Nix store. They will not run on systems without those libraries
  available. Stick to the Docker path if you need to copy the binary to
  another machine.
</Callout>

  </Tab>

  <Tab value="Windows (Docker)">

```powershell
Invoke-WebRequest `
  -Uri "https://raw.githubusercontent.com/TraceMachina/nativelink/v1.6.5/nativelink-config/examples/basic_cas.json5" `
  -OutFile "basic_cas.json5"

docker run `
  -v ${PWD}/basic_cas.json5:/config `
  -p 50051:50051 `
  ghcr.io/tracemachina/nativelink:v1.6.5 config
```

The image is a Linux image published for x86_64 and ARM64, so Docker
Desktop runs it on either kind of Windows machine. WSL2 with the Linux
instructions above works too.

  </Tab>

</Tabs>

## Verify it's running [#verify-its-running]

In another terminal:

```bash
curl -v http://localhost:50051/
```

You should see the connection open and an HTTP response come back. The
status will be an error, since `/` is not a gRPC route; the exact body
doesn't matter, the connection succeeding does.

## Point your build system at it [#point-your-build-system-at-it]

<Tabs items={["Bazel", "Buck2", "Siso", "Pants"]}>

  <Tab value="Bazel">

Add to your `.bazelrc`:

```
build --remote_cache=grpc://localhost:50051
build --remote_instance_name=main
```

Or pass them per invocation:

```bash
bazel build //... \
  --remote_cache=grpc://localhost:50051 \
  --remote_instance_name=main
```

Run any build target. Subsequent rebuilds of unchanged targets should
report a `remote cache hit` count in the final `processes:` summary line.
Leave `--remote_executor` off for now: the container image holds only the
NativeLink binary, no shell or compilers, so actions sent to its worker
cannot run. [Remote execution](/remote-execution) sets up workers with a
toolchain.

  </Tab>

  <Tab value="Buck2">

In your project root, add `.buckconfig`:

```ini
[buck2_re_client]
engine_address = localhost:50051
action_cache_address = localhost:50051
cas_address = localhost:50051
tls = false
instance_name = main
```

Then build as usual:

```bash
buck2 build //...
```

Buck2 also needs an execution platform with `remote_enabled = True`; the
[Buck2](/getting-started/connect-your-build/buck2) page shows the one the
checked-in integration test uses.

  </Tab>

  <Tab value="Siso">

Set the endpoint variables Siso reads:

```bash
export SISO_REAPI_ADDRESS=localhost:50051
export SISO_REAPI_INSTANCE=main
export RBE_service_no_security=true   # only for local TLS-free dev
```

The Chromium build is the canonical Siso consumer; see
[Chromium](/getting-started/connect-your-build/chromium) for a full example.

  </Tab>

  <Tab value="Pants">

In `pants.toml`:

```toml
[GLOBAL]
remote_cache_read = true
remote_cache_write = true
remote_store_address = "grpc://localhost:50051"
remote_instance_name = "main"
```

Then run any goal as usual; Pants will route through the cache.

  </Tab>

</Tabs>

## Confirm cache hits [#confirm-cache-hits]

Re-run your build a second time without changing any source files. Every
action should land in the cache and report a hit. With Bazel:

```bash
bazel build //... \
  --remote_cache=grpc://localhost:50051 \
  --execution_log_json_file=/tmp/exec.json

# Count the cache hits vs misses
grep -c '"remoteCacheHit": true' /tmp/exec.json
```

If you see hits, you're done. If not, jump to the troubleshooting
section below.

## Troubleshooting [#troubleshooting]

<Callout type="warn" title="Connection refused">
  The NativeLink server isn't listening on the port your client is
  pointing at. Confirm with `lsof -i :50051` (macOS / Linux) or
  `netstat -ano | findstr 50051` (Windows). Check the Docker container is
  still running with `docker ps`.
</Callout>

<Callout type="warn" title="Cache writes succeed, reads always miss">
  Most often this means two clients are using different `instance_name`
  values or different toolchain configurations. Compare the
  `--remote_instance_name` flag (Bazel) or the equivalent in your build
  system across the two invocations. Toolchain mismatches surface as
  action hashes that never collide.
</Callout>

<Callout type="warn" title="Permission denied on /config in Docker">
  SELinux-enforcing distributions (Fedora, RHEL) require an explicit
  `:Z` label on the mounted volume:
  `-v $(pwd)/basic_cas.json5:/config:Z`.
</Callout>

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="What is remote caching actually doing?" id="what-is-remote-caching-actually-doing">
    Every action's inputs (source digests, the compiler binary, the
    exact command line, declared headers) are hashed together, and the
    action's outputs are stored under that hash. Anyone whose action
    hashes identically gets the stored outputs back in milliseconds
    instead of redoing the work.
  </Accordion>
  <Accordion title="Why does a one-byte change cause a miss?" id="why-does-a-one-byte-change-cause-a-miss">
    The hash covers every declared input, so any difference (a header,
    a compiler patch version, a leaked environment variable) produces a
    new hash. That's by design: a hit is a guarantee the result is
    identical. If you see misses where you expect hits, start with the
    [troubleshooting](#troubleshooting) section above and
    [correctness and hermeticity](/explanations/correctness-hermeticity#hermeticity-is-the-clients-job).
  </Accordion>
  <Accordion title="Do I need --remote_cache when --remote_executor is set?" id="do-i-need-remote-cache-when-remote-executor-is-set">
    No. When `--remote_cache` is empty Bazel reuses the `--remote_executor`
    endpoint for the cache, so one flag is enough when the same NativeLink
    serves both. Set `--remote_cache` explicitly only when the cache lives
    at a different address from the scheduler.
  </Accordion>
</Accordions>

<NextStep href="/getting-started/verify-your-cache" title="Verify your cache">
  A server that starts and a build that succeeds are not evidence that
  anything was cached. This is the line to read, and what it means when it
  says zero.
</NextStep>

<NextStep href="/configuration" kind="aside" title="Configuration">
  What is in that JSON5 file, and how to grow it from a single node to a
  fleet.
</NextStep>

<NextStep href="/explanations/architecture" kind="aside" title="Architecture">
  What each piece of the cluster is responsible for, and why the split is
  drawn where it is.
</NextStep>

# https://docs.nativelink.com/getting-started/verify-your-cache

**Verify your cache**: Prove the cache is actually serving hits rather than quietly falling back to local work, using your build tool's own accounting, not a status page.

**Who this is for:** anyone who has pointed a build at NativeLink and wants
evidence it is working, rather than the absence of an error. **What you'll
have at the end:** a measurement you can repeat, a number you can put in a
message to your team, and the ability to tell a cold cache from a broken one.
**Time:** about ten minutes.

<Prerequisites>
  A running NativeLink and a build pointed at it. See the [Quickstart](/getting-started/quickstart) and [Connect your build](/getting-started/connect-your-build).
</Prerequisites>

A misconfigured remote cache almost never fails loudly. Bazel's default is to
treat cache errors as soft: if the endpoint is unreachable, the digest function
does not match, or the instance name is wrong, it logs a line you will not
notice and builds everything locally. The build succeeds. It is just not
using the cache.

"No errors" proves nothing. What proves something is the client's own
accounting of where each action's result came from.

<Callout type="warn" title="There is no /metrics endpoint to curl">
  NativeLink exports telemetry over OTLP and nothing else: there is no HTTP
  metrics route in the binary, and the `/status` path a `health` service
  mounts only reports whether the process is up and its stores pass a
  self-check. It will happily return OK for a server no client is
  successfully talking to. Use it as a liveness check and nothing more; see
  <a href="/operate/observability">Observability</a> for the real telemetry
  path.
</Callout>

## The measurement [#the-measurement]

The test is a build, a clean, and the same build again. The first populates
the cache; the second must not do the work.

<Steps>

<li>

### Build once, cold [#build-once-cold]

```bash
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_instance_name=main
```

Read the last line of the output. It looks like this:

```
INFO: 1,284 processes: 81 internal, 1,203 linux-sandbox.
```

Every process is local, which is what you expect: nothing was in the cache
yet. If this run reports remote cache hits, something built these outputs
before. That is fine, but you are no longer measuring a cold start, so pick
a target nobody has built or change a source file.

</li>

<li>

### Discard the local results [#discard-the-local-results]

```bash
bazel clean
```

Not `bazel clean --expunge`, which also throws away the external repositories
and turns the next step into a ten-minute fetch that tells you nothing about
the cache.

</li>

<li>

### Build again, warm [#build-again-warm]

```bash
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_instance_name=main
```

Now the same line should read differently:

```
INFO: 1,284 processes: 1,203 remote cache hit, 81 internal.
```

That number, `remote cache hit`, is the whole measurement. It is the count
of actions Bazel did not run because NativeLink handed back a result.

</li>

</Steps>

<VerifyBlock>

- The second build's `processes:` line reports a non-zero `remote cache hit`
  count.
- The second build is substantially faster in wall time than the first.
- The store's `content_path` on the server has grown; for the shipped
  `basic_cas.json5` that is
  `/tmp/nativelink/data-worker-test/content_path-cas` (inside the container
  on the Docker path), and the action cache is beside it under
  `content_path-ac`.
- Stopping NativeLink and re-running the build makes the hits disappear. This
  one matters more than it looks: it is the only check that proves the hits
  were coming from *this* server rather than from a `--disk_cache` on the
  client.

</VerifyBlock>

## Reading the number honestly [#reading-the-number-honestly]

A hit rate below 100% on an unchanged tree is normal. Bazel counts several
kinds of work in that line, and only some of it is cacheable:

`internal` actions (symlink trees, file writes, workspace status) never go
to a remote cache. They are cheap and they will always appear.

Actions whose inputs genuinely changed are supposed to miss. If you edited a
header between the two runs, everything downstream of it will rebuild, and
that is the cache working correctly rather than failing.

Actions that are not deterministic will miss forever. A rule that embeds a
timestamp, an absolute path, or a hostname produces a different action digest
on every machine and every run, so it can never hit. If your rate is stuck low
on an unchanged tree, this is usually why, and
[Correctness and hermeticity](/explanations/correctness-hermeticity) is the
page that explains how to find them.

What should worry you is *zero* hits on a second run of an unchanged tree.
That is not a tuning problem; that is the cache not being used at all.

## Getting more detail than the summary line [#getting-more-detail-than-the-summary-line]

When the summary is not enough, ask Bazel for the per-action record:

```bash
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_instance_name=main \
  --execution_log_compact_file=/tmp/exec.log
```

The execution log records, for each action, whether it was a cache hit and
what its digest was. It is the tool for answering "why did *this specific*
action miss": compare the digest for the same action across two runs, and if
it changed, the action's inputs or command line are not stable.

For wire-level questions (is the client even reaching the server, is it
calling `FindMissingBlobs`) turn on gRPC logging instead:

```bash
bazel build //... --remote_cache=grpc://127.0.0.1:50051 --remote_grpc_log=/tmp/grpc.log
```

An empty log means Bazel never opened a connection, which points at the flag
rather than at the server.

## Other build tools [#other-build-tools]

The measurement is the same everywhere: build, clean, rebuild, and read the
tool's own accounting. Only the vocabulary changes. Buck2 shows where each
action ran in `buck2 log what-ran`; Pants counts remote cache requests in
its end-of-run stats when `[stats].log` is on; recc logs an
`Action Cache hit` line per compile with `RECC_LOG_LEVEL=info`. Look for
whatever your tool calls a cache hit rather than for a NativeLink-side signal;
the server cannot tell you whether the client asked it the right question.

<Accordions type="multiple">

<Accordion title="Zero cache hits, and no errors anywhere" id="zero-cache-hits-and-no-errors">
The overwhelmingly common cause is `--remote_instance_name`. NativeLink
registers services under the instance names its config lists, and the shipped
`basic_cas.json5` registers each service twice, once under `""` and once
under `"main"`, precisely so both work. A config that registers only `main`
answers a client that sends the empty instance name with an error,
`'instance_name' not configured for ''`; Bazel treats a cache error as a
warning, prints it once, and builds locally, which reads as zero hits.
<SourceLink file="nativelink-service/src/ac_server.rs" symbol="get_action_result" />

Second most common: the two runs did not use the same digest function. Bazel
sends SHA-256 by default; if you set `--digest_function=blake3` on one run and
not the other, every digest differs and nothing can hit.
</Accordion>

<Accordion title="Hits on the second run, but not from a colleague's machine" id="hits-locally-but-not-from-another-machine">
Two builds share cache entries only if they produce identical action digests,
and the digest covers the full command line, the input file contents, *and*
the platform properties. Different compiler versions, different absolute paths
to the workspace, or a different `--platforms` value all produce different
digests.

This is the problem remote execution and toolchain pinning exist to solve, and
[Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
covers it properly. For a cache-only setup, the practical first step is making
sure everyone builds with the same toolchain rather than the one on their
`PATH`.
</Accordion>

<Accordion title="The second build is faster but reports no remote hits" id="faster-but-no-remote-hits">
`bazel clean` removes the output tree but not the repository cache or a
`--disk_cache` if you have one configured, and disk cache hits are counted
separately from remote ones. A run that is fast with zero remote hits is
being served locally. Stop NativeLink and run the build again: if it is
still fast, you have been measuring Bazel's local caches the whole time.
</Accordion>

<Accordion title="Hits disappear after a while" id="hits-disappear-after-a-while">
Eviction. The shipped example caps the CAS at 10 GB and the action cache at
1 GB, and a filesystem store evicts least-recently-used content when it
crosses the limit. On a real workload those limits are small.

The failure mode worth knowing is that the two stores evict independently: a
CAS eviction can leave an action cache entry pointing at blobs that no longer
exist, so the client gets a hit and then fails to download it. That is what a
completeness-checking store prevents, and
[Scaling the CAS](/operate/scaling-cas) covers sizing.
</Accordion>

</Accordions>

<NextStep href="/getting-started/connect-your-build" title="Connect your build">
  The flags for Bazel, Buck2, Pants, Siso, and CMake with recc, now that you
  know how to tell whether they worked.
</NextStep>

<NextStep href="/remote-execution" kind="aside" title="Remote execution">
  A cache serving hits is the prerequisite for running the actions themselves
  somewhere else.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build

**Connect your build tool**: Use NativeLink without Bazel: Buck2, Siso, Pants, BuildStream, and CMake with recc.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

NativeLink speaks the standard Remote Execution API. If your build tool
speaks the same protocol, you can point it at NativeLink without rewriting
your build files.

## Options [#options]

- [Buck2](/getting-started/connect-your-build/buck2): use Buck2's
  built-in remote execution client with NativeLink as CAS, Action Cache,
  and executor.
- [Siso](/getting-started/connect-your-build/siso): configure
  Chromium's Ninja replacement to use NativeLink through the RE API.
- [Pants](/getting-started/connect-your-build/pants): enable remote
  cache reads and writes from `pants.toml`.
- [BuildStream](/getting-started/connect-your-build/buildstream): point
  BuildStream artifact storage and remote execution at NativeLink.
- [CMake with recc](/getting-started/connect-your-build/cmake-recc):
  wrap CMake compile actions with `recc` and share compile outputs
  through NativeLink.

## Common NativeLink endpoint [#common-nativelink-endpoint]

The examples below assume a NativeLink server exposing CAS, Action Cache,
execution, capabilities, and ByteStream on port `50051` with instance name
`main`.

The BuildStream page follows the checked-in integration test, which uses
the default empty instance on the same local port.

For a local Docker server, start with
[Getting Started → Quickstart](/getting-started/quickstart). For the Buck2 integration
test shape, see
<SourceLink file="integration_tests/buck2/buck2_cas.json5" symbol="integration_tests/buck2/buck2_cas.json5" />.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Can different build tools share one cache?" id="can-different-build-tools-share-one-cache">
    Yes: they speak the same protocol against the same endpoint. Each
    tool hashes its actions differently, so tools won't share entries
    with each other, but every client of the same tool and toolchain
    will.
  </Accordion>
  <Accordion title="Why does the instance name matter so much?" id="why-does-the-instance-name-matter-so-much">
    The instance name selects which configured store or scheduler a
    request is routed to. If a client sends an instance name the server
    does not expose, NativeLink rejects the request with an error of the
    form `'instance_name' not configured for '...'`; depending on the
    client, that shows up as a cache warning or a failed build rather than
    a cache hit. Every client targeting a cache must send an instance name
    the server exposes.
    <SourceLink file="nativelink-service/src/ac_server.rs" symbol="inner_get_action_result" />
  </Accordion>
  <Accordion title="Which client is the fastest way to try NativeLink?" id="which-client-is-the-fastest-way-to-try-nativelink">
    If you're on CMake,
    [recc in cache-only mode](/getting-started/connect-your-build/cmake-recc)
    needs no worker fleet and no build-file changes. For Bazel, two
    `.bazelrc` lines against the [Quickstart](/getting-started/quickstart) server
    do it.
  </Accordion>
</Accordions>

# https://docs.nativelink.com/getting-started/connect-your-build/buck2

**Buck2**: Configure Buck2 to use NativeLink for remote cache and execution.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

Buck2 has a native Remote Execution API client. NativeLink can serve the
CAS, Action Cache, capabilities, and execution services that Buck2 expects.

The repo includes a runnable Buck2 integration test at
<SourceLink file="integration_tests/buck2" dir symbol="integration_tests/buck2/" />.
It starts NativeLink with a combined scheduler and local worker, runs
`buck2 build //...`, and checks for `BUILD SUCCEEDED`.

## Start NativeLink [#start-nativelink]

Use a config that exposes these services on `localhost:50051`:

- CAS
- Action Cache
- execution
- capabilities
- ByteStream

The integration test uses
<SourceLink file="integration_tests/buck2/buck2_cas.json5" symbol="integration_tests/buck2/buck2_cas.json5" />
and instance name `main`.

## Configure Buck2 [#configure-buck2]

In your Buck2 project, add `.buckconfig`:

```ini
[cells]
root = .

[buck2_re_client]
engine_address = localhost:50051
action_cache_address = localhost:50051
cas_address = localhost:50051
tls = false
instance_name = main

[build]
execution_platforms = root//platforms:platforms
```

The checked-in integration test uses the same settings in
<SourceLink file="integration_tests/buck2/.buckconfig" symbol="integration_tests/buck2/.buckconfig" />.

## Register an execution platform [#register-an-execution-platform]

Buck2 needs an execution platform that allows remote actions. The
integration test registers one with `remote_enabled = true` and
`local_enabled = true`:

```python
platform = ExecutionPlatformInfo(
    label = ctx.label.raw_target(),
    configuration = configuration,
    executor_config = CommandExecutorConfig(
        local_enabled = True,
        remote_enabled = True,
        use_limited_hybrid = True,
        remote_execution_properties = {},
        remote_execution_use_case = "buck2-default",
        remote_output_paths = "output_paths",
    ),
)
```

See
<SourceLink file="integration_tests/buck2/platforms/defs.bzl" symbol="integration_tests/buck2/platforms/defs.bzl" />
for the complete rule.

## Run a build [#run-a-build]

```bash
buck2 build //...
```

The integration test uses a small staged build graph that mixes local-only
and remote-capable actions, then verifies the output with `diff`. That shape
is useful when validating a new cluster because it proves both upload and
download paths are working.

## Troubleshooting [#troubleshooting]

- **Buck2 cannot connect.** Confirm NativeLink is listening on `50051` and
  Buck2 is not expecting TLS.
- **Everything runs locally.** Check that your selected execution platform
  sets `remote_enabled = True`.
- **Instance-name errors.** Make sure Buck2's `instance_name` matches the
  NativeLink config. The integration test uses `main`.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Why do all three addresses point at the same listener?" id="why-do-all-three-addresses-point-at-the-same-listener">
    `engine_address`, `action_cache_address`, and `cas_address` are
    separate settings because Buck2 lets you split those services across
    hosts. NativeLink serves them all from one listener, so all three
    point at the same `host:port`.
  </Accordion>
  <Accordion title="Does NativeLink support Buck2 remote execution, or only caching?" id="does-nativelink-support-buck2-remote-execution-or-only-caching">
    Both. Execution needs an execution platform registered with
    `remote_enabled = True` (above) plus a NativeLink scheduler and
    worker; the integration test config runs one of each.
  </Accordion>
</Accordions>

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  Buck2 is talking to the cache on one machine. The step that makes it worth the setup is the same cache answering for everyone.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build/siso

**Siso**: Point Siso at NativeLink for Chromium-style remote caching and execution.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

[Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md)
is Chromium's Ninja replacement. It runs build actions on RBE natively,
and NativeLink can be the RE API backend for Chromium and other
Siso-driven builds.

For a production Chromium walkthrough, see
[Chromium](/getting-started/connect-your-build/chromium).

## Local endpoint setup [#local-endpoint-setup]

Start NativeLink on `localhost:50051` with instance name `main`, then
export the environment variables Siso reads:

```bash
export SISO_REAPI_ADDRESS=localhost:50051
export SISO_REAPI_INSTANCE=main
export RBE_service_no_security=true
```

For CMake projects, [`recc`](/getting-started/connect-your-build/cmake-recc)
is often the smaller setup. It uses the same Remote Execution API but fits
CMake compiler-launcher workflows directly.

## Chromium-style setup [#chromium-style-setup]

Chromium checks in Siso configs. Point them at NativeLink by enabling
remote execution in GN:

```bash
gn gen out/Default --args='
  use_remoteexec=true
  use_siso=true
'
```

Build normally with `autoninja`; it reads `use_siso=true` from `args.gn`
and invokes `siso ninja`:

```bash
autoninja -C out/Default chrome
```

The repository's own Chromium example does the same thing from a script:
<SourceLink file="deploy/chromium-example/build_chromium_tests.sh" symbol="deploy/chromium-example/build_chromium_tests.sh" />.
Chromium's Siso config also expects a `backend.star` that names the
platform properties (`OSFamily`, `container-image`) your workers must
advertise; see [Chromium](/getting-started/connect-your-build/chromium#worker-requirements).

## Production TLS settings [#production-tls-settings]

For a secured NativeLink endpoint, set the Siso TLS variables instead
of `RBE_service_no_security=true`:

```bash
export SISO_REAPI_ADDRESS=cas.nativelink.internal:50051
export SISO_REAPI_INSTANCE=main
export RBE_service_no_security=false
export RBE_tls_client_auth_cert=/etc/nativelink/tls/client.crt
export RBE_tls_client_auth_key=/etc/nativelink/tls/client.key
export RBE_tls_ca_cert=/etc/nativelink/tls/ca.crt
```

## Confirm it is working [#confirm-it-is-working]

On a warm cache, cache hits should climb and network errors should stay at
zero. If actions fall back to local execution, compare the platform
properties Siso requests with the properties your NativeLink workers
advertise.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Is RBE_service_no_security=true safe?" id="is-rbe-service-no-security-true-safe">
    Only for TLS-free local development. Against any shared endpoint, use
    the TLS variables shown above instead.
  </Accordion>
  <Accordion title="Siso or recc: which do I want?" id="siso-or-recc-which-do-i-want">
    Siso replaces Ninja and speaks full remote execution; it's the path
    for Chromium-style builds. recc is a compiler launcher for ordinary
    CMake projects that want caching with minimal setup; see
    [CMake with recc](/getting-started/connect-your-build/cmake-recc).
  </Accordion>
</Accordions>

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  Siso is pointed at NativeLink. Next is making that endpoint one your team and your CI both hit.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build/chromium

**Chromium**: Building Chromium with NativeLink as the Siso backend, a worked example for the largest public consumer.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

Chromium's build system shells out to
[Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md),
which speaks the Remote Execution API. Pointing it at a NativeLink cluster is a
configuration change, with no patches to the Chromium tree.

The same pattern works for upstream Chromium and any Chromium-based
browser.

## Prerequisites [#prerequisites]

- A NativeLink cluster reachable from your build machines. Cache-only
  is fine to start; remote execution drops build times further.
- A Chromium checkout with `depot_tools` on `$PATH`.
- The Chromium source checkout (`fetch chromium`).

## Configure Siso [#configure-siso]

Siso reads its server settings from environment variables. Add
to your build shell:

```bash
export SISO_REAPI_ADDRESS=cas.nativelink.internal:50051
export SISO_REAPI_INSTANCE=main
export RBE_service_no_security=false   # mTLS in production
export RBE_tls_client_auth_cert=/etc/nativelink/tls/client.crt
export RBE_tls_client_auth_key=/etc/nativelink/tls/client.key
export RBE_tls_ca_cert=/etc/nativelink/tls/ca.crt
```

`SISO_REAPI_INSTANCE` must match an `instance_name` your NativeLink
server exposes. The repository's own Chromium example uses the default
empty instance name (`SISO_REAPI_INSTANCE=''`) and also sets
`SISO_REAPI_CAS_ADDRESS` to the same endpoint:
<SourceLink file="deploy/chromium-example/build_chromium_tests.sh" symbol="deploy/chromium-example/build_chromium_tests.sh" />.

For local development against a non-TLS NativeLink:

```bash
export SISO_REAPI_ADDRESS=localhost:50051
export SISO_REAPI_INSTANCE=main
export RBE_service_no_security=true
```

## Generate Chromium build files [#generate-chromium-build-files]

In your Chromium checkout:

```bash
gn gen out/Default --args='
  use_remoteexec=true
  use_siso=true
'
```

`autoninja` reads `args.gn`; with `use_siso=true`, it invokes
`siso ninja` for the build.

## Build [#build]

```bash
autoninja -C out/Default chrome
```

Watch the build go. On a fresh cluster the first build will be
mostly misses; subsequent builds hit the cache and run in a fraction
of the time.

## What good looks like [#what-good-looks-like]

- **Cache hit rate**: on a warm cache the hit rate for an incremental
  build should be high and climb from build to build; watch it in the
  NativeLink cache metrics.
- **Local fallback rate**: close to zero. Anything higher means workers
  are rejecting actions (platform mismatch, capacity).
- **Network errors**: zero. Anything else is a problem.

## Worker requirements [#worker-requirements]

Chromium's Siso config takes its remote platform properties from
`build/config/siso/backend_config/backend.star` (copy `template.star`
and edit it, or point the `reapi_backend_config_path` gclient custom
var at your own file). Each platform entry names an `OSFamily` and a
`container-image`, and every NativeLink worker that should run Chromium
actions must advertise the same values, with the property key spelled
`container-image` (hyphen).

The repository ships a Chromium worker image built from Chromium's
public `siso-chromium/linux` RBE image, and a kustomization that deploys
it alongside a NativeLink cluster:

- <SourceLink file="deploy/chromium-example" dir symbol="deploy/chromium-example/" />
  for the deployment,
- <SourceLink file="kubernetes/workers/siso-chromium/kustomization.yaml" symbol="kubernetes/workers/siso-chromium/kustomization.yaml" />
  for the worker, which uses the shared worker config in
  <SourceLink file="kubernetes/components/worker/worker.json5" symbol="kubernetes/components/worker/worker.json5" />
  (`OSFamily: Linux`, `container-image` from the worker image, `cpu_count`
  from `nproc`).

## Troubleshooting [#troubleshooting]

- **All actions falling back to local.** Likely a platform-property
  mismatch. Compare what Chromium is requesting (in the siso
  log) against what your workers advertise.
- **Siso is not being selected.** If the output directory was already
  generated for Ninja, run `gn clean out/Default`, regenerate with
  `use_siso=true`, and build again.
- **Cache writes succeed, reads always miss.** Almost always a
  toolchain version mismatch between the action that wrote the
  cache and the one trying to read.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Do I need to patch the Chromium tree?" id="do-i-need-to-patch-the-chromium-tree">
    No. Pointing Siso at NativeLink is configuration only: GN args, the
    Siso environment variables, and a local `backend.star` for the
    platform properties. The same applies to Chromium forks.
  </Accordion>
  <Accordion title="Can I start cache-only?" id="can-i-start-cache-only">
    Yes. A cache-only cluster already cuts incremental build times; add
    remote execution later for the full win on link-heavy full builds.
  </Accordion>
</Accordions>

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  A Chromium build is where the difference between one machine's cache and
  the team's is largest. This is how to make it the team's.
</NextStep>

<NextStep href="/remote-execution/persistent-workers" kind="aside" title="Persistent workers">
  For the hot toolchain paths, where process startup is a measurable share of
  the action.
</NextStep>

<NextStep href="/operate/observability" kind="aside" title="Observability">
  Chromium builds produce a lot of telemetry; it is worth having the
  dashboards before you need them.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build/pants

**Pants**: Enable Pants remote caching with NativeLink.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

Pants can read and write remote cache entries through the Remote Execution
API. NativeLink can provide the remote store and Action Cache for Pants
goals without changing BUILD files.

## Start NativeLink [#start-nativelink]

Start a NativeLink server that exposes CAS and Action Cache on
`localhost:50051` with instance name `main`. The Docker quickstart in
[Getting Started → Quickstart](/getting-started/quickstart) is enough for local
cache validation.

## Configure Pants [#configure-pants]

Add the remote cache settings to `pants.toml`:

```toml
[GLOBAL]
remote_cache_read = true
remote_cache_write = true
remote_store_address = "grpc://localhost:50051"
remote_instance_name = "main"
```

For a shared deployment, replace `localhost:50051` with your NativeLink
endpoint and use the same instance name your server config exposes.

## Run a goal [#run-a-goal]

Run any Pants goal as usual:

```bash
pants test ::
```

The first run uploads results. A second run against the same inputs should
read cached process results from NativeLink.

## Remote execution [#remote-execution]

Pants remote execution uses the same Remote Execution API, but it needs a
worker environment that matches the actions Pants sends. Start with remote
caching, then add remote execution once you have worker platform properties
and toolchains aligned.

## Troubleshooting [#troubleshooting]

- **No cache hits on the second run.** Check that `remote_cache_write` and
  `remote_cache_read` are both enabled.
- **Instance-name errors.** Match `remote_instance_name` to the NativeLink
  config. The local examples use `main`.
- **Connection failures.** Pants expects a `grpc://` URL in
  `remote_store_address` for a plaintext endpoint, or `grpcs://` for
  TLS.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Why are read and write separate flags?" id="why-are-read-and-write-separate-flags">
    To let you split roles: CI can populate the cache
    (`remote_cache_write = true`) while developer laptops only read from
    it. For a single shared setup, enable both.
  </Accordion>
  <Accordion title="Can Pants run actions remotely on NativeLink?" id="can-pants-run-actions-remotely-on-nativelink">
    Yes (it's the same Remote Execution API), but the worker environment
    must match what Pants sends. Get remote caching working first, then
    align worker platform properties and toolchains before enabling
    execution.
  </Accordion>
</Accordions>

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  Pants is reading and writing the cache from here. Next is making the cache something the whole team and CI share.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build/buildstream

**BuildStream**: Configure BuildStream to use NativeLink for artifact storage and remote execution.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

BuildStream can use a Remote Execution API backend for artifact storage,
Action Cache, CAS, and remote execution. NativeLink can serve those
endpoints from the same `localhost:50051` listener used by the other local
examples.

The repo includes a runnable BuildStream integration test at
<SourceLink file="integration_tests/buildstream" dir symbol="integration_tests/buildstream/" />.
It starts NativeLink with a local worker, runs `bst build hello.bst`, and
checks for `SUCCESS Build`.

## Start NativeLink [#start-nativelink]

Use a config that exposes CAS, Action Cache, execution, capabilities,
ByteStream, fetch, and push services on `localhost:50051`.

The integration test uses
<SourceLink file="integration_tests/buildstream/buildstream_cas.json5" symbol="integration_tests/buildstream/buildstream_cas.json5" />.
It also starts a private worker API on `localhost:50061` for the local
worker.

```bash
nativelink -- integration_tests/buildstream/buildstream_cas.json5
```

## Configure BuildStream [#configure-buildstream]

Add a BuildStream config file that points artifact storage and remote
execution services at NativeLink:

```yaml
cachedir: /tmp/buildstream

artifacts:
  servers:
  - url: http://localhost:50051
    push: true

remote-execution:
  execution-service:
    url: http://localhost:50051
  action-cache-service:
    url: http://localhost:50051
  storage-service:
    url: http://localhost:50051
```

The checked-in integration test uses the same settings in
<SourceLink file="integration_tests/buildstream/buildstream.conf" symbol="integration_tests/buildstream/buildstream.conf" />.

## Configure a project [#configure-a-project]

A minimal BuildStream project needs a project name, a supported BuildStream
version, and an element directory:

```yaml
name: first-project
min-version: 2.4
element-path: elements

aliases:
  alpine: https://bst-integration-test-images.ams3.cdn.digitaloceanspaces.com/
```

The integration test uses this shape in
<SourceLink file="integration_tests/buildstream/project.conf" symbol="integration_tests/buildstream/project.conf" />.

## Add an element [#add-an-element]

This small element stages local source files and runs a build command:

```yaml
kind: manual
description: |
  Building manually

depends:
- base.bst

sources:
  - kind: local
    path: files/src

config:
  build-commands:
  - echo hello.c
```

See
<SourceLink file="integration_tests/buildstream/elements/hello.bst" symbol="integration_tests/buildstream/elements/hello.bst" />
for the complete test element.

## Run a build [#run-a-build]

Run BuildStream with the config file:

```bash
bst -c buildstream.conf build hello.bst
```

The integration test checks for `SUCCESS Build` in the BuildStream output
and checks that NativeLink did not log an error.

## Troubleshooting [#troubleshooting]

- **BuildStream cannot connect.** Match the `http://localhost:50051`
  endpoint in `buildstream.conf` to the NativeLink listener. The local test
  uses an insecure HTTP URL.
- **Uploads are missing.** Set `push: true` under `artifacts.servers` so
  BuildStream can write artifacts back to NativeLink.
- **Remote actions do not match a worker.** Compare the platform properties
  BuildStream requests with the properties your NativeLink worker advertises.
  The integration test worker advertises `ISA: x86-64`.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Why does the config have both artifacts and remote-execution blocks?" id="why-does-the-config-have-both-artifacts-and-remote-execution-blocks">
    They configure different BuildStream subsystems: artifact push/pull
    storage versus the execution, action-cache, and storage services used
    for remote builds. Pointing every URL at the same NativeLink listener
    is fine; it exposes all of those services on one port.
  </Accordion>
  <Accordion title="Why http:// URLs instead of grpc://?" id="why-http-urls-instead-of-grpc">
    BuildStream expresses these endpoints as URLs rather than bare
    `host:port` pairs. The checked-in integration test runs without TLS,
    hence `http://localhost:50051`.
  </Accordion>
</Accordions>

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  BuildStream is pushing and pulling artifacts against one server. Next is making that server the team's.
</NextStep>

# https://docs.nativelink.com/getting-started/connect-your-build/cmake-recc

**CMake with recc**: Accelerate CMake builds with NativeLink as a remote cache, using recc as the bridge.

<Prerequisites>
  A running NativeLink instance. If you don't have one, start with the [Quickstart](/getting-started/quickstart).
</Prerequisites>

<Callout title="Inspired by Reid Kleckner" type="idea">
This tutorial follows the approach Reid Kleckner laid out in
[Distributed builds of LLVM with CMake, recc, and NativeLink](https://reidkleckner.dev/posts/llvm-recc-nativelink/).
Read his post for the LLVM-scale walkthrough and the reasoning behind
each piece. This page is the short, hands-on version for your own
projects.
</Callout>

[reid]: https://reidkleckner.dev/posts/llvm-recc-nativelink/

*Read time: ~5 minutes. Hands-on time: another ~5 if you have Docker and a
package manager handy; longer on Linux if you build `recc` from source.*

This tutorial gets you from zero to a CMake build hitting a NativeLink
remote cache. Works on Linux and macOS (tested on macOS Apple Silicon;
Linux x86_64 follows the exact same commands).

We use [`recc`](https://buildgrid.gitlab.io/recc), a small Remote Execution
v2 client originally from Bloomberg, as a CMake compiler launcher. We run
it in **cache-only mode**: compiles still happen on your machine, only the
*outputs* travel through NativeLink. That keeps the setup tiny while still
giving you cross-machine and cross-branch caching.

## Prerequisites [#prerequisites]

- Docker
- CMake 3.16+
- A C/C++ compiler (clang or gcc)

## 1. Start NativeLink [#1-start-nativelink]

```sh
curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/v1.6.5/nativelink-config/examples/basic_cas.json5

docker run -d --name nativelink \
  -v $(pwd)/basic_cas.json5:/config \
  -p 50051:50051 -p 50061:50061 \
  ghcr.io/tracemachina/nativelink:v1.6.5 config
```

The image is multi-arch (x86_64 and ARM64) as of v1.6.0, so it runs
natively on Linux and Apple Silicon alike.

Confirm it came up:

```sh
docker logs nativelink 2>&1 | grep "Ready, listening on 0.0.0.0:50051"
```

## 2. Install recc [#2-install-recc]

<Tabs items={["macOS / Homebrew", "Linux from source (apt)"]}>
  <Tab value="macOS / Homebrew">
    ```sh
    brew install recc
    ```

    Works on Linux too if you have [Homebrew on Linux](https://docs.brew.sh/Homebrew-on-Linux).
  </Tab>
  <Tab value="Linux from source (apt)">
    `recc` now lives in the BuildBox monorepo (the old `recc` and
    `buildbox-common` repositories redirect there). On Ubuntu / Debian,
    install the build deps listed in the BuildBox README:

    ```sh
    sudo apt-get update
    sudo apt-get install -y cmake g++ gcc git googletest libabsl-dev \
      libgmock-dev libgrpc++-dev libprotobuf-dev protobuf-compiler-grpc \
      libssl-dev pkg-config uuid-dev nlohmann-json3-dev libcurl4-openssl-dev
    ```

    Then build BuildBox, which builds and installs `recc` with it:

    ```sh
    git clone https://gitlab.com/BuildGrid/buildbox/buildbox.git
    cd buildbox && mkdir build && cd build
    cmake .. && make -j"$(nproc)"
    sudo make install
    ```

    `recc` is now on your `PATH`. If you hit issues, see the
    [BuildBox install docs](https://buildgrid.gitlab.io/buildbox/buildbox/installation.html)
    and the [packaged recc page](https://buildgrid.gitlab.io/buildbox/buildbox/recc-installation.html).
  </Tab>
</Tabs>

## 3. Create the example [#3-create-the-example]

Drop these two files into a fresh project directory and `cd` into it.
You can also copy them from
<SourceLink file="templates/cmake" dir symbol="templates/cmake/" />
in the repo.

```cmake
# CMakeLists.txt
cmake_minimum_required(VERSION 3.16)
project(hello LANGUAGES CXX)
set(CMAKE_CXX_STANDARD 17)
add_executable(hello main.cpp)
```

```cpp
// main.cpp
#include <iostream>

int main() {
    std::cout << "Hello, NativeLink!\n";
    return 0;
}
```

## 4. Point recc at NativeLink [#4-point-recc-at-nativelink]

```sh
export RECC_SERVER=localhost:50051
export RECC_INSTANCE=main
export RECC_CACHE_ONLY=1
export RECC_CACHE_UPLOAD_LOCAL_BUILD=1
```

What each one does:

- `RECC_SERVER`: gRPC endpoint for the CAS / Action Cache.
- `RECC_INSTANCE=main`: must match an `instance_name` in your NativeLink
  config. `basic_cas.json5` exposes both `""` and `"main"`; `recc`
  defaults to `""` unless it was built with a different
  `DEFAULT_RECC_INSTANCE`, so setting it explicitly removes the guesswork.
- `RECC_CACHE_ONLY=1`: if there's no cached result, build locally
  instead of waiting on a remote worker.
- `RECC_CACHE_UPLOAD_LOCAL_BUILD=1`: push the result of a local compile
  back up so the next build (or the next teammate) gets a hit. It only
  has an effect together with `RECC_CACHE_ONLY`.

## 5. Build it [#5-build-it]

```sh
RECC=$(command -v recc)

cmake -B build -S . \
  -DCMAKE_C_COMPILER_LAUNCHER=$RECC \
  -DCMAKE_CXX_COMPILER_LAUNCHER=$RECC

cmake --build build
./build/hello
```

You should see `Hello, NativeLink!`.

## 6. Confirm the cache is live [#6-confirm-the-cache-is-live]

Wipe the build directory and rebuild. This time the compile output
should come straight out of NativeLink:

```sh
rm -rf build

cmake -B build -S . \
  -DCMAKE_C_COMPILER_LAUNCHER=$RECC \
  -DCMAKE_CXX_COMPILER_LAUNCHER=$RECC

RECC_LOG_LEVEL=info cmake --build build
```

You should see an `Action Cache hit` line for the compile, like the
captured output below:

```text
[ 50%] Building CXX object CMakeFiles/hello.dir/main.cpp.o
[INFO] Action Cache hit for [cf20db769511312cf9c66cb3ac1b82512bf4572b42b99f9a73b13a00e90e14d3/199]
[100%] Linking CXX executable hello
[100%] Built target hello
```

That's NativeLink serving the compile result instead of the compiler
running again. Linking still happens locally. `recc` only ships compile
actions, not link actions, by default.

## Teardown [#teardown]

```sh
docker stop nativelink && docker rm nativelink
```

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Does recc offload my compiles to remote workers?" id="does-recc-offload-my-compiles-to-remote-workers">
    Not in this setup: `RECC_CACHE_ONLY=1` means misses compile locally
    and only the outputs travel through NativeLink. Drop that variable
    once you have remote workers configured and recc will dispatch
    compiles to them.
  </Accordion>
  <Accordion title="Why doesn't linking get cache hits?" id="why-doesn-t-linking-get-cache-hits">
    recc ships compile actions only by default; links always run locally.
    For remote link steps you want a full RBE client; see
    [Examples and templates](/remote-execution/examples-and-templates).
  </Accordion>
  <Accordion title="How do I share the cache with my team?" id="how-do-i-share-the-cache-with-my-team">
    Point `RECC_SERVER` at a shared NativeLink deployment and keep
    `RECC_INSTANCE` identical across machines. NativeLink rejects
    requests for an instance name it does not serve, so a mismatch shows
    up as recc errors on stderr and every compile running locally.
  </Accordion>
</Accordions>

## Going further [#going-further]

- Point `RECC_SERVER` at a shared NativeLink deployment to share the
  cache across your team.
- Drop `RECC_CACHE_ONLY` once you have remote workers configured to
  actually offload compile work. See
  [Remote execution](/remote-execution).
- For PCH, LTO, and other gnarly C++ build features at scale, read
  [Reid Kleckner's writeup](https://reidkleckner.dev/posts/llvm-recc-nativelink/).

<NextStep href="/getting-started/shared-cache" title="Share the cache">
  recc is caching compiles against one server. Next is making that server shared, which is where a compiler cache stops being a local optimisation.
</NextStep>

# https://docs.nativelink.com/getting-started/shared-cache

**Shared cache**: Self-host NativeLink on hardware you control, with the operational checklist a team rollout actually needs.

<Prerequisites>
  A cache working on your own machine. The [Quickstart](/getting-started/quickstart) gets you there.
</Prerequisites>

The source-available release of NativeLink is designed to run on your
own infrastructure. This page covers what changes between the
[10-minute quickstart](/getting-started/quickstart) and a deployment that
serves your team without 3 AM pages.

## When to self-host [#when-to-self-host]

Pick on-prem when one of these is true:

- **Data residency.** Build artifacts can carry source code,
  toolchains, even credentials. Self-hosting keeps them in whatever
  region or boundary you need (EU GDPR, US FedRAMP boundaries,
  air-gapped corp networks).
- **Specialised hardware.** GPU workers, ARM cross-compile fleets,
  in-house silicon. On-prem lets you use anything that can run a
  Linux binary.
- **You already operate stateful services.** If your team owns
  Kubernetes, Postgres, S3-compatible storage, adding NativeLink is
  marginal work.

If you'd rather not run it yourself, Enterprise offers a managed
deployment; see [Open source and Enterprise](/reference/oss-and-enterprise).

## What ships in the box [#what-ships-in-the-box]

A NativeLink deployment is composed of four roles. The same binary
serves all of them; the JSON5 config decides which subset to run.

| Role             | What it does                                              | Statefulness       |
| ---------------- | --------------------------------------------------------- | ------------------ |
| **CAS server**   | Stores and serves content-addressed blobs.                | Stateful           |
| **AC server**    | Maps `Action` digests to `ActionResult`s.                 | Stateful           |
| **Scheduler**    | Receives `Execute` calls, dispatches to workers.          | Queue in memory by default, or in Redis |
| **Worker**       | Runs the action in a per-action work directory, uploads outputs to CAS. | Stateless          |

A single binary can run all four. For anything beyond a single
developer, run them on separate processes so you can scale and restart
each independently.

## The rollout checklist [#the-rollout-checklist]

<Steps>
  <li>
    **Pick the storage backend.** The quickstart's `basic_cas.json5` keeps
    everything under `/tmp` on one machine, which is fine for a 10-minute
    demo and useless for a team. Pick one before rolling out:
    - **Filesystem**: single-node clusters, sub-100GB caches.
    - **Cloud object storage**: anything multi-node. The
      `experimental_cloud_object_store` has providers for S3, R2, GCS,
      Azure Blob, ONTAP S3 and OCI; see
      [S3 and compatible](/how-to/stores/s3-and-compatible).
    - **Redis**: hot-key acceleration in front of the durable store.
      Optional but cheap latency wins.
  </li>
  <li>
    **Pick the deployment substrate.** Most teams land on one of:
    - **Kubernetes**: Helm charts for Kubernetes ship with
      [NativeLink Enterprise](https://enterprise.nativelink.com); the `kubernetes/`
      directory in this repository holds only Kustomize building blocks used
      by its own examples.
    - **Bare VMs**: `systemd` units, one binary per role, with a load
      balancer in front. See [Deploy on bare metal](/operate/deploy-bare-metal).
    - **Docker Compose**: a reasonable starting point for a handful of
      developers. See [Deploy with Docker Compose](/operate/deploy-docker-compose).
  </li>
  <li>
    **Plan capacity.** Heuristics from production clusters:
    - CAS storage: 5 to 20 GB per active developer, depending on
      language. C++ skews high; Go skews low.
    - Worker CPU: 1 vCPU per concurrent action. Headroom matters more
      than peak.
    - Network: cache reads are the hot path. Provision at least 1 Gbps
      between workers and CAS.
  </li>
  <li>
    **Set up TLS.** TLS on every listener, and client certificates where
    the network is not trusted. The
    [Production configuration](/operate/production-config) page has the
    certificate layout and [TLS and auth](/how-to/tls-and-auth) the
    per-hop settings.
  </li>
  <li>
    **Wire up metrics.** NativeLink pushes OTLP metrics; there is no
    Prometheus scrape endpoint. Send them to an OpenTelemetry Collector
    and let Prometheus and Grafana read from there. See
    [Observability](/operate/observability).
  </li>
</Steps>

## Container registry [#container-registry]

Official images are published to GitHub Container Registry. Pull a
specific version tag; there is no `latest` tag to fall back on, and
pinning is what you want in production anyway.

```bash
docker pull ghcr.io/tracemachina/nativelink:v1.6.5
```

The [`/pkgs/container/nativelink`](https://github.com/TraceMachina/nativelink/pkgs/container/nativelink)
page lists every published tag.

## Backups & recovery [#backups--recovery]

The CAS is the only stateful piece you can't trivially rebuild from
clients. Snapshot strategy depends on the backend:

- **Filesystem**: `rsync` or your filesystem's snapshot facility
  (ZFS, Btrfs). Restore by stopping the CAS, swapping the directory,
  starting again.
- **S3-compatible**: versioning + lifecycle policies handle the
  primary copy. For DR, cross-region replication.
- **Redis**: treat as ephemeral. Loss is a cache miss, not data loss.

The Action Cache can be wiped without data loss; you'll re-execute
everything until it warms back up.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Is self-hosting free?" id="is-self-hosting-free">
    Yes, for permitted use. Most of the monorepo is
    `FSL-1.1-Apache-2.0`: internal use, modification, and redistribution
    for non-competing purposes are allowed. Metrics and remote persistent
    workers are Business Source License modules: fine for an individual
    developer's cache, but shared, production, or commercial use of those
    modules needs NativeLink Enterprise or an intentionally
    inexpensive commercial license. Meaningful contributors may be
    eligible for waivers; contact the maintainers before relying on one.
    [Open source and Enterprise](/reference/oss-and-enterprise) works through which files
    carry which licence and why nothing in the binary tells you; the operative
    text is the [license page](https://nativelink.com/license).
  </Accordion>
  <Accordion title="What am I actually paying for when I self-host?" id="what-am-i-actually-paying-for-when-i-self-host">
    Compute for the control plane (a few small VMs), CAS storage (the
    dominant cost in a healthy cluster), worker compute, and the engineer
    hours to run it all. Which of those dominates depends on your fleet and
    your build, and the honest answer is that you will not know the shape of
    the bill until you have run it for a quarter.
    [Open source and Enterprise](/reference/oss-and-enterprise) sets out what the
    managed Enterprise tier does instead of you.
  </Accordion>
  <Accordion title="Does the free version limit cache or team size?" id="does-the-free-version-limit-cache-or-team-size">
    No. Nothing artificially limits cache size, action count,
    worker count, or team size: the self-hosted build is the same
    codebase run in production, subject to each module's license, minus
    the operational tooling and support the paid tiers add.
  </Accordion>
</Accordions>

<NextStep href="/remote-execution" title="Run actions remotely">
  A cache your whole team hits is the prerequisite for the next section: having
  the actions themselves run somewhere other than a laptop.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  The JSON5 shape for a real cluster, and the settings that only start to
  matter under load.
</NextStep>

<NextStep href="/operate/deploy-bare-metal" kind="aside" title="Deploy on bare metal">
  One binary per role under systemd, if that is where this is going to live.
</NextStep>

# https://docs.nativelink.com/remote-execution

**Remote execution**: Workers that run the actions your cache misses.

**Who this is for:** anyone with a working cache who wants the misses to stop
running locally. **What you'll have at the end of this section:** a scheduler,
at least one worker, and a build whose cache misses execute on the farm
instead of on your machine. **Time:** an hour for the first remote action,
longer if your toolchain isn't hermetic yet.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

## What this section adds [#what-this-rung-adds]

A cache answers the question "has this exact action already been run?" When
the answer is no, something still has to run it, and with caching alone,
that something is your laptop. Remote execution changes the answer to that
second question.

Three things are new in the deployment: a **scheduler** that decides which
worker gets which action, at least one **worker** that actually runs it, and
the **execution service** on the server. One thing is new in your build
invocation: you tell the build tool where the executor is, and you declare
what kind of machine your actions need. One thing is deliberately *not* new:
your CAS and action cache are exactly the ones Getting started left you with. That's
why the sections are in this order.

[From cache to execution](/remote-execution/from-cache-to-execution) is the
long version of that paragraph, and it's the page to read first.

## What to expect the first time [#what-to-expect-the-first-time]

The first remote action almost never works on the first try, and it's nearly
always one of two walls.

**Platform properties.** The scheduler only hands an action to a worker whose
advertised properties satisfy what the action asked for. When they don't line
up, nothing errors: actions queue, indefinitely, which reads like a hang
rather than a misconfiguration.

**Toolchain hermeticity.** An action that runs on your machine because your
machine happens to have the right compiler will not run on a worker that
doesn't. Remote execution turns every implicit dependency on the local
environment into a failure, which is uncomfortable the first time and
extremely valuable afterwards.

Both walls get their own page here rather than a footnote, because everyone
hits them.

## The pages, in order [#the-pages-in-order]

**Understand the change**

- [From cache to execution](/remote-execution/from-cache-to-execution):
  what execution adds on top of a working cache, what's new in the
  deployment and the build invocation, and what stays the same.

**Get it running**

- [Your first remote action](/remote-execution/first-remote-action): a
  scheduler and one worker on your own machine, and a build you can watch
  execute on them.

**Get past the two walls**

- [Platform properties](/remote-execution/platform-properties): the
  matching contract, why a mismatch queues forever, and the scheduler's own
  diagnostics.
- [Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity):
  pinning the toolchain so local and remote actions hash identically.
- [Containers and images](/remote-execution/containers-and-images): the
  other answer to the same problem, and what `container-image` really does.

**Go further**

- [Persistent workers](/remote-execution/persistent-workers): pooled
  long-lived worker processes for toolchains with a startup tax.
- [Local testing](/remote-execution/local-testing): keeping a local cluster
  as a test fixture, in the two shapes the repo itself uses.
- [Examples and templates](/remote-execution/examples-and-templates):
  configs, deployment topologies, and project scaffolds to copy.

## Start here, or skip ahead [#start-here-or-skip-ahead]

**Start here** if you have a cache serving hits and haven't run anything
remotely yet.

**Go back to Getting started** if you don't have a cache serving hits yet.
[Getting started](/getting-started) is the precondition for every page in
this section, and pages here will fail in confusing ways without it.

**Skip ahead** to [Configuration](/configuration) if you already have
workers running and what you actually want is to understand the config file
you copy-pasted to get here.

**Jump straight to [Platform properties](/remote-execution/platform-properties)**
if you have workers and your build is stuck at `0s remote`. That's the page.

<NextStep href="/remote-execution/from-cache-to-execution" title="From cache to execution">
  What changes when the misses stop running on your laptop: in the
  deployment, in the build invocation, and in what breaks first.
</NextStep>

<NextStep href="/configuration" kind="aside" title="Configuration">
  Once actions are executing, the config file is the next thing worth
  actually understanding rather than copying.
</NextStep>

# https://docs.nativelink.com/remote-execution/from-cache-to-execution

**From cache to execution**: You have a cache serving hits. This is what changes when the misses stop running on your laptop: in the deployment, in the build invocation, and in what breaks first.

**Who this is for:** anyone who finished Getting started and is deciding whether to
take the next step. **What you'll have at the end:** an accurate picture of
what remote execution adds, what it costs, and which wall you'll hit first.
**Time:** ten minutes of reading, no commands.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

A cache answers exactly one question: *has this exact action already been
run somewhere?* When the answer is yes, you get the result back in
milliseconds. When the answer is no (and on a first build, or after a
toolchain bump, the answer is no for almost everything) something still has
to run the action. With caching alone, that something is the machine you're
sitting at.

Remote execution changes the answer to that second question. The misses go
to a fleet.

## What execution buys you [#what-execution-buys-you]

Two things, and it's worth being clear that they're separate.

**The misses stop being your problem.** A cache makes repeated work free; it
does nothing for work nobody has done yet. A clean build, a dependency bump,
a merge that invalidates half the graph: these are all cache misses by
definition, and they are exactly the builds that hurt. Remote execution is
what turns those into a fan-out across as many cores as you're willing to
pay for, instead of a fan-out across the eight your laptop has.

**Actions get to run on hardware that fits them.** Once a scheduler is
routing work, actions stop being homogeneous. A link step that needs 64 GB
of RAM, a test that needs a GPU, a cross-compile that needs a specific libc:
each can be routed to a worker that has it, without every developer
needing that machine under their desk. This is the part that has no local
equivalent at all, and for some teams it's the whole reason to adopt
NativeLink.

A third thing people expect and don't get: remote execution is not
automatically *faster* per action. A single action that takes 200 ms locally
will take 200 ms plus a network round-trip plus input materialization
remotely. The win is parallelism and placement, not per-action latency,
which is why [Classic patterns](/remote-execution/examples-and-templates)
covers keeping small actions local.

## What's new in the deployment [#whats-new-in-the-deployment]

Three components, and one of them is a security boundary.

<Mermaid>
{`flowchart LR
    C[your build tool]
    S[scheduler]
    W[worker]
    CAS[("CAS + Action Cache\\n(already yours from Getting started)")]

    C -->|"Execute()"| S
    C -->|"read / write blobs"| CAS
    S -->|"worker API :50061"| W
    W -->|"fetch inputs\\nupload outputs"| CAS
    S -.->|reads action digests| CAS`}
</Mermaid>

**A scheduler.** It owns the queue and the matching engine. It holds no
blobs of its own; it reads the CAS through a `grpc` store pointed at the
CAS process, and its whole job is deciding which worker gets which action.

**At least one worker.** It connects *outward* to the scheduler, announces
what it can do, and then pulls actions. Workers do not listen for client
traffic; a worker config has an empty `servers` array.

**The `execution` service on a server.** This is the client-facing half:
the `Execute` and `WaitExecution` RPCs your build tool calls. It sits
alongside the `cas` and `ac` services you already have, and it needs a
`scheduler` reference.

And one port that is genuinely new in a way that matters: the **worker API**
listener, `50061` in every example config in the repo that has one. Workers register
there. It is a backend port with a completely different permission set from
the client port, and it must not be reachable from where your clients are.
[Production configuration](/operate/production-config#which-ports-are-public)
is blunt about why.

<Callout type="warn" title="The worker API is not a client port">
  Anything that can reach the worker API can register as a worker and
  receive actions to execute, which means arbitrary commands, from your
  build, running on its machine, with your CAS credentials. Bind it to a
  private network. The repo's own example config says so in a comment.
  <SourceLink file="nativelink-config/examples/local_rbe_self_test.json5" />
</Callout>

## What's new in the build invocation [#whats-new-in-the-build-invocation]

Less than you'd think. Two flags and a declaration.

```bash
bazel test \
  --remote_cache=grpc://nativelink.internal:50051 \
  --remote_executor=grpc://nativelink.internal:50052 \
  --remote_default_exec_properties=cpu_count=1 \
  //your:target
```

`--remote_executor` is the new one. `--remote_cache` you already had. If
you leave it out, Bazel reuses the `--remote_executor` endpoint as the
cache, which is fine when one listener serves both; keep it explicit when
the cache and the executor are different endpoints, which is the usual
production shape. Whichever store the `execution` service reads must be
the store the client uploads to, or `Execute` fails with
`FAILED_PRECONDITION: Action ... is missing from CAS`.

`--remote_default_exec_properties` is the declaration. It's how your build
tells the scheduler what kind of machine the action needs. Getting this
wrong is the single most common way a first remote build fails, which is why
[Platform properties](/remote-execution/platform-properties) is its own
page rather than a paragraph here.

## What stays exactly the same [#what-stays-exactly-the-same]

Your CAS and your action cache. The same stores, the same digests, the same
hits you were already getting.

This is not a coincidence; it's why the sections are in this order. Remote
execution is a *consumer* of the cache you built in Getting started, not a
replacement for it. The worker fetches an action's inputs from your CAS by
digest and uploads its outputs back to the same place, so every result a
worker produces is immediately a cache hit for the next person who asks for
it. If your cache was serving hits before you added a scheduler, it is
serving the same hits afterwards, plus new ones.

The practical consequence: **if something breaks after you turn on remote
execution, the cache is almost never the thing that broke.** You can drop
`--remote_executor`, confirm your build still works cache-only, and know the
problem is in the two components you just added.

## What to expect the first time [#what-to-expect-the-first-time]

The first remote action rarely works on the first attempt, and it is nearly
always one of two walls. Both are ordinary; both have a page.

**Wall one: platform properties.** The scheduler will only hand an action to
a worker whose advertised properties satisfy what the action asked for. When
they don't line up, nothing errors: the action sits in the queue
indefinitely, which reads like a hang rather than a misconfiguration. Your
build appears to stall at "0s remote" and stays there.
[Platform properties](/remote-execution/platform-properties) covers the
matching rules and how to read the scheduler's own diagnostic logging.

**Wall two: toolchain hermeticity.** An action that succeeds on your machine
because your machine happens to have the right compiler will fail on a
worker that doesn't. Remote execution converts every unstated dependency on
your local environment into a hard failure. This is uncomfortable the first
time and extremely valuable afterwards; it's the same property that makes
builds reproducible.
[Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
is the fix.

A reliable way to keep these two walls from arriving together: run
the whole cluster on your own machine first, with whatever toolchain is
already on your `PATH`. That isolates the protocol wiring from the toolchain
question, and it's what [Your first remote action](/remote-execution/first-remote-action)
does next.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Do I have to run the scheduler and CAS as separate processes?" id="do-i-have-to-run-the-scheduler-and-cas-as-separate-processes">
    No. One `nativelink` process can host the CAS, the action cache, a
    scheduler, and a worker at once. That's exactly what
    <SourceLink file="nativelink-config/examples/local_rbe_self_test.json5" />
    does, and it's the right shape for learning. The split into separate
    processes is an operational decision you make when you want to scale or
    restart the pieces independently, not a protocol requirement. See
    [Production configuration](/operate/production-config) for the shape a
    real cluster uses.
  </Accordion>
  <Accordion title="Can I add execution to a cache that's already serving my team?" id="can-i-add-execution-to-a-cache-thats-already-serving-my-team">
    Yes, and it's additive: the scheduler and workers read and write the CAS
    you already have. The change on the server side is a `schedulers` entry,
    an `execution` service (plus a `capabilities` service with
    `remote_execution` set, so clients learn execution is available) on a
    client-facing listener, and a `worker_api`
    listener on a private port. Nothing about the existing `cas`, `ac`, or
    `bytestream` services changes, so clients that only pass
    `--remote_cache` keep working exactly as before.
  </Accordion>
  <Accordion title="What actually crosses the wire when an action runs remotely?" id="what-actually-crosses-the-wire-when-an-action-runs-remotely">
    The action: input file digests, the command line, the environment, and
    the platform properties. Not the input files themselves: the worker
    fetches those from the CAS by hash, and skips anything it already has.
    Outputs go back the same way. This is why a fat input tree is cheap to
    re-run and a cold worker is expensive to start.
  </Accordion>
  <Accordion title="How many workers do I need to start?" id="how-many-workers-do-i-need-to-start">
    One. A single worker proves the whole path end to end, and the number of
    actions it will run concurrently is governed by its `cpu_count` property
    (and how much of it each action requests) rather than by worker count. Add workers when the queue has depth, not
    before. [Tuning](/operate/tuning) has the signal to watch.
  </Accordion>
</Accordions>

<NextStep href="/remote-execution/first-remote-action" title="Your first remote action">
  A scheduler and one worker on your own machine, and a build you can watch
  execute on them.
</NextStep>

<NextStep href="/remote-execution/platform-properties" kind="aside" title="Platform properties">
  If you already have workers and your actions are queueing forever, skip
  ahead: this is almost certainly why.
</NextStep>

# https://docs.nativelink.com/remote-execution/first-remote-action

**Your first remote action**: Run a scheduler and a worker on your own machine, point Bazel at them, and watch a build action execute somewhere other than where you typed the command.

**Who this is for:** anyone with a working cache who wants to see remote
execution actually run. **What you'll have at the end:** a build whose
actions Bazel reports as `remote`, running against a scheduler and worker
you started yourself. **Time:** about fifteen minutes.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

The goal here is deliberately small: prove the *protocol* works before you
touch anything else. Everything runs on your machine, in one process, using
the toolchain already on your `PATH`. No containers to build, no fleet to
provision, no hermeticity questions yet; those come next, and they're much
easier to debug once you know the wiring is sound.

<Callout type="info" title="What you need">
  A Bazel workspace you can build (any one will do; the NativeLink repo
  itself works), and either Docker or Nix. Up to about 4 GB of free disk
  under `/tmp` for the caches this creates (the three filesystem stores in
  the config cap at 2 GB, 200 MB, and 2 GB).
</Callout>

## Get the config [#get-the-config]

The repo ships a config that is exactly this tutorial: CAS, action cache,
scheduler, and one worker, all in a single process on localhost.

```bash
curl -O https://raw.githubusercontent.com/TraceMachina/nativelink/refs/tags/v1.6.5/nativelink-config/examples/local_rbe_self_test.json5
```

<SourceLink file="nativelink-config/examples/local_rbe_self_test.json5" />

It is worth reading before you run it: it is short, and every section of it
maps onto something you already know or are about to learn.

<Steps>

<li>

### Start the cluster [#start-the-cluster]

```bash
nix run github:TraceMachina/nativelink ./local_rbe_self_test.json5
```

Or, if you'd rather use the container image, mount the config in and publish
both ports:

```bash
docker run \
  -v $(pwd)/local_rbe_self_test.json5:/config \
  -p 50051:50051 -p 50061:50061 \
  ghcr.io/tracemachina/nativelink:v1.6.5 config
```

Leave it running. You'll want its output visible; the scheduler logs here
are the primary diagnostic for the rest of this page.

</li>

<li>

### Confirm the worker registered [#confirm-the-worker-registered]

The worker connects *outward* to the scheduler on `50061` a moment after
startup. In the server's log you should see it announce itself before you
run anything, at the default `INFO` level:

```plaintext
Worker registered with scheduler worker_id=...
```

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="LocalWorker::run" />

If instead the log repeats `Error` with a connection failure every half
second, the worker cannot reach the `worker_api` listener; check that the
`servers` block in the config you downloaded still has the `50061` listener.
NativeLink does not serve gRPC reflection, so `grpcurl ... list` is not a
way to check which services are up; the config file is.

</li>

<li>

### Point a build at it [#point-a-build-at-it]

From any Bazel workspace:

```bash
bazel test \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  //your:target
```

Both flags point at the same port here, because this config puts the cache
and the execution service on the same listener. In a real deployment they're
usually different processes and different ports; the flags don't change.

`--remote_default_exec_properties=cpu_count=1` is worth passing even though
this config would accept an action without it. The scheduler here declares
`cpu_count` as a `minimum` property and the worker advertises exactly `1`;
an action that asks for nothing matches any worker and consumes none of
that capacity, which happens to work here, but declaring it explicitly is
the habit that will save you on the next page.

</li>

<li>

### Read the result [#read-the-result]

Ask Bazel where each action ran rather than trusting the wall-clock number:

```bash
bazel test \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  --nocache_test_results \
  --build_event_json_file=/tmp/bep.json \
  //your:target

jq -r 'select(.id.testResult) | .testResult.executionInfo.strategy' /tmp/bep.json
```

```plaintext
remote
```

That single word is the whole tutorial. `--nocache_test_results` forces the
action to actually execute instead of being served from the action cache,
which is what you want the first time; otherwise a cache hit will look
like a success and tell you nothing about whether execution works.

</li>

</Steps>

<VerifyBlock>

- The server's log shows `Worker registered with scheduler` before you run
  the build.
- The build event log reports `strategy` as `remote`, not `local` or
  `worker`.
- The server's log shows `Executing command` from the in-process worker, so
  the action left the queue rather than sitting in it.
- Running the same build again *without* `--nocache_test_results` completes
  near-instantly: the worker's output went into your action cache, so the
  second run is a cache hit.

</VerifyBlock>

That last one is the point of the whole reading path in miniature: remote execution
doesn't replace your cache, it *fills* it. Every action a worker completes
becomes a hit for whoever asks next.

## What just happened [#what-just-happened]

Four processes' worth of work happened inside one:

1. Bazel uploaded the action's inputs to the **CAS** and asked the
   **execution service** to run it.
2. The execution service handed the action to the **scheduler**, which
   compared what the action asked for (`cpu_count=1`) against what the
   registered worker advertised (`cpu_count: 1`) and found a match.
3. The **worker** fetched the inputs from the CAS by digest, hardlinked them
   into a sandbox under its `work_directory`, and ran the command.
4. The worker uploaded the outputs back to the CAS and the result to the
   **action cache**, and the scheduler streamed completion back to Bazel.

Everything after this is that same loop with more machines in it.

## When it doesn't work [#when-it-doesnt-work]

<Accordions type="multiple">

<Accordion title="The build hangs at &quot;0s remote&quot; and never progresses" id="the-build-hangs-at-0s-remote-and-never-progresses">
Your action's platform properties don't match any registered worker, so the
scheduler is holding it in the queue. This is the single most common
first-run failure and it is silent by design: the scheduler keeps waiting
in case a suitable worker shows up later.

Check the server log for the scheduler's periodic match logging: `Property
mismatch`, `No candidate workers due to a lack of key`, or `No workers
available to match!` (that last one means the worker never registered).
Either way,
[Platform properties](/remote-execution/platform-properties) walks through
reading the scheduler's own matching diagnostics.
</Accordion>

<Accordion title="FAILED_PRECONDITION: Action ... is missing from CAS" id="failed-precondition-action-is-missing-from-cas">
The execution service looked up the action digest in its `cas_store` and
did not find it, so the store Bazel uploaded to is not the store the
`execution` service reads. The usual causes are `--remote_cache` and
`--remote_executor` pointing at different deployments, or an `execution`
entry whose `cas_store` is not the same store (or a `ref_store` to it) as
the `cas` and `bytestream` services on the client listener. NativeLink
returns the error with a `PreconditionFailure` detail so Bazel can re-upload
the missing blobs and retry; if it keeps repeating, the two stores are not
the same store.
<SourceLink file="nativelink-service/src/execution_server.rs" symbol="missing_blobs_failed_precondition" />
</Accordion>

<Accordion title="The action runs but fails with &quot;command not found&quot; or a missing header" id="the-action-runs-but-fails-with-command-not-found">
The worker executed your action in a sandbox that doesn't have something
your local environment provides implicitly. This is remote execution doing
its job: it converted an unstated dependency into a visible failure. It's
also out of scope for this page:
[Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
is the fix.

Note that on this config the worker runs as *you*, on *your* machine, with
your `PATH`. If a command isn't found even here, the action is depending on
something Bazel isn't declaring at all.
</Accordion>

<Accordion title="Bazel reports strategy as &quot;local&quot; even though the flags are set" id="bazel-reports-strategy-as-local">
Some action types are excluded from remote execution by the workspace's own
configuration; check for `--strategy` or `--spawn_strategy` settings in the
workspace's `.bazelrc`, and for `no-remote` tags on the target. Try a
different target before concluding the cluster is at fault.
</Accordion>

<Accordion title="Permission or namespace errors on startup" id="permission-or-namespace-errors-on-startup">
This config leaves `use_namespaces` and `use_mount_namespace` at their
`false` defaults precisely so it runs unprivileged and on macOS. If you
enabled them, you now need Linux and the corresponding capabilities. Turn
them back off for this tutorial.
</Accordion>

</Accordions>

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Why does this run the worker in the same process as the CAS?" id="why-does-this-run-the-worker-in-the-same-process-as-the-cas">
Because it's the fewest moving parts that still exercises the real code
paths: the worker here uses the same gRPC worker API, the same scheduler,
and the same store implementations a production cluster does. Splitting into
separate processes changes the deployment, not the protocol. See
[Production configuration](/operate/production-config) for the split
version.
</Accordion>

<Accordion title="Can I keep using this for day-to-day local builds?" id="can-i-keep-using-this-for-day-to-day-local-builds">
You can, and some people do; a single-process cluster is a reasonable way
to get sandboxed, cached builds on one machine. But it gives you no
parallelism beyond one worker's `cpu_count`, and the filesystem stores are
pointed at `/tmp`. [Local testing](/remote-execution/local-testing) covers
the longer-lived version of this setup.
</Accordion>

<Accordion title="What if I'm not using Bazel?" id="what-if-im-not-using-bazel">
The cluster is build-tool agnostic: it speaks the Remote Execution API, and
Buck2, Siso, Pants, and recc all connect to the same endpoints.
[Connect your build](/getting-started/connect-your-build) has the flag
equivalents. The verification step differs per tool; look for whatever your
tool calls its execution strategy in its own logs.
</Accordion>

<Accordion title="Where did my data go, and how do I clean it up?" id="where-did-my-data-go-and-how-do-i-clean-it-up">
Everything this config writes lives under
`/tmp/nativelink-local-rbe-test/`: CAS content, action cache, the worker's
fast store, and its work directory. Stop the server and delete that tree to
start clean.
</Accordion>

</Accordions>

<NextStep href="/remote-execution/platform-properties" title="Platform properties">
  How the scheduler decides which worker gets an action, and why a
  mismatch queues forever instead of erroring.
</NextStep>

<NextStep href="/remote-execution/toolchains-and-hermeticity" kind="aside" title="Toolchains and hermeticity">
  If your action ran but failed on a missing compiler, this is the page you
  want.
</NextStep>

# https://docs.nativelink.com/remote-execution/platform-properties

**Platform properties**: How the scheduler decides which worker gets an action, why a mismatch queues forever instead of failing, and how to read the matching diagnostics it already emits.

**Who this is for:** anyone whose actions are queueing, and anyone about to
run more than one kind of worker. **What you'll have at the end:** an
accurate model of the matching rules, plus the log lines that tell you which
rule you tripped. **Time:** fifteen minutes.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

Platform properties are the only vocabulary a build tool and a worker fleet
share. The action says *I need this*; the worker says *I have this*; the
scheduler decides whether those two statements are compatible. No
type checking happens across that boundary beyond what you configure, and no
negotiation, which is why a typo produces a build that waits rather than a
build that fails.

## The three places a property appears [#the-three-places-a-property-appears]

A property key has to be declared in all three, and they have to agree.

<Mermaid>
{`flowchart TD
    A["action<br/>--remote_default_exec_properties=cpu_count=1"]
    S["scheduler<br/>supported_platform_properties: { cpu_count: minimum }"]
    W["worker<br/>platform_properties: { cpu_count: { values: [16] } }"]

    A -->|"Execute()"| S
    W -->|"connect_worker()"| S
    S -->|"assign if satisfied"| W`}
</Mermaid>

The scheduler's `supported_platform_properties` is the schema. It maps each
key to a **type**, and that type decides how the value on the action is
compared against the value on the worker. A key that isn't in this map is
not a valid key anywhere, for either side.

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="PropertyType" />

## The four property types [#the-four-property-types]

| Type | What the action's value means | What it takes to match |
| --- | --- | --- |
| `minimum` | A numeric floor. Parsed as `u64`. | The worker's value must be **≥** the action's. |
| `exact` | A literal string. | The worker's value must be **byte-for-byte identical**. |
| `priority` | Informational; passed through to the worker. | Any value matches, but the worker must still **have the key**. |
| `ignore` | Requestable but unenforced. | Always matches, and the worker need not have the key at all. |

Most documentation you'll find elsewhere describes three of these. `ignore`
exists and is the only one that exempts a worker from carrying the key.

<SourceLink file="nativelink-util/src/platform_properties.rs" symbol="is_satisfied_by" />

### Why `priority` is not the escape hatch people assume [#why-priority-is-not-the-escape-hatch-people-assume]

The matching loop checks key presence *before* it checks value
compatibility. For every property the action requests, the scheduler looks
the key up on the worker; if it isn't there, the worker is rejected,
regardless of type. `priority` only tells the scheduler to stop caring about
the *value*.

A worker that omits `OSFamily` entirely will never receive an action that
requests `OSFamily`, even though the scheduler declares `OSFamily` as
`priority`. This is why the example configs advertise `OSFamily: { values:
[""] }`: an empty string is still a value, and presence is what's being
tested. Only `ignore` skips the presence check.

### `minimum` is a consumable resource, not a filter [#minimum-is-a-consumable-resource-not-a-filter]

This is the most consequential and least visible behaviour in the system.

When the scheduler assigns an action to a worker, it **subtracts** every
`minimum` value the action requested from the worker's advertised value. When
the action completes, it adds them back.

<SourceLink file="nativelink-scheduler/src/worker.rs" symbol="reduce_platform_properties" />

A worker advertising `cpu_count: 16` that receives four actions each asking
for `cpu_count=4` is, at that moment, advertising `cpu_count: 0`, and will
match nothing further until one of them finishes. `minimum` properties are
therefore your real concurrency control, and `cpu_count` in particular is the
knob that governs how many actions a worker runs at once.

Two consequences worth internalising:

- An action requesting `cpu_count=32` against workers that advertise `16`
  will never be scheduled. Not slowly; never. No bin-packing across
  workers happens, and no error is raised.
- Setting `cpu_count=1` on every action, as the getting-started examples do,
  means a 16-core worker runs 16 concurrent actions. That's usually what you
  want; it is a decision you're making either way.

`max_inflight_tasks` on the worker is the separate, blunter cap: it counts
tasks rather than resources, and `0` means unlimited.

## What happens when it doesn't match [#what-happens-when-it-doesnt-match]

The two failure modes are asymmetric, and knowing which one you're in tells
you where to look.

**An unknown key on a worker is a hard failure at registration.** Worker
properties go through the same type map as action properties, so a worker
advertising a key that isn't in `supported_platform_properties` is rejected
with `Bad Property during connect_worker()` and never joins the pool. You'll
see this in the worker's log at startup, and the effect is a fleet that looks
smaller than you provisioned.

<SourceLink file="nativelink-service/src/worker_api_server.rs" symbol="connect_worker" />

**An unknown key on an action queues forever.** The scheduler types an
action's properties at match time, not at submission time. When that typing
fails, the scheduler logs `Error while running do_try_match` (wrapping
`Unknown platform property 'x'`), moves on to the other queued actions, and
leaves this one where it is; the action is not errored out and the client
is not told anything. Because the pass ended in an error, the scheduler
re-runs it after 100 ms, failing identically each time. From the build's
side this is indistinguishable from a busy queue.

<SourceLink file="nativelink-scheduler/src/simple_scheduler.rs" symbol="do_try_match" />

<Callout type="warn" title="A queued build is a configuration bug until proven otherwise">
  If a build sits at `0s remote` for more than a few seconds on an idle
  cluster, do not wait it out. The scheduler is not going to time out and
  tell you. Read the match logging described below; it is on by default.
</Callout>

## Reading the scheduler's own diagnostics [#reading-the-schedulers-own-diagnostics]

The scheduler has a built-in periodic dump of exactly this. It's on by
default at a ten-second interval:

```json5
schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: {
        cpu_count: "minimum",
        OSFamily: "priority",
        "container-image": "priority",
      },
      // Seconds between full match-attempt dumps. -1 or 0 disables.
      worker_match_logging_interval_s: 10,
    },
  },
]
```

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="SimpleSpec" />

On each interval it logs, for every queued action, why no worker took it,
plus the oldest five actions in each state (`Oldest actions in state`).
Matching happens in two passes, and each pass has its own lines. First a
capability index narrows the fleet by key presence and by `exact` value:

```plaintext
No candidate workers due to a lack of key 'container-image'. Job asked for Priority("")
No candidate workers due to a lack of matching 'gpu' = Exact("a100"). Workers have: [Exact("none")]
No workers in capability index match required properties
```

Then the surviving candidates are checked for free capacity, which is where
`minimum` values are compared:

```plaintext
Property mismatch on worker property cpu_count. Minimum(8) < Minimum(32)
Worker <id> cannot accept work: is_paused=false, is_draining=false, inflight=4/4
```

<SourceLink file="nativelink-scheduler/src/worker_capability_index.rs" symbol="find_matching_workers" />

Read those lines literally, because they mean different things:

- **`lack of key`**: no worker advertises the key at all (for a `priority`
  or `minimum` property). Add it to the worker's `platform_properties` (an
  empty string is fine for `priority`), or change the scheduler's
  declaration to `ignore`.
- **`lack of matching`**: for an `exact` property, no worker carries that
  exact value; the line lists the values workers do have, so check for
  trailing whitespace and case. A worker that omits the key entirely shows
  up here too, with an empty list.
- **`Property mismatch`**: the worker has the key, but its remaining
  `minimum` capacity (on the left) is below what the action asks for.
- **`All workers are fully allocated`** or **`cannot accept work`**: the
  properties are fine; every worker is paused, draining, or at its
  `max_inflight_tasks` cap.

None of these appearing at all, while actions queue, points at typing rather
than matching; look for `Unknown platform property` in the same log, which
is the unknown-key-on-an-action case above.

The example config
<SourceLink file="nativelink-config/examples/scheduler_match_logging_disable.json5" />
shows the opposite setting for clusters where this output is too noisy in
steady state.

## A worked example: routing to a GPU pool [#a-worked-example-routing-to-a-gpu-pool]

Say you have two worker pools and want tests tagged for GPU to land only on
the GPU machines.

**Scheduler**: declare the key and its type once:

```json5
supported_platform_properties: {
  cpu_count: "minimum",
  OSFamily: "priority",
  gpu: "exact",
}
```

**CPU workers**: they can omit the key entirely; an `exact` key a worker
doesn't carry never matches an action that requests it, and actions
that don't request `gpu` are unaffected either way. Advertising a value no
action asks for behaves the same and keeps the intent visible in the config:

```json5
platform_properties: {
  cpu_count: { query_cmd: "nproc" },
  OSFamily: { values: [""] },
  gpu: { values: ["none"] },
}
```

**GPU workers:**

```json5
platform_properties: {
  cpu_count: { query_cmd: "nproc" },
  OSFamily: { values: [""] },
  gpu: { values: ["a100"] },
}
```

**The build**: request it per target rather than globally:

```python
# BUILD.bazel
cc_test(
    name = "kernel_test",
    exec_properties = {"gpu": "a100"},
    ...
)
```

Targets that say nothing about `gpu` still match every worker, because the
scheduler only checks properties the *action* requested. The asymmetry is
deliberate: adding a capability to your fleet doesn't invalidate the actions
that don't need it.

Note `query_cmd` above: a worker property can be computed at startup by
running a command (not a shell; the string is split into words shell-style
with `shlex` and executed directly, with a cleared environment) and
splitting its output on newlines, one value per line. `nproc` is the
canonical use.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="WorkerProperty" />
<SourceLink file="nativelink-worker/src/worker_utils.rs" symbol="make_connect_worker_request" />

## About `container-image` [#about-container-image]

It appears in nearly every example config and it is worth being precise about
what it does, because the name suggests something that isn't happening:
**NativeLink does not pull or launch a container per action.** No
code sits behind this key. It is an ordinary string property like any other.

What it actually means is *this worker process is running inside this image*,
which the worker asserts, and the scheduler matches on. The value has to be
byte-for-byte what your build requests, or actions queue forever in the usual
silent way. [Containers and images](/remote-execution/containers-and-images)
covers using it correctly.

## Troubleshooting [#troubleshooting]

| Symptom | Likely cause | Fix |
| --- | --- | --- |
| Build sits at `0s remote`, no errors anywhere | No worker satisfies the action's properties | Read the match logging (on by default every 10 s; `worker_match_logging_interval_s`) |
| `Unknown platform property 'x'` repeating in scheduler log | Action requests a key the scheduler doesn't declare | Add `x` to `supported_platform_properties`, or remove it from the build |
| Fewer workers in the pool than you started | Worker advertises a key the scheduler doesn't declare | Look for `Bad Property during connect_worker()` in the *worker's* log |
| `Property mismatch ... Minimum(0) < Minimum(1)` | Worker is fully allocated, not misconfigured | This is backpressure; add workers or raise `cpu_count` |
| Works on one worker, queues on another | The pools advertise different key sets | Every worker needs every key the actions request |
| `No candidate workers due to a lack of key 'OSFamily'` | Worker omits a `priority` or `minimum` key | Add it with an empty value, or make it `ignore` |
| `No candidate workers due to a lack of matching 'x' = Exact(...)` | No worker carries that exact value | Compare against the `Workers have:` list in the same line |

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Should I use exact or priority for OSFamily?" id="should-i-use-exact-or-priority-for-osfamily">
`priority` if you have one OS in the fleet, `exact` the moment you have two.
The example configs use `priority` with an empty value because a homogeneous
fleet gains nothing from enforcing it, but that's a default for a tutorial,
not a recommendation for a mixed fleet. With `exact` you get a real routing
guarantee; with `priority` you get a label.
</Accordion>

<Accordion title="Can an action request a property the scheduler declares but no worker has?" id="can-an-action-request-a-property-no-worker-has">
Yes, and it will queue indefinitely. Declaring a key in
`supported_platform_properties` makes it *legal*, not *satisfiable*. This is
the usual shape of the mistake when a new capability is half-rolled-out:
builds start requesting it before the fleet advertises it.
</Accordion>

<Accordion title="How do I set properties per target instead of for the whole build?" id="how-do-i-set-properties-per-target">
`exec_properties` on the target, or on a platform definition the target
resolves to. `--remote_default_exec_properties` is a fallback that applies
only where nothing more specific was set, which makes it right for
`cpu_count` and wrong for anything that varies.
</Accordion>

<Accordion title="Does the worker enforce what it advertises?" id="does-the-worker-enforce-what-it-advertises">
No. `cpu_count: 16` is an assertion the scheduler believes; nothing pins the
action to 16 cores or stops it using 64. Properties are for *routing* and
for accounting, not for isolation. If you need enforcement, that's the
worker's sandboxing configuration, not this.
</Accordion>

<Accordion title="Why not just error out an action that can't be matched?" id="why-not-just-error-out-an-unmatched-action">
Because a worker fleet is dynamic: the machine that satisfies an action may
be thirty seconds from registering. Holding the action is the right default
for autoscaling clusters, and it's why the diagnostic logging exists as the
counterweight. In practice, treat "queued longer than your autoscaler's warm-up
time" as a failure and alert on it. [Tuning](/operate/tuning) has the
queue-depth signals.
</Accordion>

</Accordions>

<NextStep href="/remote-execution/toolchains-and-hermeticity" title="Toolchains and hermeticity">
  The other wall: making sure the action a worker receives can actually run
  on it.
</NextStep>

<NextStep href="/remote-execution/containers-and-images" kind="aside" title="Containers and images">
  What `container-image` really does, and how to keep worker images and
  build requests in sync.
</NextStep>

# https://docs.nativelink.com/remote-execution/toolchains-and-hermeticity

**Toolchains and hermeticity**: Your action ran on a worker and failed on a missing compiler. Here is why that happens, and how to pin the toolchain so local and remote builds hash identically.

**Who this is for:** anyone whose actions execute remotely but fail on the
worker for reasons that never happen locally. **What you'll have at the end:**
a Nix-pinned toolchain that produces the same action digests on your machine
and on a worker. **Time:** thirty minutes for the walkthrough.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

## The problem [#the-problem]

An action is a closed description of some work: input digests, a command
line, an environment, and platform properties. If the compiler your command
invokes isn't in that description, the action is a lie: it works on your
machine because your machine happens to have the compiler, and it fails on a
worker that doesn't.

This is not a NativeLink behaviour. It is what remote execution *is*: the
protocol will faithfully reproduce whatever you described, and no more.
Caching lets you get away with an under-described action because the only
machine that ever ran it was yours. Execution doesn't.

Two ways out exist.

1. **Make the environment part of the image.** Bake the toolchain into the
   worker's container, and route actions to it with the `container-image`
   property. Minimal, and it's what most fleets do first.
   [Containers and images](/remote-execution/containers-and-images) covers
   this path.
2. **Make the toolchain part of the action.** Pin every tool at a
   content-addressed path so the toolchain itself is an input the CAS knows
   about. This is Local Remote Execution, and it's the rest of this page.

The second is strictly stronger: it gets you identical action digests
locally and remotely, which means a local build and a remote build share
cache entries instead of each maintaining their own.
[What LRE is and why](/explanations/lre) is the conceptual write-up; this
page is the mechanism.

<Callout type="info" title="You don't need LRE to use remote execution">
  A worker image with your toolchain in it is a legitimate answer, and it's
  the right one if you don't already use Nix. Read the problem statement
  above, then decide which of the two paths you're on.
</Callout>

## What ships in `@local-remote-execution` [#what-ships-in-local-remote-execution]

Three toolchain families, and they do not behave the same way once you leave
`x86_64-linux`.

| Family | Bazel target prefix | Platforms with a native config |
| --- | --- | --- |
| Rust | `@local-remote-execution//rust/...` | `aarch64-darwin`, `aarch64-linux`, `x86_64-darwin`, `x86_64-linux` |
| C++ | `@local-remote-execution//generated-cc/...` | `x86_64-linux` only (container `nativelink-worker-lre-cc`) |
| Java | `@local-remote-execution//generated-java/...` | `x86_64-linux` only (container `nativelink-worker-lre-java`) |

<SourceLink file="local-remote-execution/README.md" />
<SourceLink file="local-remote-execution/flake-module.nix" />

The flake module always emits the Rust platform and toolchain flags, and adds
the C++ ones only on Linux, where the generated config is `x86_64` and
carries a `container-image` property. On an `x86_64-linux` host that means
C++ actions can run locally out of the same `/nix/store` paths the worker
image has, or remotely on that image, with the same digest either way. On
macOS there is no local C++ or Java toolchain at all, so those actions can
only execute on a remote worker running the matching image, and even a
Nix-pinned toolchain costs a network round-trip: **LRE gives you
hermeticity, not offline-ness.**

<Callout type="warn" title="This is the single most common surprise">
  Reading "local remote execution" as "everything runs locally" leads to a
  confusing first hour on macOS. What is local is the *toolchain definition*.
  Whether the action executes locally depends on which family it uses.
</Callout>

## Set it up [#set-it-up]

<Steps>

<li>

### Install Nix with flakes enabled [#install-nix-with-flakes-enabled]

The [next-gen installer](https://github.com/NixOS/experimental-nix-installer)
is the shortest path. If you already have Nix, make sure
`experimental-features = nix-command flakes` is in your `nix.conf`.

The pinning *is* the guarantee here: every tool lives at a
content-addressed `/nix/store` path, and that path is what makes action
hashes match across machines. No non-Nix version of this mechanism exists.

</li>

<li>

### Pull the template [#pull-the-template]

```bash
mkdir my-lre-test && cd my-lre-test
nix flake init -t github:TraceMachina/nativelink#bazel
```

This writes a `flake.nix` importing `nativelink.flakeModules.lre` and
pinning `lre = { inherit (pkgs.lre.lre-cc.meta) Env; }`, plus a
`hello-world.cpp` example, its `BUILD.bazel`, and `platforms/BUILD.bazel`.

`bazel` is the only template the flake ships;
[Examples and templates](/remote-execution/examples-and-templates) has the
full list and what to do if you want a different starting point.

</li>

<li>

### Initialise git before you enter the shell [#initialise-git-before-you-enter-the-shell]

```bash
git init && git add -A
```

This step is not optional and it fails quietly.

<Callout type="warn" title="Skip this and lre.bazelrc is silently never written">
  The flake module's install script checks for a `.git` directory before
  generating anything. In a plain non-git folder, `nix develop` prints this
  and carries on without writing `lre.bazelrc` (no error, no toolchain):

  ```plaintext
  WARNING: lre: .git not found; skipping installation.
  ```
</Callout>

</li>

<li>

### Enter the dev shell [#enter-the-dev-shell]

```bash
nix develop
```

This fetches the Nix-pinned toolchain and writes `lre.bazelrc` (a symlink
into the Nix store). On `aarch64-darwin` the generated file looks like this,
abbreviated; your store paths will differ:

```bash
# These flags are dynamically generated by the lre flake module.
#
# PATH=/nix/store/...-binutils-wrapper-.../bin:/nix/store/...-customClang/bin:...
# CC=/nix/store/...-customClang/bin/customClang

# Bazel-side configuration for LRE.
build --define=EXECUTOR=remote
build --extra_execution_platforms=@local-remote-execution//rust/platforms:aarch64-apple-darwin,@local-remote-execution//rust/platforms:aarch64-apple-darwin
build --extra_toolchains=@local-remote-execution//rust:rust-aarch64-darwin
build --extra_toolchains=@local-remote-execution//rust:rustfmt-aarch64-darwin
build --platforms=@local-remote-execution//rust/platforms:aarch64-apple-darwin
```

Note that it registered only the **Rust** platform and toolchains even
though the flake asked for the `lre-cc` `Env`: the module emits the Rust
lines on every system and adds the `generated-cc` platform and
`cc-toolchain` lines only on Linux. `Env` only decides which store paths
are listed (and therefore fetched) in the commented header. The template's
own `.bazelrc` separately adds
`--extra_execution_platforms=@//platforms:lre-cc` for the C++ example, which
is the container-based platform from the table above.

</li>

<li>

### Point `user.bazelrc` at your cluster [#point-userbazelrc-at-your-cluster]

`nix flake init` writes a `user.bazelrc` with the literal string `TODO` as
every value. That is what lands on disk, not a placeholder left in this page:

```bash
build --remote_cache=grpcs://TODO
build --bes_backend=grpcs://TODO
build --remote_timeout=600
build --remote_executor=grpcs://TODO
```

Filled in against a plain local cluster: no TLS, so `grpc://`, and no BES
endpoint to point at:

```bash
build --remote_cache=grpc://127.0.0.1:50051
build --remote_timeout=600
build --remote_executor=grpc://127.0.0.1:50051
```

A hosted or self-managed cluster is the same shape with `grpcs://` and real
hostnames, plus a `bes_backend` line if you want build results streamed.

If you build before editing this file, Bazel fails on the literal hostname
`TODO`:

```plaintext
[8 / 10] Compiling src/hello-world.cpp; 0s remote, remote-cache
ERROR: BUILD.bazel:3:10: Compiling src/hello-world.cpp failed:
  Failed to query remote execution capabilities: UNAVAILABLE: Unable to resolve host TODO
```

Everything before that DNS failure worked: the Nix toolchain, the platform
wiring, and the `lre-cc` execution platform selection. `0s remote,
remote-cache` means Bazel had already committed to executing remotely.

</li>

<li>

### Build [#build]

```bash
bazel build hello-world
```

With a real endpoint and a worker running the `nativelink-worker-lre-cc`
image, this compiles `src/hello-world.cpp` on that worker and returns the
binary through the same CAS your local Bazel reads from. Run it again and
it's a cache hit.

</li>

</Steps>

<VerifyBlock>

- `lre.bazelrc` exists in the workspace root and contains
  `--extra_toolchains` lines.
- `bazel build` reports the action as `remote` (for `lre-cc`), or, run
  without `--remote_executor`, builds locally with the pinned toolchain
  (Rust on any of the four systems, C++ on `x86_64-linux`).
- The **same target built on two different machines** produces a cache hit
  on the second. This is the actual test of hermeticity; same-machine
  re-runs prove nothing about it.

</VerifyBlock>

<Callout type="info" title="No container-capable worker handy?">
  You can still validate the cache half.
  [Local testing](/remote-execution/local-testing) starts a NativeLink
  instance on `localhost`, which is enough to exercise `--remote_cache`
  against real credentials before you also need a worker matching
  `lre-cc`'s platform requirements.
</Callout>

## Adding LRE to a project you already have [#adding-lre-to-a-project-you-already-have]

The template bootstraps a *new* workspace. For an existing one you need the
flake-parts module wiring, the `bazel_dep` override, and a `try-import` of
the generated `lre.bazelrc`.
<SourceLink file="local-remote-execution/README.md" symbol="local-remote-execution/README.md" />
has that path, and also covers verifying the setup against a Kubernetes
cluster.

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| `WARNING: lre: .git not found; skipping installation.` | `nix develop` ran outside a git repo | `git init && git add -A`, then re-enter the shell |
| `Unable to resolve host TODO` | `user.bazelrc` still has the generated placeholders | Replace all three `TODO`s |
| Local and remote stop sharing cache hits | `lre.bazelrc` has stale `/nix/store` paths | Compare against the `@local-remote-execution` module at the commit pinned in `MODULE.bazel` / `flake.nix`; version drift is the usual cause |
| C++ actions queue forever | No worker advertises the `lre-cc` container image | See [Containers and images](/remote-execution/containers-and-images) and [Platform properties](/remote-execution/platform-properties) |
| Action runs remotely but can't find a header | The toolchain is pinned, the *inputs* aren't | Check that the header is a declared Bazel dependency, not something on the include path by accident |

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Can I get hermeticity without Nix?" id="can-i-get-hermeticity-without-nix">
You can get *reproducibility across a fleet* by baking the toolchain into
one worker image and routing every action to it. That's path 1 at the top
of this page, and for many teams it's enough. What you don't get is matching
action digests between a developer's laptop and the fleet, because the
laptop isn't running that image. Nix pinning is what closes that gap.

Note this is a client-side requirement only. The NativeLink server, the
scheduler, and the workers run fine with no Nix anywhere.
</Accordion>

<Accordion title="How is this different from local testing?" id="how-is-this-different-from-local-testing">
[Local testing](/remote-execution/local-testing) validates the *protocol*
with whatever toolchain is on your `PATH`; it deliberately has no toolchain
opinion, which is what makes it a good first step. This page pins the
toolchain so local and remote actions hash identically. Do that one first if
you just want to see `Execute` work.
</Accordion>

<Accordion title="Does hermeticity slow my build down?" id="does-hermeticity-slow-my-build-down">
The first `nix develop` is slow; it's fetching a full toolchain closure,
and 15 to 20 GB of free disk is a reasonable expectation. After that it's a
no-op. At build time, pinned toolchains are marginally *faster* to cache
against, because the digest stability means fewer spurious misses.
</Accordion>

<Accordion title="Which family should I use for a mixed-language repo?" id="which-family-for-a-mixed-language-repo">
They compose: a workspace can register Rust, C++, and Java toolchains at
once, and Bazel resolves per target. The practical constraint is that only
Rust has native non-Linux configs, so on macOS your C++ and Java actions
need a reachable worker while your Rust actions can still run locally.
</Accordion>

</Accordions>

<NextStep href="/remote-execution/containers-and-images" title="Containers and images">
  The other path to a reproducible worker environment, and what the
  `container-image` property actually does.
</NextStep>

<NextStep href="/explanations/lre" kind="aside" title="What LRE is and why">
  The conceptual write-up behind the mechanism on this page.
</NextStep>

# https://docs.nativelink.com/remote-execution/containers-and-images

**Containers and images**: What the container-image property actually does (and doesn't), plus how to build worker images and keep what your build requests in sync with what your fleet advertises.

**Who this is for:** anyone whose actions need a specific environment, and
anyone running more than one kind of worker image. **What you'll have at the
end:** worker images that carry your toolchain, and a routing scheme that
sends each action to the right one. **Time:** twenty minutes.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

## What `container-image` is not [#what-container-image-is-not]

Start here, because the name is misleading and the misunderstanding is
expensive.

**NativeLink does not pull an image and launch a container for each action.**
No code sits behind the `container-image` key. Search the source for it
and you will find it in example configs, in the LRE platform definitions under
`local-remote-execution/`, and in CI wiring; nothing in the scheduler or worker
source reads it. It is an ordinary string platform property with an evocative
name.

What it actually expresses is: *this worker process is running inside this
image*. The worker asserts a value, the build requests a value, and the
scheduler matches them like any other `exact` or `priority` property. The
environment is real; it's just established by how you *started the worker*,
not by anything the scheduler does per action.

<Callout type="warn" title="Byte-for-byte or nothing">
  The repo's own CI action puts it plainly: the value you request "must
  byte-for-byte match the `container-image` the claim's worker pool
  advertises, or actions queue forever." No normalisation, no tag
  resolution, no digest comparison. `debian:bookworm-slim` and
  `docker.io/library/debian:bookworm-slim` are different strings and
  therefore different pools.
</Callout>

That failure is the silent-queue failure from
[Platform properties](/remote-execution/platform-properties), and it is by
far the most common way a container-image scheme goes wrong.

## The actual model [#the-actual-model]

<Mermaid>
{`flowchart LR
    I["base image<br/>(your toolchain)"] -->|createWorker| WI["nativelink-worker-*<br/>image"]
    WI -->|"docker run / k8s pod"| WP["worker process<br/>advertises container-image"]
    WP -->|connect_worker| S[scheduler]
    A["action<br/>requests container-image"] --> S
    S -->|match| WP`}
</Mermaid>

One image per worker *pool*. Every action a given worker runs shares that
worker's environment. Heterogeneity comes from running several pools and
routing between them, not from varying the image per action.

This has a consequence worth planning around: **image churn is a fleet
operation.** Changing a toolchain means building a new image, rolling a pool
onto it, and changing what builds request, in an order that doesn't strand
actions in the queue. There's a safe sequence below.

## Build a worker image [#build-a-worker-image]

A NativeLink worker image is any image that contains your toolchain and can
run the `nativelink` binary with a worker config. The repo's Nix helper,
`createWorker`, turns a toolchain image into a `nativelink-worker-*` image:

```nix
nativelink-worker-lre-cc = createWorker pkgs.lre.lre-cc.image;
nativelink-worker-lre-java = createWorker lre-java;
nativelink-worker-lre-rs = createWorker pkgs.lre.lre-rs.image;
nativelink-worker-siso-chromium = createWorker siso-chromium;
nativelink-worker-toolchain-drake = createWorker toolchain-drake;
nativelink-worker-toolchain-buck2 = createWorker toolchain-buck2;
```

<SourceLink file="tools/public/create-worker.nix" symbol="createWorker" />

What it adds on top of the base image is small: a `nativelink` user (uid
1000) and home directory, a world-writable `/tmp`, `/usr/bin/env`, and
`coreutils`, `bash` and `gnused` on `/bin`. It keeps the base image's tag so
the worker image and the toolchain it wraps stay visibly paired. It does
*not* bake the `nativelink` binary in, whatever its header comment says: the
Kubernetes examples inject the binary at pod start with the
`nativelink-worker-init` init container, which copies it into a shared
volume, and the worker container then runs `/shared/nativelink /worker.json5`.
That keeps the toolchain image and the NativeLink version independently
upgradable.

<SourceLink file="tools/nativelink-worker-init.nix" />
<SourceLink file="kubernetes/components/worker/worker.yaml" />

Push one to a registry the way the repo's own tooling does:

```bash
nix run .#nativelink-worker-lre-cc.copyTo docker://localhost:5001/nativelink-worker-lre-cc:latest
```

You do not have to use Nix for this. Any image that contains your toolchain
and runs `nativelink` with a config is a valid worker image: a Dockerfile
that starts `FROM your-toolchain-base`, copies in the binary, and sets the
entrypoint works identically. Nix is how this repo gets *reproducible* images;
it isn't a requirement of the worker protocol.

## Wire up the routing [#wire-up-the-routing]

<Steps>

<li>

### Declare the key on the scheduler [#declare-the-key-on-the-scheduler]

```json5
schedulers: [{
  name: "MAIN_SCHEDULER",
  simple: {
    supported_platform_properties: {
      cpu_count: "minimum",
      OSFamily: "priority",
      "container-image": "exact",
    },
  },
}]
```

Use `exact` once you have more than one image. The examples ship `priority`,
which matches any value: fine for a single-pool tutorial, useless as a
routing mechanism.

</li>

<li>

### Advertise it on each worker [#advertise-it-on-each-worker]

```json5
workers: [{
  local: {
    worker_api_endpoint: { uri: "grpc://scheduler.internal:50061" },
    cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
    upload_action_result: { ac_store: "AC_MAIN_STORE" },
    work_directory: "/tmp/nativelink/work",
    platform_properties: {
      cpu_count: { query_cmd: "nproc" },
      OSFamily: { values: [""] },
      "container-image": { values: ["ghcr.io/example/toolchain:2026-07-01"] },
    },
  },
}]
```

Pin a tag you can change deliberately. `:latest` makes the advertised string
stable while the actual contents drift, which defeats the entire point: the
scheduler will keep matching, and your builds will silently change behaviour.

Better still, template it from the deployment so the value can't disagree
with the image that's actually running:

```json5
"container-image": { values: ["${NL_WORKER_IMAGE}"] },
```

</li>

<li>

### Request it from the build [#request-it-from-the-build]

Globally, if your whole build wants one environment:

```bash
bazel build \
  --remote_executor=grpc://scheduler.internal:50052 \
  --remote_default_exec_properties=container-image=ghcr.io/example/toolchain:2026-07-01 \
  //...
```

Or per target, when only some actions need it:

```python
cc_test(
    name = "integration_test",
    exec_properties = {
        "container-image": "ghcr.io/example/toolchain:2026-07-01",
    },
    ...
)
```

</li>

</Steps>

<VerifyBlock>

- The build reports actions as `remote` rather than sitting at `0s remote`.
- Stopping the pool that advertises the requested image makes builds queue,
  and starting it drains them. That's the proof the routing is real rather
  than everything falling through to one pool.
- With `exact`, requesting a *wrong* image string queues indefinitely. If it
  succeeds instead, the key is still declared `priority` somewhere.

</VerifyBlock>

## Rolling an image without stranding actions [#rolling-an-image-without-stranding-actions]

Order matters, because the two sides are matched on a literal string.

1. **Bring up the new pool first**, advertising the new image string, while
   the old pool keeps running and advertising the old one. Both are now
   satisfiable.
2. **Switch what builds request.** New builds land on the new pool;
   in-flight ones finish on the old.
3. **Drain and remove the old pool** once nothing requests it.

Doing this in the other order (changing the request before the pool exists,
or removing the pool before builds stop asking for it) produces a queue that
never clears and no error anywhere.

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| Everything queues after an image bump | Requested string doesn't match any advertised string | Compare them character by character; registry prefixes and tags both count |
| `Property missing on worker property container-image` | A pool doesn't advertise the key at all | Add it to that pool's `platform_properties`, even with an empty value |
| Routing "works" but actions land on the wrong pool | Key is declared `priority`, which matches any value | Change the scheduler's declaration to `exact` |
| Builds change behaviour with no config change | Advertised tag is mutable (`:latest`) | Pin an immutable tag or digest |
| Worker never joins the pool | Key not declared in `supported_platform_properties` | Look for `Bad Property during connect_worker()` in the worker log |

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Can a single worker serve multiple images?" id="can-a-single-worker-serve-multiple-images">
Not meaningfully. A worker process runs in one environment; the property is
an assertion about that environment. You can advertise several values for a
`priority` key, but that's a routing label, not a second toolchain. Run one
pool per image.
</Accordion>

<Accordion title="What if I want a real container per action?" id="what-if-i-want-a-real-container-per-action">
The worker's `entrypoint` option lets you interpose a wrapper: the configured
program is prepended to every action's command line as its first argument
(the config comment says "command + arguments", but the code prepends the
string as a single program path, so point it at one executable and let that
executable take the action's command as its arguments), which makes a script
that launches a container and forwards the command possible. If that wrapper
enforces its own timeouts, set
`timeout_handled_externally` so NativeLink doesn't also time the action out.
This is a real deployment shape, but it is something you build, not something
`container-image` gives you.
<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />
<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="inner_execute" />
</Accordion>

<Accordion title="Do I need containers at all?" id="do-i-need-containers-at-all">
No. Workers run fine as plain processes on bare metal or a VM; the property
is only a matching string, so a fleet that doesn't use containers
doesn't declare the key.
[Deploy on bare metal](/operate/deploy-bare-metal) covers that shape.
</Accordion>

<Accordion title="How does this relate to LRE?" id="how-does-this-relate-to-lre">
They're the two answers to the same question. Images make the *fleet*
reproducible: every action in a pool sees the same environment.
[LRE](/remote-execution/toolchains-and-hermeticity) makes the *action*
reproducible by pinning the toolchain into the action itself, which
additionally lets a developer's laptop share cache entries with the fleet.
They compose: the `lre-cc` and `lre-java` workers are container images whose
contents are Nix-pinned.
</Accordion>

<Accordion title="Should the value be a tag or a digest?" id="should-the-value-be-a-tag-or-a-digest">
A digest is stricter and safer; an immutable dated tag is easier to read in
build logs and error messages. Either is fine. What matters is that the string
changes when the contents change, so that a stale pool stops matching instead
of silently serving the old environment.
</Accordion>

</Accordions>

<NextStep href="/remote-execution/persistent-workers" title="Persistent workers">
  Keeping a warm process between actions, for toolchains where startup cost
  dominates.
</NextStep>

<NextStep href="/how-to/run-multiple-workers" kind="aside" title="Run multiple workers">
  Running several worker pools, each on its own image, as a real fleet.
</NextStep>

# https://docs.nativelink.com/remote-execution/persistent-workers

**Persistent workers**: Keep a JVM or Node compiler process warm across actions instead of paying its startup cost every time, and understand exactly what NativeLink gives up to do it.

**Who this is for:** anyone whose remote builds are dominated by JVM or
TypeScript compile actions. **What you'll have at the end:** compile actions
reusing a warm process, and a clear-eyed view of the isolation you trade for it.
**Time:** an hour, most of it in your build rules rather than in NativeLink.

<Prerequisites>
  Remote execution working end to end. See [Your first remote action](/remote-execution/first-remote-action).
</Prerequisites>

A `javac` invocation spends a large fraction of its wall time before it compiles
anything: start a JVM, load the compiler, warm the JIT. Locally you amortize
that across a build because Bazel keeps the process alive. Under
[remote execution](/remote-execution) the default is one process per action, so
every single compile pays the full tax, and a thousand-target Java build pays it
a thousand times.

Persistent workers remove that. The tool is started once, told it is a
persistent worker, and then fed one request per action over stdin while staying
alive between them.

<Mermaid>
{`flowchart TB
  subgraph OS["Default: one process per action"]
    A1["action 1"] --> P1["start JVM → compile → exit"]
    A2["action 2"] --> P2["start JVM → compile → exit"]
    A3["action 3"] --> P3["start JVM → compile → exit"]
  end
  subgraph PW["Persistent: one process, many actions"]
    B1["action 1"] --> W["javac_worker --persistent_worker"]
    B2["action 2"] --> W
    B3["action 3"] --> W
    W --> WW["JVM started once, stays warm"]
  end`}
</Mermaid>

<SourceLink file="nativelink-worker/src/persistent_worker/pool.rs" symbol="PersistentWorkerPool" />

## Two things to know before you start [#two-things-to-know-before-you-start]

**Almost nothing needs configuring in NativeLink.** No field anywhere in
[the configuration model](/configuration) turns this on, tunes the pool, or
selects a protocol; `LocalWorkerConfig` has no persistent-worker section at
all, and every pool parameter is a compiled-in constant. The feature is
driven by two platform properties on each action. The one thing NativeLink
does need from you is on the scheduler: those two keys must be listed in
`supported_platform_properties`, because the scheduler rejects an action that
carries a key it does not know. If you have seen a `persistent_workers`
config block in older documentation, it did not correspond to anything in the
binary.

**The module is licensed separately.** Most of the repository is
`FSL-1.1-Apache-2.0`. Every file under `nativelink-worker/src/persistent_worker/`
instead carries a Business Source License 1.1 header stating that use of the
module requires an enterprise license agreement. That is a different obligation
from the rest of the binary, and nothing in the binary enforces it; the module
is compiled in and activates from the action, not from a flag you set. Read
[Open source and Enterprise](/reference/oss-and-enterprise#persistent-workers-activate-from-the-action-not-from-your-config)
for what that means in practice, and
[the license](https://nativelink.com/license) against your intended use, before
you build on this.

## Opt an action in [#opt-an-action-in]

The worker looks at two of the action's platform properties (the REAPI
`Platform` on the `Action`): `supports-workers` must be exactly `1`, and
`requires-worker-protocol` picks the wire format. Nothing on the worker's own
`platform_properties` is involved.

<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="action_supports_persistent_workers" />

Because the scheduler fails to match any action carrying a platform property
key it has not been told about (`Unknown platform property`), declare both keys
on the scheduler. `ignore` lets actions request a key without any worker having
to advertise it:

```json5
schedulers: [{
  name: "MAIN_SCHEDULER",
  simple: {
    supported_platform_properties: {
      cpu_count: "minimum",
      "supports-workers": "ignore",
      "requires-worker-protocol": "ignore",
    },
  },
}]
```

Then put the two keys on the action. The rule sketches in
`deployment-examples/persistent-workers/` (Java, Kotlin and TypeScript) show
them as `execution_requirements`:

```starlark
def _javac_worker_impl(ctx):
    args = ctx.actions.args()
    args.add("@%s" % ctx.outputs.argfile.path)

    ctx.actions.run(
        executable = ctx.executable.javac_worker,
        arguments = [args],
        inputs = ctx.files.srcs + [ctx.outputs.argfile],
        outputs = [ctx.outputs.jar],
        mnemonic = "Javac",
        execution_requirements = {
            "supports-workers": "1",
            "requires-worker-protocol": "proto",
        },
    )
```

Kotlin is identical in shape. TypeScript wrappers conventionally use the JSON
framing instead:

```starlark
execution_requirements = {
    "supports-workers": "1",
    "requires-worker-protocol": "json",
}
```

<Callout type="warn" title="The keys have to reach the remote Platform">
  The worker reads the `Platform` proto, not Bazel's execution info. Bazel
  builds the remote `Platform` from `exec_properties` (on the target or the
  execution platform) and `--remote_default_exec_properties`; it does not copy
  `execution_requirements` into it. If your actions never take the persistent
  path, set the same two keys as `exec_properties` on the target, or pass
  `--remote_default_exec_properties=supports-workers=1` for a whole build, and
  check the worker log for `Spawned new persistent worker`.
</Callout>

Both wire formats are fully implemented. `proto` is length-delimited protobuf;
`json` is newline-delimited JSON with camelCase field names, where a request's
input digests are base64. `proto` is the default when the property is absent.
Any other value is logged and the action falls back to ordinary one-shot
execution.

The decision is made on the worker, per action, at the moment it is about to
execute. The scheduler only needs to know the keys exist; it does not treat
these actions differently.

## What makes two actions share a process [#what-makes-two-actions-share-a-process]

Actions are pooled by a key derived from the command line:

The **executable**: the first argument.

The **startup arguments**: every argument after the executable *up to the
first one beginning with `@`*, which is Bazel's response-file convention.
Everything from that `@` onward is per-request payload, sent in the
`WorkRequest` rather than used for pooling.

The **wire format** from `requires-worker-protocol`.

Two actions with the same three share a process; anything else gets its own.
This is why the argfile convention matters more than it looks: an action with
**no** `@argfile` puts its entire command line into the key, so every action
with different arguments becomes its own worker and you get no reuse at all
while everything still appears to work.

<Callout type="warn" title="`entrypoint` shifts the key">
  If the worker has an `entrypoint` configured, it is prepended to the command
  before the key is derived, so the executable in the key is the entrypoint,
  not your tool, and the tool name becomes a startup argument. It still pools
  correctly, but the key is not what you would predict from the rule.
</Callout>

## The isolation you give up [#the-isolation-you-give-up]

This is the part to read twice. NativeLink applies real isolation to ordinary
actions and **applies none of it to persistent workers**. The persistent path
returns before any of it runs.

**No namespace sandboxing.** The PID, user, IPC, UTS and mount namespaces that
ordinary actions get are not applied. `use_namespaces` and `use_mount_namespace`
have no effect on a persistent worker.

**No environment control.** Ordinary actions get a cleared environment plus
exactly what the action and config specify. A persistent worker process
inherits the NativeLink worker process's own environment, and
`environment_variables` on the command, `additional_environment` on the config,
and the side-channel environment sources are all ignored.

**A shared, long-lived working directory.** The process's own working directory
is the worker's root action directory, not the per-action one; it has to be,
because per-action directories are deleted when their action finishes. The
per-action directory is instead handed to the tool as `sandbox_dir` on each
request, and **it is the tool's job to honor it**. A worker that writes relative
to its own cwd will scribble into a directory shared with every other action it
ever serves.

**No input digests.** The request carries an empty input list, so a tool cannot
use it to validate or invalidate its own cached state. Anything the tool caches
across requests, it caches on trust.

The consequence is a real correctness requirement, not a footnote: **a
persistent worker must be hermetic by construction, because nothing is enforcing
it.** State that leaks between requests produces wrong outputs that are then
cached as if they were right, which is the worst failure mode a build system
has. Only enable this for tools written for the worker protocol and tested under
it.

<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="inner_execute" />

## The pool's actual behaviour [#the-pools-actual-behaviour]

Every parameter below is a compiled-in constant. None is configurable.

**Four workers per key.** A fifth concurrent action for the same key does not
queue; it falls back to normal one-shot execution and logs that it did. Your
build stays correct and quietly gets slower, which is worth knowing when you are
measuring.

**Two hundred requests per worker.** After that the process is retired and the
next action starts a fresh one. This is deliberate recycling of accumulated JVM
state, and it means a long build will restart workers periodically by design.

**No idle timeout, in practice.** The sweeper that would retire an idle worker
exists in the code but nothing calls it. A worker lives until the request cap
retires it, until it dies on its own, or until the NativeLink worker process
exits.

**Dead workers are noticed on the next acquire**, not proactively. A JVM that
OOMs while idle is discovered when the next action asks for it, and that action
starts a replacement.

**Shutdown is not graceful.** Stdin is closed, giving the tool five seconds to
exit on EOF, and then it is killed outright. No `SIGTERM` is sent.

**One request at a time per process.** Multiplexed workers are not supported; a
request carrying a non-zero request ID is rejected.

## Steps [#steps]

<Steps>

<li>

**Confirm the tool is actually a worker.** It must accept `--persistent_worker`,
read framed `WorkRequest` messages from stdin, and write framed `WorkResponse`
messages to stdout. Most JVM compilers have a worker wrapper already; a plain
compiler binary does not become one by being declared as one.

</li>

<li>

**Declare the execution requirements** on the action, with an `@argfile` holding
everything that varies per action so that the pooling key stays stable.

</li>

<li>

**Audit the tool for cross-request state.** Anything cached in memory between
requests must be keyed on inputs the request actually carries. Anything written
to disk must go under `sandbox_dir`.

</li>

<li>

**Run a build and read the worker's log.** You are looking for one process
start serving many actions, not one start per action.

</li>

<li>

**Compare compile durations** against the same build with the execution
requirements removed. The first action of each key should be unchanged; the
rest should drop sharply.

</li>

</Steps>

<VerifyBlock>

- The action-duration distribution for that mnemonic becomes bimodal: a few slow
  first-of-key actions, the rest much faster.
- Process count on the worker host stays flat during a build instead of churning
  once per action.
- Building the same targets twice in a row, cache disabled, is faster the second
  time within a single build: the pool is warm.
- Nothing in the NativeLink worker log says it fell back to one-shot execution.
- Outputs are byte-identical to the same build without persistent workers. This
  is the one that matters.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Nothing changes: actions still start a process each time" id="nothing-changes--actions-still-start-a-process-each-time">

Either the properties are not reaching the action's `Platform`, or every action
is getting its own key. Check `supports-workers` is exactly `"1"` and is sent as
a platform property (see the callout above), and that the scheduler declares
both keys; an undeclared key leaves the action queued forever. Then check the
command line has an `@argfile`: without one, every distinct command line is a
distinct key, and you get a warm process per action, which is no process reuse
at all.

</Accordion>

<Accordion title="Outputs are wrong, but only sometimes" id="outputs-are-wrong-but-only-sometimes">

Stop and disable persistent workers for that mnemonic. This is the signature of
state leaking between requests: a cache in the tool keyed on something the
request no longer determines, or output written relative to the process's cwd
instead of `sandbox_dir`. Wrong outputs get cached, so also consider what needs
invalidating.

</Accordion>

<Accordion title="Builds get slower under high parallelism" id="builds-get-slower-under-high-parallelism">

More than four concurrent actions for one key fall back to one-shot execution,
and one-shot execution of a worker-wrapped tool can be slower than the plain
tool. The per-key cap is not configurable, so the levers are reducing
concurrency for that mnemonic or splitting the work across more distinct keys.

</Accordion>

<Accordion title="The tool can't find its inputs" id="the-tool-cant-find-its-inputs">

It is resolving paths relative to its own working directory, which is the
worker's root directory rather than the action's. Every path must be resolved
against the `sandbox_dir` the request carries. Tools written for local Bazel
workers usually get this right; wrappers written in-house usually do not, at
first.

</Accordion>

<Accordion title="`Unsupported requires-worker-protocol value`" id="unsupported-requires-worker-protocol-value">

The property must be exactly `proto` or `json`. Anything else is rejected and
the action falls back to one-shot execution.

</Accordion>

<Accordion title="Where did the tool's output go?" id="where-did-the-tools-output-go">

A persistent worker's stderr is discarded, because it belongs to the process
rather than to any one action and cannot be attributed safely. What surfaces as
the action's stderr is the `output` field of the `WorkResponse`, and the
action's stdout is always empty. If a tool reports errors on its real stderr
instead of in the response, those errors are invisible; that is a bug in the
worker wrapper.

</Accordion>

<Accordion title="Memory on the worker host keeps climbing" id="memory-on-the-worker-host-keeps-climbing">

Warm JVMs are the point and they are not free: up to four live processes per
distinct key, each holding a heap. The 200-request cap recycles them eventually,
but the steady state is higher than one-shot execution. Size worker hosts for
the pool, not for one compiler.

</Accordion>

</Accordions>

<NextStep href="/operate/tuning" title="Tuning">
  Worker-host sizing and the levers that matter once a pool of warm processes is
  part of your steady state.
</NextStep>

<NextStep href="/remote-execution/toolchains-and-hermeticity" kind="aside" title="Toolchains and hermeticity">
  Why hermeticity is the property everything else rests on, and what it means
  that persistent workers rely on the tool to preserve it.
</NextStep>

# https://docs.nativelink.com/remote-execution/local-testing

**Local testing**: Keep a NativeLink cluster on your own machine to test config changes, reproduce scheduler behaviour, and prove a client works, in the two shapes the repo itself uses.

**Who this is for:** anyone changing a config, debugging a client, or
reproducing something a shared cluster did. **What you'll have at the end:**
a local cluster in whichever of the two shapes matches what you're testing,
and a reliable way to tell what actually ran where. **Time:** ten minutes
for the single-process shape.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

[Your first remote action](/remote-execution/first-remote-action) gets you
one remote build. This page is what you do afterwards: keep a cluster around
as a test fixture, and know which shape to run for what you're testing.

<Callout type="info" title="This is not LRE">
  Nothing here has a toolchain opinion; it uses whatever is on your `PATH`,
  and needs no Nix. That's deliberate: it isolates *protocol and config*
  problems from *toolchain* problems.
  [Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
  is the other axis.
</Callout>

## Which shape to run [#which-shape-to-run]

| You're testing | Run | Why |
| --- | --- | --- |
| A client's flags, a store config, scheduler matching | Single process | Fastest loop; one log to read |
| Cross-process wiring, `grpc` stores, separate CAS and scheduler | The compose split | Same topology as production, and what CI exercises |
| Toolchain hermeticity | Neither; see [Toolchains](/remote-execution/toolchains-and-hermeticity) | Different problem |

## Shape 1: one process [#shape-1-one-process]

<Mermaid>
{`flowchart LR
    subgraph client[Your machine]
        B[Bazel]
        subgraph nl["nativelink process"]
            direction LR
            L["0.0.0.0:50051\\ncas · ac · bytestream\\nexecution · capabilities"]
            W["0.0.0.0:50061\\nworker_api (private)"]
            WK[local worker]
        end
    end
    B -->|"remote_cache\\n+ remote_executor"| L
    L --- W
    W --- WK`}
</Mermaid>

`--remote_cache` and `--remote_executor` both point at `127.0.0.1:50051`.
`50061` exists only so the worker can register with the scheduler; Bazel
never talks to it.

```bash
nativelink ./nativelink-config/examples/local_rbe_self_test.json5
```

```plaintext
INFO nativelink: Ready, listening on 0.0.0.0:50051
INFO nativelink: Ready, listening on 0.0.0.0:50061
INFO nativelink_worker::local_worker: Worker registered with scheduler, worker_id: 1f175824-...
```

<SourceLink file="nativelink-config/examples/local_rbe_self_test.json5" />

The interesting parts of that config, for when you start editing it:

```json5
{
  stores: [
    { name: "CAS_MAIN_STORE", filesystem: { /* ... */ } },
    { name: "AC_MAIN_STORE", filesystem: { /* ... */ } },
    {
      // The worker's fast tier and the client-facing CAS share one store,
      // so anything the worker produces is immediately visible to Bazel.
      // `fast` must be a filesystem store: the worker hardlinks out of it
      // to build each action's sandbox.
      name: "WORKER_FAST_SLOW_STORE",
      fast_slow: {
        fast: { filesystem: { /* ... */ } },
        slow: { ref_store: { name: "CAS_MAIN_STORE" } },
      },
    },
  ],
  schedulers: [{
    name: "MAIN_SCHEDULER",
    simple: { supported_platform_properties: { cpu_count: "minimum" /* ... */ } },
  }],
  workers: [{
    local: {
      worker_api_endpoint: { uri: "grpc://127.0.0.1:50061" },
      cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
      upload_action_result: { ac_store: "AC_MAIN_STORE" },
      platform_properties: { cpu_count: { values: ["1"] } /* ... */ },
    },
  }],
  servers: [
    { name: "local", /* cas · ac · bytestream · execution · capabilities on :50051 */ },
    { name: "worker_api", /* worker_api · health on :50061 */ },
  ],
}
```

Then:

```bash
bazel test \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50051 \
  --remote_default_exec_properties=cpu_count=1 \
  //your:target
```

<Callout type="warn" title="Set both flags, even to the same address">
  It's tempting to assume `--remote_executor` alone is enough, since the
  executor reads and writes the CAS anyway. It isn't. Without an explicit
  `--remote_cache`, Bazel calls `Execute` with an action digest it never
  uploaded, and the server correctly rejects it:
  `FAILED_PRECONDITION: Action ... is missing from CAS`.
</Callout>

## Shape 2: the compose split [#shape-2-the-compose-split]

When you need separate processes (a CAS process and a scheduler process
talking over `grpc` stores, which is what a real deployment looks like),
the repo ships that too, and its own integration tests run against it.

```bash
cd deployment-examples/docker-compose
docker compose up
```

This publishes the CAS on `50051` (plus a TLS listener for the same services on
`50071`) and the **scheduler, with the execution and AC services, on
`50052`**. The scheduler's worker API on `50061` is not published from the
scheduler container at all, so it is only reachable inside the compose
network; the worker reaches it by service name.

That port split is why the repo's `.bazelrc` reads the way it does:

```bash
build:self_test --remote_cache=grpc://127.0.0.1:50051

build:self_execute --remote_executor=grpc://127.0.0.1:50052
build:self_execute --remote_default_exec_properties=cpu_count=1
build:self_execute --platform_suffix=self-execute
```

<SourceLink file=".bazelrc" />

```bash
bazel test --config self_test --config self_execute //:dummy_test
```

<Callout type="warn" title="These configs are for the compose split, not the single-process example">
  `self_execute` points at `50052`. If you're running
  `local_rbe_self_test.json5`, which puts everything on `50051`, these two
  `--config` flags will not work; use the explicit flags from shape 1
  instead. This mistake is common because both live in the same
  repo.
</Callout>

[Production configuration](/operate/production-config) explains the split
topology this is a miniature of.

## Verify what actually ran [#verify-what-actually-ran]

Wall-clock time is a bad signal and Bazel's summary line is only a rough
one. Ask the build event log directly:

```bash
bazel test \
  --config self_test --config self_execute \
  --nocache_test_results \
  --build_event_json_file=/tmp/bep.json \
  //:dummy_test

jq --slurp -r '.[] | select(.id.testResult.label=="//:dummy_test")
  | .testResult.executionInfo.strategy' /tmp/bep.json
```

```plaintext
remote
```

<SourceLink file="integration_tests/simple_remote_execution_test.sh" />

That is exactly what CI asserts, twice: once cold, then again after
`bazel clean` to confirm the second run is served remotely rather than from
Bazel's local disk cache.

For per-action detail rather than per-test, `--execution_log_json_file`
records the runner for every spawn.

<VerifyBlock>

- First run: Bazel's `INFO: N processes: ...` summary line counts the
  genrule and the test as `remote`, with no `remote cache hit`.
- After `bazel clean` and a re-run: the genrule that produces
  `dummy_test.sh` shows up as `remote cache hit` while the test itself
  re-executes remotely, because `--nocache_test_results` is set.
- `jq` reports `remote` both times.

</VerifyBlock>

## Troubleshooting [#troubleshooting]

<Accordions type="multiple">

<Accordion title="Actions stay queued forever" id="actions-stay-queued-forever">
At least one worker's `platform_properties` must satisfy what your actions
request, and every requested key must be declared in the scheduler's
`supported_platform_properties`. Starting with just `cpu_count: "minimum"`
on the scheduler and `cpu_count: { values: ["1"] }` on the worker unblocks
most local setups.
[Platform properties](/remote-execution/platform-properties) has the full
matching rules and the scheduler's own diagnostic logging.
</Accordion>

<Accordion title="'instance_name' not configured for ''" id="instance-name-not-configured">
Every service in the example config omits `instance_name`, so it defaults to
the empty string, the same default Bazel uses when you don't pass
`--remote_instance_name`. If you add an `instance_name` server-side, pass
the matching flag client-side, or every request fails this way.
</Accordion>

<Accordion title="The worker exits immediately on macOS" id="the-worker-exits-immediately-on-macos">
`use_namespaces` and `use_mount_namespace` are Linux-only and default to
`false`. Setting either to `true` on an unsupported platform makes the
worker exit at startup. Leave them unset for local testing.
</Accordion>

<Accordion title="Stale state between runs" id="stale-state-between-runs">
Everything the example config writes lives under
`/tmp/nativelink-local-rbe-test/`. Stop the server and delete that tree for
a genuinely cold start; `bazel clean` only clears Bazel's side.
</Accordion>

<Accordion title="Port already in use" id="port-already-in-use">
Usually a previous run still holding `50051` or `50061`, or the compose
stack still up. `docker compose rm --stop -f` in
`deployment-examples/docker-compose` clears the second case.
</Accordion>

</Accordions>

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Can clients other than Bazel use this?" id="can-clients-other-than-bazel-use-this">
Yes: the config exposes standard Remote Execution API services, so Buck2,
Pants, Siso, and recc point at `127.0.0.1:50051` the same way. The Bazel
flags are just the worked example;
[Connect your build](/getting-started/connect-your-build) has the others.
</Accordion>

<Accordion title="Is a local cluster a reasonable everyday setup?" id="is-a-local-cluster-a-reasonable-everyday-setup">
For sandboxed, cached single-machine builds, yes. What it can't give you is
parallelism beyond one worker's `cpu_count`, or sharing with anyone else.
The moment you want either, you want a real deployment:
[Production configuration](/operate/production-config).
</Accordion>

<Accordion title="Should I test config changes here before deploying them?" id="should-i-test-config-changes-here-before-deploying">
Yes, and it's the main reason to keep a local cluster around. Store
composition, scheduler property maps, and service wiring all fail the same
way locally as they do in a cluster, and the feedback loop is seconds
instead of a deploy.
</Accordion>

</Accordions>

<NextStep href="/remote-execution/examples-and-templates" title="Examples and templates">
  Configs and project scaffolds to copy, for the shape you actually want.
</NextStep>

<NextStep href="/configuration" kind="aside" title="Configuration">
  When you want to write these config files yourself rather than
  edit an example.
</NextStep>

# https://docs.nativelink.com/remote-execution/examples-and-templates

**Examples and templates**: Runnable configs, deployment topologies, and project scaffolds you can copy, plus the three client-side patterns most teams actually adopt.

**Who this is for:** anyone who wants a working starting point rather than a
blank file. **What you'll have at the end:** the right example for your
situation, and a client-side pattern that matches how you want work split
between laptops and a fleet. **Time:** varies; the point of this page is to
find the thing, not to read it.

<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with [Getting started](/getting-started).
</Prerequisites>

Three kinds of starting point ship in the repo, and they answer different
questions.

| You want | Look in |
| --- | --- |
| A NativeLink config for a specific backend or shape | [`nativelink-config/examples/`](#server-configs) |
| A whole deployment topology to run | [`deployment-examples/`](#deployment-topologies) |
| A client project pre-wired to a hermetic toolchain | [`nix flake init`](#project-templates) |

## Server configs [#server-configs]

<SourceLink file="nativelink-config/examples" dir symbol="nativelink-config/examples/" />
is the largest and most useful collection. Every file is a complete,
loadable config.

| File | What it demonstrates |
| --- | --- |
| `basic_cas.json5` | The minimum cache, the quickstart's config |
| `filesystem_cas.json5` | On-disk CAS with compression, dedup, and size partitioning in front of it |
| `local_rbe_self_test.json5` | A complete cluster in one process: CAS, AC, scheduler, worker |
| `stores-config.json5` | One block per store type, generated from the `stores.rs` doc comments for testing |
| `chunking_cas.json5` | Content-defined chunking |
| `advanced_http.json5` | HTTP/2 listener tuning (keepalive, stream limits, window sizes) |
| `s3_backend_with_local_fast_cas.json5` | `fast_slow` over S3, the canonical production store |
| `gcs_backend.json5`, `azure_blob_backend.json5`, `r2_backend.json5`, `oci_backend.json5`, `ontap_backend.json5` | The other object-storage backends |
| `redis.json5`, `mongo.json5` | Redis and MongoDB stores |
| `worker_with_redis_scheduler.json5` | A scheduler whose state lives in Redis |
| `scheduler_match_logging_disable.json5` | Turning off the periodic match dump |
| `tmpfs-worker.json5` | A worker whose fast tier and work directory sit on a tmpfs mount |
| `legacy_service_config.json5` | The older service config shape, for migrations |

<SourceLink file="nativelink-config/examples/README.md" />

Start from the file closest to your situation and read its comments;
several of them explain constraints you'd otherwise discover the hard way,
like why a worker's `fast` store must be a filesystem store.

[How-to guides](/how-to) is the task-oriented version of that table if you
already know which backend you want.

## Deployment topologies [#deployment-topologies]

<SourceLink file="deployment-examples" dir />
holds deployment material rather than single config files: the
`docker-compose/` stack, a RHEL 8 `Dockerfile` (`rhel/`), the metrics
pipeline configs and dashboards (`metrics/`), and the persistent-worker rule
sketches (`persistent-workers/`).

The `docker-compose/` one is the most useful of them, because it is the
topology the project's own integration tests run against: a CAS process on
`50051`, a scheduler with the execution and AC services on `50052`, a worker,
and the scheduler's worker API on `50061` left unpublished so it is only
reachable inside the compose network.

```bash
cd deployment-examples/docker-compose
docker compose up
```

[Local testing](/remote-execution/local-testing) covers running this and
pointing a build at it; [Production configuration](/operate/production-config)
covers what changes when it stops being a demo.

## Project templates [#project-templates]

The flake ships templates for the client side: a project already wired to a
Nix-pinned toolchain.

```bash
nix flake show github:TraceMachina/nativelink
```

<Callout type="info" title="That command is the source of truth">
  Templates get added and removed between releases. Run it rather than
  trusting a hardcoded list, including the one below, which reflects what
  ships as of this page's last update.
</Callout>

| Template | What it gives you |
| --- | --- |
| `bazel` | A Bazel `cc_binary` project pre-wired for [Local Remote Execution](/remote-execution/toolchains-and-hermeticity): Nix-pinned toolchain, the LRE Bazel module, and a `user.bazelrc` you point at a cache and executor |

```bash
mkdir my-rbe-test && cd my-rbe-test
nix flake init -t github:TraceMachina/nativelink#bazel
git init && git add -A   # what templates/README.md tells you to do; a flake in a git repo only sees tracked files
nix develop
```

The template does **not** bundle a NativeLink server; you point it at one you
already have. It is also Linux-only, because the `lre-cc` toolchain it wires
in currently supports `x86_64-linux` only.
[Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity)
walks the whole flow with real output, including the `user.bazelrc` values.

<Callout type="warn" title="cargo, lre, and kubernetes templates do not exist">
  Older material referenced them. `nix flake init -t
  github:TraceMachina/nativelink#lre` (or `#cargo`, `#kubernetes`) fails with
  `does not provide attribute 'templates.<name>'`. If you came looking for one:

  - **A local cluster with no Nix**:
    [Local testing](/remote-execution/local-testing).
  - **Cargo / sccache**: not packaged as a template; the `bazel` template
    is the maintained path.

  The repo also has a `templates/cmake/` directory (CMake with `recc`,
  cache-only), but `flake.nix` does not expose it under `templates`, so
  `nix flake init -t github:TraceMachina/nativelink#cmake` fails the same way;
  copy the directory by hand if you want it.
</Callout>

To customise: it's a flake, so fork and edit. The usual changes are pointing
`user.bazelrc` at a real cluster, adding platform properties in
`platforms/BUILD.bazel`, and pinning `inputs.nativelink.url` in `flake.nix`
to a specific commit, kept in sync with the `local-remote-execution`
override in `MODULE.bazel`.

## Three client-side patterns [#three-client-side-patterns]

Independently of which config you start from, there are three ways teams
wire their builds. Most adopt them in this order.

### Pattern A: Cache only [#pattern-a--cache-only]

Every action runs locally; results land in the shared cache. No worker fleet
needed, which makes it the cheapest thing that helps.

```bash
bazel build //... \
  --remote_cache=grpc://nativelink.internal:50051 \
  --remote_instance_name=main
```

Right when you have one CI environment building similar things repeatedly,
when you want shared caching across laptops without operating workers, or
when you're still evaluating. This is
[Getting started](/getting-started).

### Pattern B: Cache plus execution [#pattern-b--cache-plus-execution]

Actions ship to a worker fleet; the cache underneath stores everything.

```bash
bazel build //... \
  --remote_cache=grpc://nativelink.internal:50051 \
  --remote_executor=grpc://nativelink.internal:50052 \
  --remote_instance_name=main \
  --jobs=200
```

`--jobs=200` is the change people forget. Without it Bazel dispatches only
as many actions as your laptop has cores, and the fleet sits idle; the
limiting factor is supposed to be pool size, not local CPU count.

Right when full builds take hours, when you're trying to shrink a CI fleet,
or when you need actions to run on hardware developers don't have.

### Pattern C: Hybrid [#pattern-c--hybrid]

Small actions local, everything else remote, decided by policy.

```bash
bazel build //... \
  --remote_cache=grpc://nativelink.internal:50051 \
  --remote_executor=grpc://nativelink.internal:50052 \
  --remote_instance_name=main \
  --modify_execution_info=Javac=+no-remote-exec \
  --strategy=Javac=local
```

This forces `Javac` local (for very small compilations the round-trip
dominates) while everything else goes remote.

Right when your workload mixes very fast and very slow actions, or when
latency to the fleet is high enough that small actions lose. Reach for it
when you have telemetry showing where time actually goes, not before;
[Tuning](/operate/tuning) covers getting that telemetry.

### Choosing [#choosing]

| Pattern | Setup cost | Where the win comes from | What it costs |
| --- | --- | --- | --- |
| Cache only | Low | Repeated work becomes free | Storage |
| Cache + execution | Medium | Cache misses fan out across a fleet | Worker compute |
| Hybrid | High | Avoids paying round-trips on trivial actions | Worker compute + tuning effort |

The speedup numbers you'll see quoted for these vary by an order of
magnitude between workloads, and mostly reflect how much of a given build is
cache-missing compilation. Measure your own before committing to a shape.

## FAQ [#faq]

<Accordions type="multiple">

<Accordion title="Which example should I start from for a real cluster?" id="which-example-for-a-real-cluster">
`s3_backend_with_local_fast_cas.json5` for the store layer and the
`docker-compose/` topology for the process split; that combination is the
shape [Production configuration](/operate/production-config) describes at
full size. Don't start from `local_rbe_self_test.json5`; it's optimised for
being readable in one screen, not for running a fleet.
</Accordion>

<Accordion title="Are these examples tested?" id="are-these-examples-tested">
Partly. Every file in `nativelink-config/examples/` is parsed by a unit test
(`nativelink-config/tests/json5_test.rs`), so they all load. The
`docker-compose/` deployment is what `run_integration_tests.sh` brings up,
so it is exercised end to end in CI. `local_rbe_self_test.json5` is only
parse-tested, and the backend-specific configs are not validated against
live cloud services, so treat credentials and endpoint syntax as the parts
to double-check.
</Accordion>

<Accordion title="Can I mix patterns per target?" id="can-i-mix-patterns-per-target">
That's what pattern C is. `--strategy` and `--modify_execution_info` take
mnemonic-scoped values, and `exec_properties` on a target routes it
specifically. The global flags are just defaults.
</Accordion>

<Accordion title="What do I need to change to use these against a hosted cluster?" id="what-to-change-for-a-hosted-cluster">
The scheme and the hostname: `grpcs://` instead of `grpc://`, your endpoint,
and credentials. The flag structure and the config shapes don't change.
</Accordion>

<Accordion title="Where do I go once none of these fit?" id="where-do-i-go-once-none-of-these-fit">
[Configuration](/configuration) is about being able to write these
files rather than adapt them, and [How-to guides](/how-to) covers specific
backends and features one at a time.
</Accordion>

</Accordions>

<NextStep href="/configuration" title="Configuration">
  Stop copying examples and write the config yourself.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  The full-size version of the compose topology, with the reasoning.
</NextStep>

# https://docs.nativelink.com/configuration

**Configuration**: The config model on one page, so you can read and write a NativeLink config instead of copying one.

**Who this is for:** anyone running a cluster they got working by copying an
example, who now needs to change it. **What you'll have at the end of this
section:** the ability to read any NativeLink config and write one from an
empty file. **Time:** an afternoon, most of it on the capstone.

<Prerequisites>
  A cluster you got running by copying an example, for instance from [Examples and templates](/remote-execution/examples-and-templates).
</Prerequisites>

## One file, six keys [#one-file-six-keys]

A NativeLink deployment of any shape (one process on a laptop, a sharded
fleet across three regions) is described by a single JSON5 file. It has
exactly six top-level keys, and only two of them are required.

| Key | Required | Shape | What it holds |
| --- | --- | --- | --- |
| `stores` | yes | array | Every storage backend this process can use, each with a name |
| `servers` | yes | array | Listeners, and which services are exposed on each |
| `schedulers` | no | array | Queues that match actions to workers |
| `workers` | no | array | Executors that run actions, in this process |
| `global` | no | object | Process-wide settings: file limits, default digest function |
| `experimental_origin_events` | no | object | Event publishing, still experimental |

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig" />

`schedulers` and `workers` are optional because a cache-only deployment has
neither. That's Getting started: a config with just `stores` and `servers` is a
complete, valid NativeLink cache.

## How a request traverses them [#how-a-request-traverses-them]

<Mermaid>
{`flowchart LR
  client[Build client] -->|"gRPC :50051"| server[servers entry]
  server --> svc{{services}}
  svc -->|cas, ac, bytestream| store[(stores entry<br/>by name)]
  svc -->|execution| sched[schedulers entry<br/>by name]
  sched -->|assigns action| worker[workers entry]
  worker -->|"worker_api :50061"| server2[a second servers entry]
  server2 --> sched
  worker --> store`}
</Mermaid>

Read that diagram as: **the `servers` array is the only entry point, and
everything else is reached from it by name.** A service doesn't contain a
store, it names one. A worker doesn't contain a scheduler, it dials one over
the worker API. The names are the wiring.

This is why the config can describe both a single process and a fleet without
changing shape. Split the same file into three (one with `stores` and a CAS
server, one with `schedulers` and an execution server, one with `workers`)
and the names become network addresses instead of in-process references.
Nothing else changes.

## Everything is a named array [#everything-is-a-named-array]

Every top-level collection is an **array of objects**, not a map keyed by
name. Stores and schedulers carry their `name` beside the type key; workers
carry it inside the `local` block; servers have an optional `name`:

```json5
schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: { cpu_count: "minimum" },
    },
  },
],
```

The `name` sits beside the type key rather than wrapping it, because the
type-specific fields are flattened into the same object.

<Callout type="warn" title="Older material shows these as maps">
  You'll find configs written with the name as a key wrapping the body:
  `schedulers` as an object mapping `MAIN_SCHEDULER` to its scheduler spec,
  rather than as an array. That shape does not parse. If you copied one and
  got an error mentioning `invalid type: map`, this is why.
</Callout>

## A typo is a startup error, not a silent default [#a-typo-is-a-startup-error-not-a-silent-default]

Nearly every struct in the config crate is `#[serde(deny_unknown_fields)]`.
Misspell a key and the process refuses to start and tells you which key it
didn't recognise, rather than ignoring it and behaving strangely three hours
later.

This is worth knowing because it changes how you should edit configs: the
fastest way to check a change is to start the binary. It parses the whole file
before it binds a single socket.

## The pages, in order [#the-pages-in-order]

**Learn the mechanics**

- [The config file](/configuration/config-file): JSON5, environment
  substitution, how the binary finds the file, and how to check one before
  you trust it.

**Learn the four sections**

- [Stores](/configuration/stores): declaring and naming stores, and the one
  idea that unlocks the rest of the system: stores compose.
- [Servers and services](/configuration/servers-and-services): listeners,
  the eleven services, and the port split that keeps workers off the public
  interface.
- [Schedulers and workers](/configuration/scheduler-and-workers): the
  minimum of each, and the properties contract between them.

**Put it together**

- [Your first full config](/configuration/your-first-full-config): an empty
  file to a running cache-and-execution cluster, one block at a time.

## Start here, or skip ahead [#start-here-or-skip-ahead]

**Start here** if you have a cluster running from a copied example and want to
understand what you copied.

**Go back a section** if nothing is running yet.
[Getting started](/getting-started) gets you a cache, and
[Remote execution](/remote-execution) gets you workers; both hand you a
working config you can then read here.

**Skip ahead** to [How-to guides](/how-to) if you know exactly which backend or
feature you want and just need the block that does it.

**Skip ahead** to
[Production configuration](/operate/production-config) if what you actually
want is the shape a real fleet runs, rather than the model behind it.

<NextStep href="/configuration/config-file" title="The config file">
  JSON5, `${VAR:-default}` substitution, and how to find out a config is
  wrong before it takes a cluster down with it.
</NextStep>

<NextStep href="/reference/nativelink-config" kind="aside" title="Configuration reference">
  Generated from the Rust source: every field, every default, every
  constraint. This section is for understanding; that one is for lookups.
</NextStep>

# https://docs.nativelink.com/configuration/config-file

**The config file**: JSON5 mechanics, environment substitution, how the binary is pointed at a config, and how to find out a config is wrong before it takes a cluster down with it.

**Who this is for:** anyone about to edit a NativeLink config for the first
time. **What you'll know at the end:** what dialect the file is written in,
how to keep secrets and addresses out of it, how the binary finds it, and how
to check one is valid without waiting for a production rollout to tell you.
**Time:** twenty minutes.

<Prerequisites>
  A config file you got from an example, for instance from [Examples and templates](/remote-execution/examples-and-templates).
</Prerequisites>

## It's JSON5, not JSON [#its-json5-not-json]

NativeLink parses its config with `serde_json5`, not `serde_json`. The
difference matters every time you edit a file:

```json5
{
  // Comments are legal. Use them: a config is the only place some
  // operational decisions get written down at all.
  stores: [
    {
      name: "CAS_MAIN_STORE", // unquoted keys are legal too
      filesystem: {
        content_path: "/tmp/nativelink/data/content_path-cas",
        temp_path: "/tmp/nativelink/tmp_path-cas",
        eviction_policy: {
          max_bytes: 10000000000, // trailing comma is legal
        },
      },
    },
  ],
}
```

Three things are legal here that plain JSON rejects: `//` and `/* */`
comments, unquoted object keys, and trailing commas. Nearly every shipped
example in `nativelink-config/examples/` uses all three, so a strict-JSON tool
pointed at one of them will report syntax errors that aren't real.

<Callout type="warn" title="Tell your editor it's JSON5">
  Most editors infer the dialect from the extension. `.json5` is the extension
  the repo uses, and it's the one that gets you comment support instead of a
  wall of red squiggles. If you rename a config to `.json` to make a tool
  happy, the tool will be happy and wrong.
</Callout>

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig::try_from_json5_file" />

## Environment substitution [#environment-substitution]

Any string field that is deserialized through the config crate's shellexpand
helpers accepts shell-style variable references, including the
default-if-unset form:

```json5
{
  worker_api_endpoint: {
    uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
  },
}
```

`${SCHEDULER_ENDPOINT:-127.0.0.1}` means "the value of `SCHEDULER_ENDPOINT`,
or `127.0.0.1` if it isn't set". That single feature is what lets one config
file serve a laptop and a cluster: the file names a default that works on your
machine, and the deployment overrides it with an environment variable.

The docker-compose deployment in this repo is built entirely on that pattern.
Its worker and scheduler configs point at
`grpc://${CAS_ENDPOINT:-127.0.0.1}:50051` and
`grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061`, so the same files run
unmodified whether the pieces are on one host or several.

Substitution also works for numeric-looking fields that take a size, because
those are parsed from strings:

```json5
{
  memory: {
    eviction_policy: {
      max_bytes: "${NATIVELINK_CAS_MEMORY_CONTENT_LIMIT:-100mb}",
    },
  },
}
```

<SourceLink file="nativelink-config/src/serde_utils.rs" symbol="convert_string_with_shellexpand" />

<Callout type="info" title="Not every field expands">
  Substitution is opt-in per field, applied by the deserializer the field
  declares. The endpoint, path, and size fields you're most likely to want to
  parameterise do expand; an arbitrary field may not. If a `${VAR}` shows up
  verbatim in an error message or a log line, that field isn't one of them.
</Callout>

## How the binary finds the config [#how-the-binary-finds-the-config]

One way only, and it is positional:

```sh
nativelink /path/to/config.json5
```

No `--config` flag, no search path, no default location, no environment
variable naming the file. The single positional argument is the whole
interface.

<SourceLink file="src/bin/nativelink.rs" symbol="Args" />

That has one practical consequence worth internalising: **a container image
does not carry its config**. Every deployment recipe in this repo mounts a
config file in and passes its path as the argument. If you're writing a
Kubernetes manifest, the config is a ConfigMap you mount, not something baked
into the image.

## Checking a config before you trust it [#checking-a-config-before-you-trust-it]

Here is the honest answer, because the alternative wastes an afternoon:
**there is no `--validate` flag and no dry-run mode.** The binary takes a
config path and starts.

What saves you is the order in which it starts. `main()` reads and parses the
entire config, and only afterwards raises the open-file limit, constructs the
stores, and binds any socket, so a config that fails to parse is rejected
before the process has touched a port, which makes "run it and see" a
genuinely safe validation step even on a machine where the real ports are
already in use by something else.

Combined with `#[serde(deny_unknown_fields)]`, which is on nearly every
struct in the config crate, that gives you a fast, precise loop:

<Steps>

<li>

**Start the binary against the config.**

```sh
nativelink ./my-config.json5
```

</li>

<li>

**Read the first error, not the last.** A parse failure names the offending
key and its position. A misspelled `evicton_policy` produces an
`unknown field` error naming `evicton_policy` and listing the fields that
were expected, not a silent default and a cache that never evicts.

</li>

<li>

**Stop it once it's listening.** Once the process reports its listeners are
up, the config parsed. Everything after that point is runtime behaviour, not
config validity.

</li>

</Steps>

<VerifyBlock>

- A deliberately misspelled key makes the process exit before it prints
  anything about listeners.
- Fixing the spelling makes it start.
- The failing run did not need a free port, because it never reached the
  bind.

</VerifyBlock>

## Editor validation, if you want it [#editor-validation-if-you-want-it]

The config crate can emit a JSON Schema for the whole `CasConfig` type, which
editors will use for completion and inline errors:

```sh
cargo run --bin build-schema --features dev-schema --package nativelink-config
```

That writes `nativelink_config.schema.json` into the working directory. The
file is **not committed to the repo**; it's generated on demand, and it's
generated from the same Rust types the binary parses with, so it can't drift
from them. Point your editor's JSON5 schema mapping at the file you generated
and misspellings show up as you type rather than at startup.

<Callout type="info" title="Generated, not maintained">
  Because the schema comes from `schema_for!(CasConfig)`, regenerating it after
  you change NativeLink versions is the whole upgrade procedure. There's no
  hand-written schema to fall out of date.
</Callout>

## FAQ [#faq]

<Accordions>

<Accordion title="Can I split the config across multiple files?" id="can-i-split-the-config-across-multiple-files">

Not with an include directive; the parser reads exactly one file. What you do
instead is split by *role*: one config for the CAS process, one for the
scheduler, one for the workers, each a complete file naming the others by
network address. That's the same split described in
[the section overview](/configuration), and it's how the docker-compose
deployment is laid out.

</Accordion>

<Accordion title="How do I keep secrets out of the file?" id="how-do-i-keep-secrets-out-of-the-file">

Use environment substitution for the fields that carry them and set the
variables from your platform's secret mechanism: a Kubernetes Secret mounted
as environment variables, a systemd `EnvironmentFile`, or your process
supervisor's equivalent. The config file itself then contains a variable name,
which is safe to commit.

</Accordion>

<Accordion title="Why did my config work as JSON but not as JSON5, or vice versa?" id="why-did-my-config-work-as-json-but-not-as-json5-or-vice-versa">

Every valid JSON document is also a valid JSON5 document, so that direction
never breaks. The other direction does: a config using comments or trailing
commas will be rejected by any strict-JSON tooling you point at it. If a
linter, formatter, or CI check complains about a file that NativeLink accepts
happily, the tool is the thing that's misconfigured.

</Accordion>

<Accordion title="The process starts, but nothing works. Was the config valid?" id="the-process-starts-but-nothing-works-was-the-config-valid">

Yes, and that's the boundary worth knowing. Parsing checks structure and
spelling. It does not check that a store name referenced by a service exists
in a useful state, that a worker can reach the scheduler address you gave it,
or that a path is writable. Those are runtime failures, and they show up in
logs rather than at startup. [Observability](/operate/observability) is where
you go to read them.

</Accordion>

</Accordions>

<NextStep href="/configuration/stores" title="Stores">
  Declaring stores, naming them, and the idea that unlocks the rest of the
  config: stores compose, and one store can refer to another by name.
</NextStep>

<NextStep href="/reference/nativelink-config" kind="aside" title="Configuration reference">
  Every field and default, generated from the same Rust types described here.
</NextStep>

# https://docs.nativelink.com/configuration/stores

**Stores**: Declaring stores, naming them, and the idea that unlocks the rest of the config: stores compose, and one store refers to another by name.

**Who this is for:** anyone who has seen a NativeLink config with a store
nested five levels deep and wanted to know why. **What you'll know at the
end:** how to declare a store, how to name one, how composition works, and
which layers are worth reaching for. **Time:** thirty minutes.

<Prerequisites>
  The config model from [The config file](/configuration/config-file).
</Prerequisites>

## A store is one entry in an array [#a-store-is-one-entry-in-an-array]

The `stores` array holds every storage backend the process can use. Each entry
is an object with a `name` and exactly one type key:

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    memory: {
      eviction_policy: { max_bytes: 1000000000 }, // 1 GB
    },
  },
  {
    name: "AC_MAIN_STORE",
    memory: {
      eviction_policy: { max_bytes: 100000000 }, // 100 MB
    },
  },
],
```

That's a complete, working cache backend pair: a content-addressable store for
blobs and an action cache for results. Everything lives in RAM, so a restart
wipes it, which is exactly right for a demo or a short-lived CI runner, and
exactly wrong for anything else.

The name is arbitrary. `CAS_MAIN_STORE` and `AC_MAIN_STORE` are conventions
the shipped examples use, not keywords; the services in your `servers` block
will refer to whatever names you pick.

## Backing it with disk [#backing-it-with-disk]

Swap the type key and the same two stores survive a restart:

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/cas",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 50000000000 }, // 50 GB
    },
  },
  {
    name: "AC_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/ac",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 500000000 }, // 500 MB
    },
  },
],
```

`content_path` is where the bulk of the data lives. `temp_path` is a staging
area for in-flight writes and deletes, and it must be **on the same block
device** as `content_path` so a completed write can be moved into place
atomically rather than copied. Everything in `temp_path` is deleted on every
startup.

On boot, `content_path` is scanned and every file found is added back to the
cache; that scan is what makes the store durable across restarts.

<Callout type="warn" title="Set an eviction policy or the store grows forever">
  `eviction_policy` is optional in the schema and unbounded when omitted. A
  store without one never removes anything, which for a `memory` store means
  the process grows until the kernel kills it, and for a `filesystem` store
  means the disk fills. Set `max_bytes` on every store you declare.
</Callout>

<Callout type="info" title="Pick a filesystem that handles many small files">
  CAS blobs are typically small: function-level outputs, header files, link
  arguments. Default ext4 works fine; a filesystem tuned for many small files
  degrades more gracefully under load.
</Callout>

<SourceLink file="nativelink-config/src/stores.rs" symbol="FilesystemSpec" />

## Stores compose [#stores-compose]

Here is the idea the rest of this page is about. **Most store types don't
store anything.** They wrap another store and change its behaviour, and the
store they wrap is written inline, as a full store spec:

```json5
{
  name: "CAS_MAIN_STORE",
  compression: {
    compression_algorithm: { lz4: {} },
    backend: {
      filesystem: {
        content_path: "/var/lib/nativelink/cas",
        temp_path: "/var/lib/nativelink/tmp",
        eviction_policy: { max_bytes: 100000000000 }, // 100 GB
      },
    },
  },
},
```

`compression` has no storage of its own. It compresses on write, decompresses
on read, and hands the bytes to whatever is in `backend`. The client sees raw
bytes either way. Build artifacts are usually highly compressible, so this
saves disk in exchange for CPU; the only algorithm offered is `lz4`, which is
fast and gives up early on data that won't compress.

Because the wrapped store is a full spec rather than a reference to a type,
wrappers nest arbitrarily. A `verify` around a `size_partitioning` whose upper
half is a `dedup` whose index is a `fast_slow` of memory-over-disk is not an
exotic config; it's `nativelink-config/examples/filesystem_cas.json5`, and
each layer is doing one comprehensible job.

## `ref_store`: naming instead of nesting [#ref_store-naming-instead-of-nesting]

Nesting alone can't express *sharing*. If two stores should write to the same
place, inlining the backend twice gives you two independent stores that
happen to have identical settings.

`ref_store` is the escape hatch. It's a store type whose entire body is the
name of another top-level store:

```json5
stores: [
  {
    name: "FS_CONTENT_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/content",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 2000000000 },
    },
  },
  {
    name: "CAS_MAIN_STORE",
    size_partitioning: {
      size: 262144, // 256 KB
      lower_store: { ref_store: { name: "FS_CONTENT_STORE" } },
      upper_store: { ref_store: { name: "FS_CONTENT_STORE" } },
    },
  },
],
```

Both halves now land in the same physical store. Swap either `ref_store` for
an inline spec and they diverge.

<SourceLink file="nativelink-config/src/stores.rs" symbol="RefSpec" />

Two properties of `ref_store` are worth knowing before you use it:

**Order doesn't matter.** A store can reference a name declared later in the
array. `nativelink-config/examples/redis.json5` has `AC_MAIN_STORE` referring
to `CAS_MAIN_STORE`, which is declared below it. References are resolved in a
post-init pass after every store has been constructed.

**A typo is not a parse error.** `deny_unknown_fields` catches a misspelled
*field*, but `ref_store: { name: "CAS_MAIN_STROE" }` is structurally valid;
the failure surfaces in that post-init pass (`Failed to find store
'CAS_MAIN_STROE' in StoreManager in RefStore`), still at startup and still
before any socket is bound, but after the parse itself has succeeded.

<SourceLink file="nativelink-store/src/ref_store.rs" symbol="post_init" />

## The layers worth knowing [#the-layers-worth-knowing]

You don't need all eighteen store types to write a good config. These are the
ones that appear in nearly every non-trivial one:

| Type | What it does | Why you'd add it |
| --- | --- | --- |
| `filesystem` | Stores blobs on disk | Durability across restarts |
| `memory` | Stores blobs in RAM | Speed, for small hot data |
| `compression` | Compresses around a backend | Disk savings on build artifacts |
| `dedup` | Splits blobs into chunks, stores each once | Large, similar artifacts |
| `fast_slow` | Reads from `fast`, falls back to `slow` | A RAM tier over a disk tier |
| `size_partitioning` | Routes by blob size to two stores | Keep small objects out of an index |
| `verify` | Checks hash and size on write | Catch corruption at the boundary |
| `completeness_checking` | Rejects AC entries whose blobs are gone | Correct action-cache results |
| `existence_cache` | Remembers what the backend already has | Skip redundant existence checks |
| `ref_store` | Names another top-level store | Sharing, instead of duplicating |
| `shard` | Spreads across weighted backends | Horizontal capacity |
| `noop` | Discards everything | A required slot you don't want to fill |

Two of these are load-bearing for correctness rather than performance.
**`verify`** with `verify_hash: true` catches a blob whose content doesn't
match the digest it was uploaded under. **`completeness_checking`** wraps an
action cache and checks the CAS still holds every blob a cached result refers
to; without it, an eviction in the CAS turns cached results into builds that
fail on a missing output.

That's why most of the shipped backend examples (`filesystem_cas.json5`,
`s3_backend_with_local_fast_cas.json5`, and the other object-storage examples)
wrap the CAS in `verify`, why `redis.json5` wraps the AC in
`completeness_checking`, and why you should do both.

The full list, with every field and default, is in the
[store reference](/reference/nativelink-config/store-overview). Recipes for
specific backends (object storage, Redis, OCI registries) live in
[How-to guides](/how-to).

<Callout type="warn" title="There is no `s3` store type">
  Cloud object storage is one store type, `experimental_cloud_object_store`,
  with a `provider` field selecting between `aws`, `gcs`, `azure`, `ontap`,
  `r2`, and `oci`. Configs and older docs showing a top-level `s3` key predate
  that consolidation and will not parse.
</Callout>

## What keys the store [#what-keys-the-store]

A store is content-addressed, so what goes in a key is the digest, and every
server, worker, and build client sharing a cache must compute that digest the
same way. SHA-256 and BLAKE3 produce different keys for identical bytes, so
uploads made under one function are cache *misses* for clients using the
other.

NativeLink defaults to SHA-256. Declare it explicitly anyway, in the `global`
block, so the file says what it depends on:

```json5
{
  global: {
    max_open_files: 24576,
    default_digest_hash_function: "sha256",
  },
  // ... stores, servers, schedulers, workers ...
}
```

Then match it on the client. For Bazel, in `.bazelrc`:

```text
startup --digest_function=sha256
```

NativeLink accepts uploads whose client-declared digest function differs from
the server default, but logs a warning the first time it sees each mismatch:
mixed digest functions divide the cache and reduce shared hits.

<SourceLink file="nativelink-service/src/bytestream_server.rs" symbol="reported_digest_function_mismatches" />

`max_open_files` is in that block for a related reason: a `filesystem` store
holds file descriptors. At startup NativeLink tries to raise its own soft
`nofile` limit to this value (default 24576) and sizes an internal semaphore
from whatever the kernel actually granted, so if the hard limit is lower the
effective value is lower too, and a warning is logged.

<SourceLink file="nativelink-util/src/fs.rs" symbol="set_open_file_limit" />

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="GlobalConfig" />

### Which function to pick [#which-function-to-pick]

We compared both algorithms on an Apple M2 Max with 12 CPU cores and 32 GB of
memory, running macOS 15.6 and Bazel 9.1.1. Each of two alternating runs built
`//:nativelink` against a fresh local NativeLink filesystem cache with hash and
size verification enabled. Uploads were synchronous, and the Bazel output and
NativeLink cache directories were isolated by algorithm.

| Phase | SHA-256 runs | SHA-256 mean | BLAKE3 runs | BLAKE3 mean |
| --- | --- | ---: | --- | ---: |
| Compile and populate an empty cache | 179.90 s, 180.18 s | 180.04 s | 181.56 s, 185.57 s | 183.57 s |
| Rebuild from the populated cache | 6.39 s, 5.93 s | 6.16 s | 6.74 s, 6.41 s | 6.58 s |

Both population builds performed 2,620 actions. Each cached rebuild reported
1,458 remote cache hits. SHA-256 was 1.9% faster while populating the cache and
6.3% faster on the cached rebuild in this test, so SHA-256 remains the default.

These are whole-build measurements from one macOS machine, not a universal
hashing benchmark; compilation, storage, scheduling, and operating-system
caching dominate much of the elapsed time. Measure your own representative
workload before changing algorithms. Keeping every participant aligned matters
far more than a small isolated timing difference.

## FAQ [#faq]

<Accordions>

<Accordion title="What happens when a store fills up?" id="what-happens-when-a-store-fills-up">

`eviction_policy.max_bytes` caps it, and once full the least-recently-used
entries are evicted to make room. An evicted CAS entry is just a future cache
miss; the client rebuilds and re-uploads. An evicted entry that a cached
action result still points at is the problem `completeness_checking` exists to
solve.

</Accordion>

<Accordion title="Do I need a separate store for the CAS and the action cache?" id="do-i-need-a-separate-store-for-the-cas-and-the-action-cache">

Yes, and the config loader enforces it: if a `cas` and an `ac` entry with the
same `instance_name` name the same store, startup fails with `CAS and AC use
the same store '...' in the config`. Beyond that check, they have very
different shapes. The CAS holds many large-ish blobs and wants compression and
deduplication; the AC holds small metadata records, wants none of that, and
wants `completeness_checking`.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="check_store_conflict" />

</Accordion>

<Accordion title="Why is `fast_slow` required for workers?" id="why-is-fast_slow-required-for-workers">

A worker materialises action inputs onto disk by hard-linking out of its
store, which only works if the store is filesystem-backed. `fast_slow` with a
`filesystem` fast tier gives it that, and lets the slow tier be a shared
remote CAS, or a `noop`, when the fast tier *is* the storage. `basic_cas.json5`
takes the second option.

</Accordion>

<Accordion title="Can two NativeLink processes share one filesystem store?" id="can-two-nativelink-processes-share-one-filesystem-store">

No. A `filesystem` store owns its directories: it scans them at boot and
manages eviction against its own view of what's there. Two processes pointed
at the same paths will evict each other's data. Share by putting a real shared
backend behind them (object storage, Redis) or by having one process serve the
CAS over gRPC to the others.

</Accordion>

</Accordions>

<NextStep href="/configuration/servers-and-services" title="Servers and services">
  Listeners, the eleven services, and the port split that keeps the worker API
  off your public interface.
</NextStep>

<NextStep href="/reference/nativelink-config/store-overview" kind="aside" title="Store reference">
  All eighteen store types, every field, every default.
</NextStep>

# https://docs.nativelink.com/configuration/servers-and-services

**Servers and services**: Listeners, the eleven services, instance names, and the port split that keeps the worker API off your public interface.

**Who this is for:** anyone deciding what a NativeLink process should expose,
and on which port. **What you'll know at the end:** what a listener is, what
each of the eleven services does, how `instance_name` partitions a cache, and
why the worker API must not share a port with anything else. **Time:** thirty
minutes.

<Prerequisites>
  Named stores you can refer to. See [Stores](/configuration/stores).
</Prerequisites>

## A server is a listener plus a set of services [#a-server-is-a-listener-plus-a-set-of-services]

The `servers` array is the only entry point into a NativeLink process.
Each entry binds one socket and decides what is reachable on it:

```json5
servers: [
  {
    name: "public",
    listener: {
      http: {
        socket_address: "0.0.0.0:50051",
      },
    },
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      capabilities: [{ instance_name: "main" }],
    },
  },
],
```

Four things are going on there, and each is worth naming.

`name` is documented as a label for telemetry and logs, defaulting to the
server's index in the array. Set it; it's the cheapest documentation in the
file.

`listener` currently has exactly one variant, `http`, which serves
HTTP/HTTP2 and therefore gRPC. `socket_address` is parsed as a Rust
`SocketAddr`, so it must be an IP literal plus port: `0.0.0.0:50051` to bind
every IPv4 interface, `[::]:50051` for IPv6, `127.0.0.1:50051` for loopback
only. A hostname or a bare `:port` is rejected as an invalid address. TLS,
HTTP/2 tuning, and compression are options on the listener, not on the
services.

`services` is optional in the schema, but the binary refuses to start a
server without it (`'services' must be configured`), so in practice every
server entry carries one.

<SourceLink file="src/bin/nativelink.rs" symbol="inner_main" />

Each service entry names a store or a scheduler as a **string**, the name
from the `stores` or `schedulers` array. This is where the wiring described in
[the section overview](/configuration) actually happens.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="ServerConfig" />

## The eleven services [#the-eleven-services]

| Service | What it serves | Wants |
| --- | --- | --- |
| `cas` | REAPI `ContentAddressableStorage`: blob upload and lookup | `cas_store` |
| `ac` | REAPI `ActionCache`: cached action results | `ac_store` |
| `bytestream` | Streaming reads and writes for large blobs | `cas_store` |
| `capabilities` | What this endpoint supports; required by Bazel | optionally a scheduler |
| `execution` | REAPI `Execution`: accepts `Execute` calls | `cas_store` and `scheduler` |
| `fetch` | Remote Asset protocol, fetch half | `fetch_store` |
| `push` | Remote Asset protocol, push half | `push_store` |
| `worker_api` | Where workers connect and are handed actions | `scheduler` |
| `experimental_bep` | Build Event Protocol consumption | `store` |
| `admin` | REST endpoint for administrative tasks | nothing (path defaults to `/admin`) |
| `health` | Health check endpoint | nothing (path defaults to `/status`) |

A cache-only deployment needs the first four. Adding execution adds
`execution` and `worker_api`. The rest are opt-in.

`capabilities` is the one that surprises people: it carries almost no config,
but Bazel queries it before doing anything else and fails the build if the
call is not served, with an error about querying capabilities rather than one
naming a missing service. And if `capabilities` has no `remote_execution`
block, the response carries no `execution_capabilities`, so the client is
told this endpoint caches but does not execute, which is correct for a
cache-only Getting started deployment and a silent bug once you add Remote
execution.

```json5
capabilities: [
  {
    instance_name: "main",
    remote_execution: { scheduler: "MAIN_SCHEDULER" },
  },
],
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="ServicesConfig" />

## Instance names [#instance-names]

Most services are declared as an **array**, and each element carries an
`instance_name`. That field is a routing key: it selects which store (or
scheduler) serves a request. The instance name is not part of the storage
key, so two instance names pointing at *different* stores are fully isolated,
and two pointing at the *same* store share every blob.

```json5
cas: [
  { instance_name: "", cas_store: "CAS_MAIN_STORE" },
  { instance_name: "main", cas_store: "CAS_MAIN_STORE" },
],
```

That pattern (the same store registered twice, once under the empty name and
once under `main`) appears throughout the shipped examples, and it relies on
the sharing: clients differ in what they send, some omit the instance name
entirely, and registering both means either kind of client works, against one
cache, without a config change on their side.

Pick one name per logical environment. `main` for the production cache,
`<repo-name>` for per-repo isolation, `<something>-experiments` for a sandbox
that shouldn't pollute the real cache, each pointed at its own store. Every
client targeting a given cache must use the same one.

<Callout type="warn" title="Unregistered instance names fail every request">
  A client using an instance name the server has not registered gets an
  error on every CAS and AC call (`'instance_name' not configured for
  '<name>'`), not a silently empty cache. Bazel surfaces that as remote cache
  warnings and zero hits. If a new client sees 0% hit rate against a cache
  that's demonstrably working, compare instance names before anything else.
</Callout>

<SourceLink file="nativelink-service/src/cas_server.rs" symbol="CasServer" />

## The port split [#the-port-split]

The `worker_api` service is different from every other service on this page,
and the difference is a security boundary rather than a design preference.

A client calling the CAS can upload and fetch blobs. A worker connected to the
worker API is *handed actions to execute* and reports their results as
authoritative. Those are not the same permission, and the worker API has no
authentication of its own to distinguish them.

The rule: **`worker_api` gets its own listener, on its own port, with no
other service sharing it.** The shipped examples put the public API on
`:50051` and the worker API on `:50061`, and the second port is the one you
never expose beyond the cluster:

```json5
servers: [
  {
    name: "public",
    listener: { http: { socket_address: "0.0.0.0:50051" } },
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      execution: [{
        instance_name: "main",
        cas_store: "CAS_MAIN_STORE",
        scheduler: "MAIN_SCHEDULER",
      }],
      capabilities: [{
        instance_name: "main",
        remote_execution: { scheduler: "MAIN_SCHEDULER" },
      }],
    },
  },
  {
    name: "private_workers_servers",
    listener: { http: { socket_address: "0.0.0.0:50061" } },
    services: {
      worker_api: { scheduler: "MAIN_SCHEDULER" },
      admin: {},
      health: {},
    },
  },
],
```

Note that `worker_api` takes a single object rather than an array; it has no
`instance_name`, because it isn't part of the client-facing protocol at all.

<Callout type="warn" title="Do not put worker_api on your public port">
  The upstream config comment is unusually direct about this: serving
  `worker_api` alongside client services "is a security risk, as workers have
  a different permission set than a client that makes the remote
  execution/cache requests." Treat the worker port as an internal backend API:
  bound to a private interface, firewalled, or reachable only inside the
  cluster network.
</Callout>

`admin` and `health` sit on the private port in the examples for the same
reason. A build event protocol endpoint (`experimental_bep`), when you enable
it, can go on its own listener so it can be scaled and secured independently
of both.

## TLS, when you need it [#tls-when-you-need-it]

TLS is a property of the listener:

```json5
listener: {
  http: {
    socket_address: "0.0.0.0:50071",
    tls: {
      cert_file: "/etc/nativelink/tls/server.crt",
      key_file: "/etc/nativelink/tls/server.key",
    },
  },
},
```

Adding `client_ca_file` turns that into mutual TLS (clients must present a
certificate signed by that CA) and `client_crl_file` lets you revoke one.
That's the closest thing NativeLink has to built-in client authentication, and
it is a reasonable answer for a cache exposed beyond a trusted network.

<Callout type="info" title="The docker-compose example ships example certs">
  `deployment-examples/docker-compose/` includes a TLS listener using files
  named `example-do-not-use-in-prod-rootca.crt` and
  `example-do-not-use-in-prod-key.pem`. The names are the documentation. Use
  them to see the shape, never to serve anything.
</Callout>

## FAQ [#faq]

<Accordions>

<Accordion title="Can one process have more than two servers?" id="can-one-process-have-more-than-two-servers">

Yes; the array is unbounded, and separating concerns by port is the intended
use. `deployment-examples/docker-compose/local-storage-cas.json5` serves the
same cache twice, plain on `:50051` and TLS on `:50071`. A third listener for
build events, or a plain-HTTP health endpoint a load balancer can probe
without TLS, follows the same pattern.

</Accordion>

<Accordion title="Why is `bytestream` separate from `cas`?" id="why-is-bytestream-separate-from-cas">

REAPI splits them deliberately. The `cas` service handles batched reads and
writes of small blobs; the ByteStream service streams large ones. Bazel
strongly prefers the streaming interface for anything sizeable, so a cache
that serves `cas` but not `bytestream` will appear to work and then fail on
your first large artifact. Point both at the same store.

</Accordion>

<Accordion title="What does the `admin` service actually do?" id="what-does-the-admin-service-actually-do">

It exposes a small REST endpoint for administrative operations at a
configurable path, defaulting to `/admin`. It is not an authenticated control
plane; it belongs on the private port with the worker API, not on anything
reachable from outside the cluster.

</Accordion>

<Accordion title="Do I need `health` if I have a load balancer?" id="do-i-need-health-if-i-have-a-load-balancer">

You need it *because* you have a load balancer. It serves `/status` by default
and takes a `timeout_seconds` for the checks it runs. Without it, a probe can
only test whether the port accepts a TCP connection, which stays true long
after a process has stopped being useful.

</Accordion>

<Accordion title="I set `read_only: true` on the action cache. What does that stop?" id="i-set-read_only-true-on-the-action-cache-what-does-that-stop">

Clients can read cached action results but cannot write new ones. It's the
right setting for an endpoint you expose to untrusted or unverified builds
(they benefit from the cache without being able to poison it) while a trusted
CI endpoint on a different port writes to the same store.

</Accordion>

</Accordions>

<NextStep href="/configuration/scheduler-and-workers" title="Schedulers and workers">
  The minimum config for each, and the properties contract that decides
  whether an action ever gets picked up.
</NextStep>

<NextStep href="/operate/observability" kind="aside" title="Observability">
  Once the ports are right, this is how you find out what's happening on them.
</NextStep>

# https://docs.nativelink.com/configuration/scheduler-and-workers

**Schedulers and workers**: The minimum config for each, the properties contract between them, and the fields that decide what happens when something goes wrong.

**Who this is for:** anyone adding execution to a config that currently only
caches. **What you'll know at the end:** the minimum scheduler and worker
blocks, how the two are wired to each other, and which of the many optional
fields actually change behaviour you'll notice. **Time:** thirty minutes.

<Prerequisites>
  A config whose `stores` and `servers` you understand. See [Stores](/configuration/stores) and [Servers and services](/configuration/servers-and-services).
</Prerequisites>

## The minimum scheduler [#the-minimum-scheduler]

A scheduler is one entry in the `schedulers` array, with a name and a type:

```json5
schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: {
        cpu_count: "minimum",
        OSFamily: "exact",
        "container-image": "priority",
      },
    },
  },
],
```

`simple` is the scheduler you want. The other four types (`grpc`,
`cache_lookup`, `property_modifier`, and `historical_resource`) either
forward to another scheduler or wrap one to change its behaviour; none of them
is a starting point.

`supported_platform_properties` is the whole contract. It declares which
property keys this scheduler understands and how each is matched:

| Value | Meaning |
| --- | --- |
| `minimum` | Parsed as a number; a worker must advertise at least this much |
| `exact` | Parsed as a string; must match the worker's value exactly |
| `priority` | Not used for filtering; passed to the worker as information |
| `ignore` | Actions may request the key; workers need not advertise it |

A property key an action requests that is **not** in this map is an error. A
key that is in the map but that no worker advertises means the action never
matches anything and queues silently. That failure mode gets its own page,
[Platform properties](/remote-execution/platform-properties), because it's
the first wall almost everyone hits.

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="SimpleSpec" />

## The minimum worker [#the-minimum-worker]

A worker is one entry in the `workers` array. `local` is currently the only
type; it means the worker runs actions in this process, on this machine:

```json5
workers: [
  {
    local: {
      name: "WORKER_1",
      worker_api_endpoint: {
        uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
      },
      cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
      upload_action_result: {
        ac_store: "AC_MAIN_STORE",
      },
      work_directory: "/tmp/nativelink/work",
      platform_properties: {
        cpu_count: { values: ["16"] },
        OSFamily: { values: ["linux"] },
        "container-image": { values: [""] },
      },
    },
  },
],
```

Four of those fields are load-bearing.

**`worker_api_endpoint`** is the address of the *scheduler's* `worker_api`
service, the private port from
[Servers and services](/configuration/servers-and-services), not the public
one. The worker dials out; the scheduler never dials in. That direction is why
a worker fleet can sit behind NAT with no inbound rules.

**`cas_fast_slow_store`** must name a `fast_slow` store whose `fast` half is a
`filesystem` store. The worker builds each action's input tree by hard-linking
files out of that fast store into the work directory, which requires real
files on a real filesystem. The `slow` half must eventually resolve to the
same CAS the client and scheduler use, or be a `noop`, when the fast tier is
the only storage.

**`work_directory`** must be on the **same filesystem** as the fast store's
`content_path`, for the same hard-linking reason. It is fully managed by the
worker and purged on startup.

**`platform_properties`** is the worker's half of the contract. Each key is
either a static list of values or a `query_cmd` that is run (as a command, not
through a shell) each time the worker connects to the scheduler, with its
output split on newlines:

```json5
platform_properties: {
  cpu_count: { query_cmd: "nproc" },
  OSFamily: { values: ["linux"] },
},
```

Every key the scheduler declares as `minimum` or `exact` must appear here, or
this worker will never be matched for any action requesting it.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

<Callout type="warn" title="`upload_action_result` is one level, not two">
  The correct shape is `upload_action_result: { ac_store: "AC_MAIN_STORE" }`.
  Configs showing `upload_action_result: { upload_action_result: { ... } }`
  are wrong and will be rejected; `ac_store` is a direct field. Omitting the
  block entirely is also valid, and means the worker runs actions but never
  caches their results, which looks exactly like a broken cache.
</Callout>

## How they find each other [#how-they-find-each-other]

Nothing in the scheduler names a worker. Workers are not declared to the
scheduler at all; they connect to it, announce their platform properties, and
are added to the pool. Removing a worker means stopping it.

That is what the three pieces of wiring add up to:

- the `execution` service names the scheduler, so clients can submit actions;
- the `worker_api` service names the same scheduler, on a different port, so
  workers can join;
- each worker names the `worker_api` address, so it knows where to join.

In a single-process config all three are the same process and the endpoint is
`127.0.0.1`. In a fleet, the scheduler is its own deployment and that endpoint
is a service address. The config shape does not change.

## The fields that change what you'll notice [#the-fields-that-change-what-youll-notice]

`SimpleSpec` has ten optional fields. Most can stay at their defaults
forever. These four are the ones worth setting deliberately:

**`worker_timeout_s`** (default 5 s): how long a silent worker stays in the
pool. Five seconds is aggressive for a fleet on a congested network; a worker
evicted mid-action has its action re-queued.

**`max_action_executing_timeout_s`** (default 0, disabled): caps how long an
action can sit in `Executing` with no update, even from a worker that is
otherwise healthy. This is the setting that catches a worker stuck on one
specific action rather than dead. Without it, such an action hangs until a
client gives up.

**`max_job_retries`** (default 3): how many times an action that fails with
an internal error is retried before the last error is returned to the client.
It exists to stop one poisonous action from cycling through and destabilising
your whole fleet.

**`retain_completed_for_s`** (default 60 s): how long a completed action's
result stays available for a late `WaitExecution` call. Clients that
disconnect and reconnect need this window to be longer than their reconnect
time.

On the worker side, **`max_inflight_tasks`** (default 0, meaning unlimited) is
the one to set on any real machine. Unlimited means the worker accepts every
action the scheduler offers and lets the OS sort out the contention.

<Callout type="info" title="Scheduler state is in memory by default">
  `experimental_backend` defaults to memory, so restarting the scheduler loses
  the queue. A Redis backend exists for shared state across scheduler
  replicas. Both are covered in
  [Production configuration](/operate/production-config); for a single
  scheduler, the default is correct.
</Callout>

## Putting the pieces together [#putting-the-pieces-together]

Adding execution to a cache-only config takes four edits, all of which you've
now seen:

<Steps>

<li>

**Add a `schedulers` array** with one `simple` entry, naming the platform
properties you intend to match on.

</li>

<li>

**Add `execution` to the public server's services**, naming the CAS store and
the scheduler.

</li>

<li>

**Add a second server on a private port** with `worker_api` naming the same
scheduler, and nothing else client-facing on that port.

</li>

<li>

**Add a `workers` array** with one `local` entry pointing at that private
port, plus the `fast_slow` store it needs.

</li>

</Steps>

<VerifyBlock>

- The worker logs `Worker registered with scheduler` with its assigned worker
  id.
- A build submitted with matching platform properties reports actions
  executing remotely rather than queuing.
- Every key the scheduler declares `minimum` or `exact` appears in the
  worker's `platform_properties`.

</VerifyBlock>

[Your first full config](/configuration/your-first-full-config) does exactly
this, from an empty file, one block at a time.

## FAQ [#faq]

<Accordions>

<Accordion title="Can one process be both scheduler and worker?" id="can-one-process-be-both-scheduler-and-worker">

Yes, and that's what `nativelink-config/examples/basic_cas.json5` is: one
process with stores, a scheduler, a worker, a public server on `:50051` and a
private one on `:50061`. It's the right shape for a laptop and for testing.
Splitting into separate processes is a deployment change, not a config
rewrite; the same blocks move into separate files.

</Accordion>

<Accordion title="How do I run more than one worker?" id="how-do-i-run-more-than-one-worker">

Run more worker processes. The `workers` array can hold several entries in one
process, but they share the machine, so the useful form of scaling is more
machines each running a worker config that dials the same scheduler. Nothing
on the scheduler side changes.

</Accordion>

<Accordion title="What does `entrypoint` do?" id="what-does-entrypoint-do">

It wraps every action. The configured command is prepended to the action's own
command, so an `entrypoint` of `run.sh` and an action of `sleep 5` executes
`run.sh sleep 5`. It's how you put actions inside a sandbox, a container, or a
timeout wrapper without the client knowing.

</Accordion>

<Accordion title="Why did my action fail with a timeout the client didn't ask for?" id="why-did-my-action-fail-with-a-timeout-the-client-didnt-ask-for">

`max_action_timeout_s` on the worker defaults to 20 minutes. An action that
requests no timeout gets that limit; an action that requests a *longer* one is
rejected outright with `InvalidArgument: Action timeout of N seconds is
greater than the maximum allowed timeout`. Either way, an action legitimately
needing longer needs this raised on the worker, not just on the client.

<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="max_action_timeout" />

</Accordion>

<Accordion title="What is `experimental_precondition_script` for?" id="what-is-experimental_precondition_script-for">

It runs before every action, and a non-zero exit pauses the worker; it stops
accepting new work until the script succeeds. The intended use is resource
gating: don't take another action while disk is nearly full or a shared
resource is saturated.

</Accordion>

</Accordions>

<NextStep href="/configuration/your-first-full-config" title="Your first full config">
  Empty file to a running cache-and-execution cluster, one block at a time,
  with a check after each.
</NextStep>

<NextStep href="/remote-execution/platform-properties" kind="aside" title="Platform properties">
  The matching contract in depth, and how to diagnose a queue that never
  drains.
</NextStep>

# https://docs.nativelink.com/configuration/your-first-full-config

**Your first full config**: An empty file to a running cache-and-execution cluster, one block at a time, checking after each.

**Who this is for:** anyone who has read the rest of this section and wants to
prove it by writing a config from nothing. **What you'll have at the end:** a
single-process NativeLink serving a cache and executing actions, from a file
you wrote yourself and can explain line by line. **Time:** about an hour.

<Prerequisites>
  The four previous pages in [Configuration](/configuration): the config file, stores, servers and services, and the scheduler and workers.
</Prerequisites>

Everything here runs in one process on one machine. That's deliberate: it
keeps every name resolvable in one file, so when something is wrong the
feedback is immediate. The same blocks split across machines unchanged; see
[Production configuration](/operate/production-config) once this works.

## Before you start [#before-you-start]

You need the `nativelink` binary and a writable directory. This walkthrough
uses `/tmp/nativelink`; if you use somewhere else, keep the **content path and
the work directory on the same filesystem**, because the worker hard-links
between them.

```sh
mkdir -p /tmp/nativelink/data /tmp/nativelink/work
```

## Step 1: Two stores and a listener [#step-1--two-stores-and-a-listener]

Start a file called `my-config.json5`. The smallest useful NativeLink is a
cache: two stores and one server.

```json5
{
  stores: [
    {
      name: "CAS_MAIN_STORE",
      filesystem: {
        content_path: "/tmp/nativelink/data/content_path-cas",
        temp_path: "/tmp/nativelink/data/tmp_path-cas",
        eviction_policy: { max_bytes: 10000000000 }, // 10 GB
      },
    },
    {
      name: "AC_MAIN_STORE",
      filesystem: {
        content_path: "/tmp/nativelink/data/content_path-ac",
        temp_path: "/tmp/nativelink/data/tmp_path-ac",
        eviction_policy: { max_bytes: 500000000 }, // 500 MB
      },
    },
  ],
  servers: [
    {
      name: "public",
      listener: {
        http: { socket_address: "0.0.0.0:50051" },
      },
      services: {
        cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
        ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
        bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
        capabilities: [{ instance_name: "main" }],
      },
    },
  ],
  global: {
    max_open_files: 24576,
    default_digest_hash_function: "sha256",
  },
}
```

Run it:

```sh
nativelink ./my-config.json5
```

<VerifyBlock title="Step 1 is right if">

- The process starts and logs `Ready, listening on 0.0.0.0:50051`.
- `/tmp/nativelink/data/` now contains a content and a temp directory for
  each store (plus a `content_path-*.exec` sibling on Unix), all empty apart
  from their `s/` and `d/` subdirectories.
- Deliberately misspelling `eviction_policy` and re-running makes it exit
  immediately, naming the key it didn't recognise.

</VerifyBlock>

That last check is worth actually doing once. It's the fastest feedback loop
you have while editing configs, and knowing what the failure looks like means
you'll recognise it later.

## Step 2: Point a build at it [#step-2--point-a-build-at-it]

Nothing about the config changes here, but proving the cache works before
adding execution keeps the next failure unambiguous.

For Bazel, in `.bazelrc`:

```text
build --remote_cache=grpc://localhost:50051
build --remote_instance_name=main
startup --digest_function=sha256
```

Build something twice. The second build should report remote cache hits.

<VerifyBlock title="Step 2 is right if">

- The first build uploads: `content_path-cas` grows.
- The second build reports remote cache hits and finishes faster.
- `--remote_instance_name` matches the `instance_name` in the config. If it
  doesn't, the server rejects every cache call with `'instance_name' not
  configured for '<name>'`, which Bazel reports as remote cache warnings and
  zero hits.

</VerifyBlock>

<Callout type="info" title="Other build tools">
  [Connect your build](/getting-started/connect-your-build) covers the
  equivalent flags for Buck2, CMake, Goma-family clients, and others. The
  config is the same either way.
</Callout>

## Step 3: Add a scheduler [#step-3--add-a-scheduler]

Execution needs a queue. Add a `schedulers` array beside `stores`:

```json5
schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: {
        cpu_count: "minimum",
        OSFamily: "exact",
        "container-image": "priority",
      },
    },
  },
],
```

Three keys is enough to start. `cpu_count` lets an action ask for a machine of
a certain size, `OSFamily` stops a Linux action landing on a macOS worker, and
`container-image` is carried through to the worker without being matched on.

Restart. Nothing observable changes yet: the scheduler exists but nothing
reaches it, and no worker has joined.

## Step 4: Expose execution, on two ports [#step-4--expose-execution-on-two-ports]

Now wire the scheduler in. Add `execution` to the public server, give
`capabilities` a `remote_execution` block, and add a **second server** for the
worker API:

```json5
servers: [
  {
    name: "public",
    listener: { http: { socket_address: "0.0.0.0:50051" } },
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      execution: [{
        instance_name: "main",
        cas_store: "CAS_MAIN_STORE",
        scheduler: "MAIN_SCHEDULER",
      }],
      capabilities: [{
        instance_name: "main",
        remote_execution: { scheduler: "MAIN_SCHEDULER" },
      }],
    },
  },
  {
    name: "private_workers_servers",
    listener: { http: { socket_address: "0.0.0.0:50061" } },
    services: {
      worker_api: { scheduler: "MAIN_SCHEDULER" },
      admin: {},
      health: {},
    },
  },
],
```

The `remote_execution` block on `capabilities` is the part that's often
forget. Without it, clients are told this endpoint caches but does not
execute, and they'll never send an `Execute` call to find out otherwise.

<VerifyBlock title="Step 4 is right if">

- Two listeners come up: `:50051` and `:50061`.
- `curl http://localhost:50061/status` answers.
- Nothing client-facing is served on `:50061`. That port is the security
  boundary from
  [Servers and services](/configuration/servers-and-services).

</VerifyBlock>

## Step 5: Add a worker [#step-5--add-a-worker]

A worker needs a `fast_slow` store with a filesystem fast tier. Replace
`CAS_MAIN_STORE` with one:

```json5
{
  name: "CAS_MAIN_STORE",
  fast_slow: {
    fast: {
      filesystem: {
        content_path: "/tmp/nativelink/data/content_path-cas",
        temp_path: "/tmp/nativelink/data/tmp_path-cas",
        eviction_policy: { max_bytes: 10000000000 },
      },
    },
    slow: { noop: {} },
  },
},
```

The `noop` slow tier looks strange and is correct here: workers *require* a
`fast_slow` store, and in a single-process deployment the fast tier already is
the storage. In a fleet, `slow` becomes a `grpc` store pointing at the shared
CAS.

Then add the `workers` array:

```json5
workers: [
  {
    local: {
      name: "WORKER_1",
      worker_api_endpoint: {
        uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
      },
      cas_fast_slow_store: "CAS_MAIN_STORE",
      upload_action_result: { ac_store: "AC_MAIN_STORE" },
      work_directory: "/tmp/nativelink/work",
      platform_properties: {
        cpu_count: { query_cmd: "nproc" },
        OSFamily: { values: ["linux"] },
        "container-image": { values: [""] },
      },
    },
  },
],
```

Every key the scheduler declared as `minimum` or `exact` is here. That is the
contract, and it is the single most common thing to get wrong.

Restart.

<VerifyBlock title="Step 5 is right if">

- The logs show `Worker registered with scheduler` with the worker id it was
  assigned.
- `content_path-cas` and `/tmp/nativelink/work` are on the same filesystem:
  `df /tmp/nativelink/data /tmp/nativelink/work` reports the same device.
- `/tmp/nativelink/work` is empty; the worker purges it on startup.

</VerifyBlock>

## Step 6: Execute something [#step-6--execute-something]

Point the build at execution rather than just the cache:

```text
build --remote_executor=grpc://localhost:50051
build --remote_instance_name=main
build --remote_default_exec_properties=OSFamily=linux
startup --digest_function=sha256
```

<VerifyBlock title="You did it right if">

- The build reports actions running remotely, not `0 remote`.
- The worker logs actions starting and finishing.
- A second build of the same targets hits the action cache instead of
  executing, proving `upload_action_result` is wired up.

</VerifyBlock>

If actions queue instead of running, the properties don't line up.
[Platform properties](/remote-execution/platform-properties) is the page for
that, and it is far more likely than anything being wrong with what you built
here.

## The finished file [#the-finished-file]

<Accordions>

<Accordion title="my-config.json5, complete" id="my-configjson5-complete">

```json5
{
  stores: [
    {
      name: "CAS_MAIN_STORE",
      fast_slow: {
        fast: {
          filesystem: {
            content_path: "/tmp/nativelink/data/content_path-cas",
            temp_path: "/tmp/nativelink/data/tmp_path-cas",
            eviction_policy: { max_bytes: 10000000000 },
          },
        },
        slow: { noop: {} },
      },
    },
    {
      name: "AC_MAIN_STORE",
      filesystem: {
        content_path: "/tmp/nativelink/data/content_path-ac",
        temp_path: "/tmp/nativelink/data/tmp_path-ac",
        eviction_policy: { max_bytes: 500000000 },
      },
    },
  ],
  schedulers: [
    {
      name: "MAIN_SCHEDULER",
      simple: {
        supported_platform_properties: {
          cpu_count: "minimum",
          OSFamily: "exact",
          "container-image": "priority",
        },
      },
    },
  ],
  workers: [
    {
      local: {
        name: "WORKER_1",
        worker_api_endpoint: {
          uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
        },
        cas_fast_slow_store: "CAS_MAIN_STORE",
        upload_action_result: { ac_store: "AC_MAIN_STORE" },
        work_directory: "/tmp/nativelink/work",
        platform_properties: {
          cpu_count: { query_cmd: "nproc" },
          OSFamily: { values: ["linux"] },
          "container-image": { values: [""] },
        },
      },
    },
  ],
  servers: [
    {
      name: "public",
      listener: { http: { socket_address: "0.0.0.0:50051" } },
      services: {
        cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
        ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
        bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
        execution: [{
          instance_name: "main",
          cas_store: "CAS_MAIN_STORE",
          scheduler: "MAIN_SCHEDULER",
        }],
        capabilities: [{
          instance_name: "main",
          remote_execution: { scheduler: "MAIN_SCHEDULER" },
        }],
      },
    },
    {
      name: "private_workers_servers",
      listener: { http: { socket_address: "0.0.0.0:50061" } },
      services: {
        worker_api: { scheduler: "MAIN_SCHEDULER" },
        admin: {},
        health: {},
      },
    },
  ],
  global: {
    max_open_files: 24576,
    default_digest_hash_function: "sha256",
  },
}
```

</Accordion>

</Accordions>

Compare it with `nativelink-config/examples/basic_cas.json5` in the source
tree. The differences are instructive: the shipped example registers every
service twice, under `instance_name: ""` and `"main"`, so clients that omit
the instance name also work, and it declares nineteen platform properties
rather than three.

## What this is not yet [#what-this-is-not-yet]

This config is correct and it is not production. Four things are missing, and
each has a page:

- **Durability.** A `noop` slow tier means the cache is exactly as durable as
  `/tmp`. [Production configuration](/operate/production-config).
- **Separation.** One process is one failure domain, and one machine's worth of
  execution capacity. [Deploy on bare metal](/operate/deploy-bare-metal).
- **Security.** `0.0.0.0:50061` is bound to every interface with no
  authentication. [Servers and services](/configuration/servers-and-services)
  explains why that port matters most.
- **Visibility.** Nothing here tells you hit rate, queue depth, or worker
  health. [Observability](/operate/observability).

<NextStep href="/how-to" title="How-to guides">
  You can now read and write a config. These are the specific blocks that do
  specific things: object storage, Redis, compression, chunking.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  The shape this evolves into once one process on one machine stops being
  enough.
</NextStep>

# https://docs.nativelink.com/how-to

**How-to guides**: Pick the task you're doing, not the artifact you're reading about.

**Who this is for:** anyone with a running NativeLink who now needs it to do
one specific thing differently. **What you'll have at the end:** the one page
that does your task, rather than a tour. **Time:** as long as the task takes.

<Prerequisites>
  A config file you wrote and understand. See [Configuration](/configuration).
</Prerequisites>

## How this section works [#how-this-section-works]

Every page here is a task, phrased as the thing you are trying to do. None of
them is a tour of a subsystem (that's what
[Concepts](/explanations/architecture) is for), and none of them is an exhaustive
field list, which is what [Reference](/reference/glossary) is for. A recipe gives you
the complete working artifact first, then explains only the parts of it you
have to decide about, then tells you how to tell it worked.

Pages assume you have read [Configuration](/configuration): you have a config
file and you know which part of it you're editing. If you don't yet, that
section is worth the detour.

## Pick your task [#pick-your-task]

**I need my cache stored somewhere else.**
[Storage backends](/how-to/stores) compares every place NativeLink can put
bytes ([filesystem](/how-to/stores/filesystem),
[S3 and compatibles](/how-to/stores/s3-and-compatible),
[GCS](/how-to/stores/gcs), [Azure Blob](/how-to/stores/azure),
[OCI](/how-to/stores/oci-object-storage), [Redis](/how-to/stores/redis) and
[Mongo](/how-to/stores/mongo)) on durability, latency and cost.
[Compose stores](/how-to/stores/compose-stores) is how you layer a fast tier
over a durable one.

**I need to shrink what I store or send.** See
[Remote cache compression](/how-to/stores/compression) and
[Content-defined chunking](/how-to/stores/chunking-and-dedup).

**I need more than one worker.**
[Run multiple workers](/how-to/run-multiple-workers) is the multi-worker
topology: how workers share the CAS, how the scheduler picks one, and which
parts of the shipped example will mislead you.

**I need this to be secure on the network.**
[TLS and authentication](/how-to/tls-and-auth) covers TLS on the public
listeners, mutual TLS as the access control, keeping the worker API off the
public interface, and what NativeLink does not authenticate.

**I need to move to a newer version.**
[Upgrade to a newer version](/how-to/upgrade-versions) is the upgrade path:
what actually breaks between releases, the order to move the processes in, and
what happens to the cache you already have.

**I'm coming from another REAPI cache.**
[Migrate from another REAPI cache](/how-to/migrate-an-existing-cache) puts
NativeLink in front of your old one as a read-only diode, so you never copy the
data and never lose the hit rate.

<NextStep href="/configuration" kind="aside" title="Configuration">
  If none of these is your task and you're here to learn the config file
  itself, Configuration is the page you want.
</NextStep>

<NextStep href="/operate" title="Operate">
  Recipes get the shape right. The Operate section is what you read once the
  thing is real and other people depend on it.
</NextStep>

# https://docs.nativelink.com/how-to/stores

**Storage backends**: Every place NativeLink can put bytes, compared on durability, latency and cost, and the composition almost every real deployment lands on.

**Who this is for:** anyone whose cache currently lives somewhere they can't
keep it. **What you'll have at the end:** the one backend page that matches
your durability and latency requirements, and the shape you'll wrap it in.
**Time:** ten minutes to choose, then as long as the backend page takes.

<Prerequisites>
  A config file you can edit. See [Configuration](/configuration).
</Prerequisites>

## Everything here is a store entry [#everything-here-is-a-store-entry]

Nothing on this page is a new concept. A backend is one entry in the `stores`
array with a name and a type, exactly as described in
[Stores](/configuration/stores):

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/cas",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: "50gb" },
    },
  },
],
```

Swapping backends means swapping the inner block. The name, and everything
that references the name, stays the same. That is why these pages are
interchangeable recipes rather than a migration guide.

## Pick a backend [#pick-a-backend]

| Backend | Survives restart | Typical read latency | Cost shape | Reach for it when |
| --- | --- | --- | --- | --- |
| `memory` | No | Microseconds | RAM | Testing, or the fast half of a `fast_slow` pair |
| [`filesystem`](/how-to/stores/filesystem) | Yes | Sub-millisecond on NVMe | Disk you already pay for | One machine, or the fast tier of any fleet |
| [S3 and compatible](/how-to/stores/s3-and-compatible) | Yes | Tens of milliseconds | Per-GB stored plus per-request | You want durability you don't operate |
| [GCS](/how-to/stores/gcs) | Yes | Tens of milliseconds | Per-GB stored plus per-request | You're on Google Cloud |
| [Azure Blob](/how-to/stores/azure) | Yes | Tens of milliseconds | Per-GB stored plus per-request | You're on Azure |
| [OCI Object Storage](/how-to/stores/oci-object-storage) | Yes | Tens of milliseconds | Per-GB stored plus per-request | You're on Oracle Cloud |
| [Redis](/how-to/stores/redis) | Configurable | Around a millisecond | RAM, priced steeply | Small hot objects shared across replicas |
| [Mongo](/how-to/stores/mongo) | Yes | Single-digit milliseconds | Per-GB plus the cluster you run | You already operate Mongo, or you need change streams |
| `grpc` | Delegated | One extra network hop | Whatever the upstream costs | Fronting another REAPI cache |
| `noop` | n/a | Zero | Free | Deliberately discarding writes |

`memory`, `grpc` and `noop` have no page of their own because there is nothing
to decide: `memory` takes an eviction policy, `grpc` takes an upstream address,
and `noop` takes nothing at all. All three are covered in
[Compose stores](/how-to/stores/compose-stores).

<Callout type="info" title="`experimental_` is about the config key, not the code path">
  Six providers (`aws`, `r2`, `ontap`, `gcs`, `azure`, `oci`) are declared
  under `experimental_cloud_object_store`, and Mongo under `experimental_mongo`. The prefix signals that the
  configuration key may be renamed in a future release. It is not a claim
  that the store is unfinished. Pin your NativeLink version and read the
  release notes before upgrading, the same as for any config surface.
</Callout>

## The shape almost everyone lands on [#the-shape-almost-everyone-lands-on]

Very few production deployments name a cloud bucket as their CAS store
directly. The read pattern of a build cache (many small objects, requested
repeatedly, in bursts) is exactly what object storage prices and latency are
worst at. The shape that works is a local fast tier in front of a durable slow
tier:

```json5
{
  name: "CAS_MAIN_STORE",
  fast_slow: {
    fast: {
      filesystem: {
        content_path: "/var/lib/nativelink/cas",
        temp_path: "/var/lib/nativelink/tmp",
        eviction_policy: { max_bytes: "50gb" },
      },
    },
    slow: {
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "my-nativelink-cas",
        key_prefix: "cas/",
      },
    },
  },
},
```

Reads hit the filesystem first and only fall through to the bucket on a miss.
Writes go to both. Every shipped cloud example in
`nativelink-config/examples/` is built this way (some with a `memory` fast
tier instead of `filesystem`), and every backend page below shows its own
version of it.

[Compose stores](/how-to/stores/compose-stores) covers the rest of the
composition vocabulary (deduplication, size partitioning, integrity
verification, sharding) and the order the layers have to go in.

## Shrinking what you move [#shrinking-what-you-move]

Two pages in this group are not backends at all. They change how many bytes
travel between the client and NativeLink, whichever backend is underneath:

- [Remote cache compression](/how-to/stores/compression): zstd on the wire,
  for compressible artifacts on a real network.
- [Content-defined chunking](/how-to/stores/chunking-and-dedup): re-transfer
  only the changed parts of large, incrementally-changing blobs.

They compose with everything above, and with each other.

<VerifyBlock title="You've chosen well if">

- Your slow tier survives the loss of any single machine, or you've decided
  out loud that losing the cache is acceptable.
- Your fast tier is on the same host as the process reading it, and has an
  `eviction_policy` with a `max_bytes` you can afford.
- If you run workers, the fast half of their `cas_fast_slow_store` is a
  `filesystem` store, because nothing else supports the hard-linking workers need.

</VerifyBlock>

<NextStep href="/how-to/stores/filesystem" title="Filesystem store">
  The one backend nearly every deployment uses, whether or not it's the
  durable one.
</NextStep>

<NextStep href="/how-to/stores/compose-stores" kind="aside" title="Compose stores">
  Wrapping, layering and splitting stores, and the order the layers have to
  go in.
</NextStep>

# https://docs.nativelink.com/how-to/stores/filesystem

**Filesystem store**: Put the cache on local disk: the two paths that must share a block device, the eviction policy you cannot skip, and the tuning knobs that matter on real hardware.

**Who this is for:** anyone who wants the cache to survive a restart without
paying for object storage, and anyone building the fast tier of a larger
composition. **What you'll have at the end:** a `filesystem` store sized to
your disk, with eviction that actually runs. **Time:** fifteen minutes.

<Prerequisites>
  A config file you can edit. See [Configuration](/configuration).
</Prerequisites>

## The recipe [#the-recipe]

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/cas/content",
      temp_path: "/var/lib/nativelink/cas/tmp",
      eviction_policy: {
        max_bytes: "50gb",
        evict_bytes: "5gb",
      },
    },
  },
  {
    name: "AC_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/ac/content",
      temp_path: "/var/lib/nativelink/ac/tmp",
      eviction_policy: {
        max_bytes: "5gb",
      },
    },
  },
],
```

Create the parent directory and start NativeLink. The store scans
`content_path` on boot and adopts every file whose name parses as a store key
(`<hash>-<size>` under `d/`, or a plain string key under `s/`); a file whose
name does not parse is deleted. The scan reads metadata, not content, so it does not check
that bytes still match their digest. That scan is what makes the cache survive
restarts, and it is also why boot time grows with the number of cached
objects.

<SourceLink file="nativelink-config/src/stores.rs" symbol="FilesystemSpec" />
<SourceLink file="nativelink-store/src/filesystem_store.rs" symbol="add_files_to_cache" />

## The two paths [#the-two-paths]

`content_path` holds the data. `temp_path` holds objects mid-upload and
mid-delete, while their contents cannot yet be trusted.

<Callout type="warn" title="`temp_path` must be on the same block device as `content_path`">
  Completing an upload is an atomic rename from the temp path into the content
  path. Across a device boundary a rename becomes a copy, which is neither
  atomic nor free. Put both under one parent directory and the problem cannot
  arise.
</Callout>

Everything in `temp_path` is deleted on every startup. Nothing you care about
should ever be written there by hand.

## The eviction policy you cannot skip [#the-eviction-policy-you-cannot-skip]

`eviction_policy` is optional in the schema and mandatory in practice. Without
it, nothing is ever removed and the store grows until the disk is full.

Eviction is LRU: touching an entry refreshes its timestamp, and inserts run the
policy until the store fits again. Four independent limits are available, all
defaulting to `0`, which means "don't evict on this axis":

| Field | Meaning |
| --- | --- |
| `max_bytes` | Total size on disk before eviction starts |
| `evict_bytes` | Keep evicting until `max_bytes - evict_bytes` is reached |
| `max_seconds` | Age since last access before an entry is evicted |
| `max_count` | Number of entries before eviction starts |

Set `evict_bytes` to something like 10% of `max_bytes`. Without it, the store
sits exactly at the limit and every single insert triggers an eviction;
a low watermark turns that thrash into one bulk pass per 10%.

Size accounting uses `block_size` (default 4 KiB), not the byte length of the
file, because a one-byte object still consumes a whole block. If your
filesystem uses a different block size, set it, or your `max_bytes` will be a
significant underestimate of real disk use.

## Tuning that matters on real hardware [#tuning-that-matters-on-real-hardware]

**`max_concurrent_writes`** (default `0`, unlimited). Every write streams into
a temp file and calls `sync_all()`. Enough concurrent writes will saturate disk
I/O and start blocking the async runtime. On a busy shared CAS, set this to
something bounded (a few dozen) rather than leaving it open.

**`read_buffer_size`** (default 32 KiB). Leave it alone unless you are
benchmarking.

**`evict_page_cache`** (default `false`). When true, the store advises the
kernel to drop the page cache for each blob after it is read or written.

<Callout type="warn" title="Leave `evict_page_cache` off unless you know you want it">
  On a real filesystem this takes a globally serialized, all-CPU kernel path
  that stalls on many-core hosts, and it throws away the page cache that
  makes a fast tier fast in the first place. It exists for deployments that
  specifically need this store's I/O kept out of the page cache.
</Callout>

## If this store feeds a worker [#if-this-store-feeds-a-worker]

A worker's `cas_fast_slow_store` must have a `filesystem` store as its `fast`
half. The worker builds each action's input tree by hard-linking files out of
that store, and hard links require real files on a real filesystem; a memory
or object store cannot supply them.

The worker's `work_directory` must be on the **same filesystem** as that
store's `content_path`, for the same reason. See
[Schedulers and workers](/configuration/scheduler-and-workers).

## Steps [#steps]

<Steps>

<li>

**Create the directories** and make sure the process user owns them:
`mkdir -p /var/lib/nativelink/cas/content /var/lib/nativelink/cas/tmp`.

</li>

<li>

**Add the store** with both paths under one parent and an `eviction_policy`
whose `max_bytes` leaves headroom on the device.

</li>

<li>

**Start NativeLink** and run a build against it.

</li>

<li>

**Restart NativeLink** and run the same build again. The second run should be
a cache hit.

</li>

</Steps>

<VerifyBlock>

- Files appear under `content_path/d/` named `<hash>-<size>` while a build
  runs.
- `temp_path` is empty when no upload is in flight, and empty after a restart.
- `du -sh` on `content_path` stabilises near `max_bytes` rather than growing
  past it.
- A build re-run after a restart of NativeLink is a cache hit, not a rebuild.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="The disk filled up" id="the-disk-filled-up">

The store has no `eviction_policy`, or has one with `max_bytes: 0`. Both mean
"never evict". Add a real limit and restart; the boot scan will bring the
store back under the limit.

If the limit is set and the disk still filled, check `block_size`: on a
filesystem with 16 KiB blocks, an accounting `block_size` of 4 KiB
underestimates real usage by up to 4x for small objects.

</Accordion>

<Accordion title="Uploads fail with a cross-device link error" id="uploads-fail-with-a-cross-device-link-error">

`temp_path` and `content_path` are on different filesystems. Check with
`df /var/lib/nativelink/cas/content /var/lib/nativelink/cas/tmp`. If the
device column differs, move one of them.

</Accordion>

<Accordion title="Startup takes minutes" id="startup-takes-minutes">

That's the boot scan reading the metadata of every file in `content_path`
(name, size, access time). It's proportional to entry count. A tighter `max_count` or
`max_seconds` keeps the store smaller; a `fast_slow` pair with a small fast
tier over a durable slow tier keeps boot cheap without giving up capacity.

</Accordion>

<Accordion title="Everything is slower than expected under load" id="everything-is-slower-than-expected-under-load">

Check `max_concurrent_writes`. Unlimited means the store will happily issue
more concurrent `sync_all()` calls than the device can retire, at which point
disk I/O becomes the bottleneck for every request including reads.

</Accordion>

<Accordion title="Cache entries vanish after a restart even though the disk is intact" id="cache-entries-vanish-after-a-restart-even-though-the-disk-is-intact">

The boot scan only deletes files whose names it cannot parse as a store key;
everything else is re-added, and the eviction policy then runs against the
whole set, so a store over `max_bytes` or `max_count` is trimmed at startup.
Separately, switching the digest function between `sha256` and `blake3`
(`default_digest_hash_function` on the server, `--digest_function` on the
client) does not delete anything, but every lookup misses because clients now
compute different digests. See [Stores](/configuration/stores).

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Put this filesystem store in front of a durable one, or behind
  deduplication and compression.
</NextStep>

<NextStep href="/how-to/stores/s3-and-compatible" kind="aside" title="S3 and compatible">
  The usual slow tier: durability you don't have to operate.
</NextStep>

# https://docs.nativelink.com/how-to/stores/s3-and-compatible

**S3 and compatible**: Back the cache with Amazon S3, Cloudflare R2 or NetApp ONTAP S3: one store type, three providers, and the shared options that decide cost and failure behaviour.

**Who this is for:** anyone who wants durability they don't operate, on S3 or
something that speaks S3. **What you'll have at the end:** a bucket-backed CAS
behind a local fast tier, with credentials that aren't in the config file.
**Time:** thirty minutes, most of it bucket and IAM setup.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and a bucket you can write to.
</Prerequisites>

Three providers share one store type and one block of common options. `aws`
and `r2` share one implementation (R2 is the S3 store with a derived endpoint
and region `auto`); `ontap` has its own, with a few differences noted below.
They differ mainly in how the endpoint and credentials are derived:

| Provider | `provider` value | Identifies the endpoint by |
| --- | --- | --- |
| Amazon S3 | `aws` | `region` + `bucket` |
| Cloudflare R2 | `r2` | `account_id` (endpoint is `https://{account_id}.r2.cloudflarestorage.com`) |
| NetApp ONTAP S3 | `ontap` | an explicit `endpoint` plus `vserver_name` |

[GCS](/how-to/stores/gcs), [Azure Blob](/how-to/stores/azure) and
[OCI Object Storage](/how-to/stores/oci-object-storage) use the same store type
with their own `provider` values and have their own pages.

## Amazon S3 [#amazon-s3]

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        filesystem: {
          content_path: "/var/lib/nativelink/cas/content",
          temp_path: "/var/lib/nativelink/cas/tmp",
          eviction_policy: { max_bytes: "50gb" },
        },
      },
      slow: {
        experimental_cloud_object_store: {
          provider: "aws",
          region: "us-east-1",
          bucket: "my-nativelink-cas",
          key_prefix: "cas/",
          retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
        },
      },
    },
  },
],
```

`region` and `bucket` are the only fields you have to set. Credentials come
from the standard AWS chain (environment variables, the shared credentials
file, the EC2/ECS/EKS instance or task role), so nothing secret goes in the
config.

`key_prefix` lets one bucket hold several logical stores. Give the CAS `cas/`
and the Action Cache `ac/` rather than provisioning two buckets; lifecycle
rules can then treat them differently.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalAwsSpec" />

## Cloudflare R2 [#cloudflare-r2]

R2 has no regions from the client's point of view: the endpoint is derived
entirely from your account ID.

```json5
{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "r2",
    account_id: "${R2_ACCOUNT_ID}",
    bucket: "nativelink-cas",
    access_key_id: "${R2_ACCESS_KEY_ID}",
    secret_access_key: "${R2_SECRET_ACCESS_KEY}",
    key_prefix: "cas/",
    retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
  },
},
```

`access_key_id` and `secret_access_key` are optional in the schema (omitting
both falls back to the AWS credential chain), but R2 tokens don't live in that
chain, so in practice you set them, from environment variables. The
`${VAR}` syntax is expanded at config-load time; see
[The config file](/configuration/config-file).

R2 charges no egress, which changes the arithmetic on how small a fast tier
you can get away with.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalR2Spec" />

## NetApp ONTAP S3 [#netapp-ontap-s3]

On-premises S3 needs an explicit endpoint, the storage VM serving it, and
usually a private CA bundle:

```json5
{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "ontap",
    endpoint: "https://ontap-s3-endpoint:443",
    vserver_name: "your-vserver",
    bucket: "your-bucket",
    root_certificates: "/etc/nativelink/ontap-ca.pem",
    key_prefix: "cas/",
    retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
    multipart_max_concurrent_uploads: 10,
  },
},
```

`root_certificates` is optional and points at a PEM bundle for the CA that
signed the endpoint's certificate. Without it, the system roots are used and
an internally-signed endpoint fails TLS verification. Credentials come from
the same AWS chain as `aws` (`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`
in practice), and `vserver_name` doubles as the signing region.

The ONTAP implementation differs from `aws` in three ways: it always uses
path-style addressing, its default `max_retry_buffer_per_request` is 20 MB
rather than 5 MB, and it is HTTPS-only with HTTP/1.1 and HTTP/2 both enabled,
so `insecure_allow_http` and `disable_http2` have no effect on it.

<SourceLink file="nativelink-store/src/ontap_s3_store.rs" symbol="OntapS3Store" />

ONTAP has one extra store type of its own, `ontap_s3_existence_cache`, which
keeps a persisted index of what the bucket contains and refreshes it on an
interval:

```json5
{
  name: "CAS_SLOW_STORE",
  ontap_s3_existence_cache: {
    index_path: "/var/lib/nativelink/ontap-index.json",
    sync_interval_seconds: 300,
    backend: {
      endpoint: "https://ontap-s3-endpoint:443",
      vserver_name: "your-vserver",
      bucket: "your-bucket",
      key_prefix: "cas/",
    },
  },
},
```

It exists because `FindMissingBlobs` against an on-premises S3 endpoint is
often the slowest thing in the build, and the answer is usually stale-tolerant.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalOntapS3Spec" />

## The options all three share [#the-options-all-three-share]

Every provider flattens the same block of common options. These are the ones
worth setting deliberately:

| Field | Default | What it decides |
| --- | --- | --- |
| `key_prefix` | none | Namespacing inside one bucket |
| `retry` | see below | How hard to try before surfacing an error |
| `consider_expired_after_s` | `0` (never) | How old an object may be and still count as present |
| `max_retry_buffer_per_request` | 5 MB (20 MB for `ontap`) | How much of a single-request upload is buffered so it can be retried |
| `multipart_max_concurrent_uploads` | 10 | Concurrent `UploadPart` requests within one multipart upload |
| `insecure_allow_http` | `false` | Permit plain HTTP; local testing only (`aws`, `r2`) |
| `disable_http2` | `false` | Fall back to HTTP/1.1 for proxies that mishandle h2 (`aws`, `r2`) |

`retry` is exponential with jitter: up to `max_retries` retries after the
first attempt, a base `delay` in seconds that is doubled before the first
retry and again for each retry after it, and `jitter` as a fraction of the
computed delay (`0.5` means each wait is randomised within plus or minus 25%).
The shipped examples all use `{ max_retries: 6, delay: 0.3, jitter: 0.5 }`:
waits of about 0.6, 1.2, 2.4, 4.8, 9.6 and 19.2 seconds, roughly forty seconds
of retrying across seven attempts.

Objects smaller than 5 MB with a known size are uploaded in one `PutObject`
request; anything larger, or of unknown size, goes through a multipart upload
in parts of at least 5 MB, each part held in memory and retried on its own.

<SourceLink file="nativelink-util/src/retry.rs" symbol="Retrier" />

`consider_expired_after_s` is the one people are surprised by. When set, an
object whose `last_modified` is older than this many seconds is reported as
*not existing*, even though it is still in the bucket. Pair it with a bucket
lifecycle rule whose window is longer by a comfortable margin (a few days):
the rule deletes the object, and this setting stops NativeLink handing out a
digest that is about to disappear. Clients that get a `NotFound`
re-upload.

<SourceLink file="nativelink-config/src/stores.rs" symbol="CommonObjectSpec" />

## Steps [#steps]

<Steps>

<li>

**Create the bucket** and a least-privilege credential that can head, get and
put objects and create, upload to, complete and abort multipart uploads on it,
nothing else. The store never lists or deletes objects (only the
`ontap_s3_existence_cache` wrapper lists the bucket, to build its index).

</li>

<li>

**Add a lifecycle rule** that aborts incomplete multipart uploads after a few
days. A process killed mid-upload leaves orphaned parts you are billed for.

</li>

<li>

**Export the credentials** into the process environment, and reference them
with `${VAR}` rather than pasting them into the config.

</li>

<li>

**Add the store** as the `slow` half of a `fast_slow` pair whose `fast` half
is a local [filesystem store](/how-to/stores/filesystem).

</li>

<li>

**Run a build twice**: once cold, once after clearing the local fast tier.

</li>

</Steps>

<VerifyBlock>

- Objects appear in the bucket under `key_prefix` during the first build.
- The second build is a cache hit even after the local `content_path` is
  emptied, which proves reads are falling through to the bucket.
- Bucket request counts are far lower than blob counts, which proves the fast
  tier is absorbing repeat reads.
- No incomplete multipart uploads accumulate between builds.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Every request fails with a credentials error" id="every-request-fails-with-a-credentials-error">

For `aws`, the credential chain found nothing. Confirm the process can see
`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, or that the instance/task
role is attached and reachable. For `r2`, set `access_key_id` and
`secret_access_key` explicitly, because R2 tokens are not in the AWS chain.

</Accordion>

<Accordion title="TLS verification fails against an on-premises endpoint" id="tls-verification-fails-against-an-on-premises-endpoint">

Set `root_certificates` to the PEM bundle for the CA that signed the
endpoint's certificate. Do not reach for `insecure_allow_http`, which
disables transport security entirely and is meant only for a local test
double.

</Accordion>

<Accordion title="Builds are slow even though the cache is hitting" id="builds-are-slow-even-though-the-cache-is-hitting">

You're going to the bucket on every read. Check that the store is the `slow`
half of a `fast_slow` pair and that the `fast` half's `eviction_policy` isn't
so small it evicts everything before it's re-read.

</Accordion>

<Accordion title="The bucket bill is dominated by requests, not storage" id="the-bucket-bill-is-dominated-by-requests-not-storage">

Same cause. A build cache issues enormous numbers of small existence checks
and reads; object storage prices those individually. A local fast tier, and
an [`existence_cache`](/how-to/stores/compose-stores) in front of the slow
store, are the two levers.

</Accordion>

<Accordion title="Large uploads fail but small ones succeed" id="large-uploads-fail-but-small-ones-succeed">

Objects of 5 MB and above go through multipart upload, so check that the
credential can `CreateMultipartUpload`, `UploadPart`, `CompleteMultipartUpload`
and `AbortMultipartUpload`; a policy that only allows `PutObject` breaks
exactly at that size. Also check `max_retry_buffer_per_request` if you have
lowered it: a single-request upload larger than the buffer cannot be replayed
after a transient failure, and setting it to zero disables upload buffering
entirely, which trades retry safety for memory.

</Accordion>

<Accordion title="Objects exist in the bucket but NativeLink reports them missing" id="objects-exist-in-the-bucket-but-nativelink-reports-them-missing">

Check `consider_expired_after_s`. A non-zero value makes objects older than
that window invisible on purpose. If you didn't intend that, set it back
to `0`.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Deduplication, compression and existence caching in front of the bucket,
  the layers that decide what it actually costs.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  What this looks like once several processes share the same bucket.
</NextStep>

# https://docs.nativelink.com/how-to/stores/gcs

**Google Cloud Storage**: Back the cache with a GCS bucket: authentication, the resumable-upload chunk size, and the timeout fields whose names disagree with their units.

**Who this is for:** anyone running NativeLink on Google Cloud who wants the
durable tier to be a GCS bucket. **What you'll have at the end:** a
bucket-backed CAS behind a local fast tier, authenticated by the ambient
service account. **Time:** twenty minutes, most of it IAM.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and a GCS bucket you can write to.
</Prerequisites>

GCS is one `provider` value of the same cloud object store used by
[S3 and compatible](/how-to/stores/s3-and-compatible),
[Azure Blob](/how-to/stores/azure) and
[OCI Object Storage](/how-to/stores/oci-object-storage). The composition
around it is identical; only the provider block changes.

## The recipe [#the-recipe]

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        filesystem: {
          content_path: "/var/lib/nativelink/cas/content",
          temp_path: "/var/lib/nativelink/cas/tmp",
          eviction_policy: { max_bytes: "50gb" },
        },
      },
      slow: {
        experimental_cloud_object_store: {
          provider: "gcs",
          bucket: "my-nativelink-cas",
          key_prefix: "cas/",
          retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
        },
      },
    },
  },
],
```

`bucket` is the only field you have to set. No region is needed: a GCS bucket's
location is a property of the bucket, decided when you create it, and the
client reaches it the same way wherever it lives. Put the bucket in the region
your workers run in; cross-region reads are both slower and billed.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalGcsSpec" />

## Authentication [#authentication]

Credentials come from the ambient Google credential chain: a credentials file
named by `GOOGLE_APPLICATION_CREDENTIALS` (or inline JSON in
`GOOGLE_APPLICATION_CREDENTIALS_JSON`), the application-default credentials
written by `gcloud auth application-default login`, or otherwise the metadata
server on GCE, GKE or Cloud Run. Nothing secret goes in the config file.

The service account needs to read and write objects in the bucket;
`roles/storage.objectUser` scoped to the bucket covers it (overwriting an
existing object in GCS also needs the delete permission that role includes).
NativeLink itself never deletes objects; expiry is a bucket lifecycle rule.

`authentication_required` defaults to `false`: if no credential is found, the
store starts anyway and issues anonymous requests to `storage.googleapis.com`
(there is no endpoint override for an emulator). Set it to `true` for anything real,
so a misconfigured credential chain fails loudly at startup rather than as a
permission error on the first request.

<SourceLink file="nativelink-store/src/gcs_client/client.rs" symbol="GcsClient" />

## Options worth setting [#options-worth-setting]

| Field | Default | What it decides |
| --- | --- | --- |
| `key_prefix` | none | Namespacing inside one bucket; give the CAS `cas/` and the AC `ac/` |
| `resumable_chunk_size` | 2 MB | Chunk size for resumable uploads of large objects; can only be lowered |
| `authentication_required` | `false` | Whether to error at startup when no credential is found |
| `retry` | | Exponential backoff with jitter, shared with every other provider |
| `consider_expired_after_s` | `0` (never) | How old an object may be and still count as present |

Objects under 5 MB with a known size go up in one request; everything else
uses a resumable upload, sent chunk by chunk with `resumable_chunk_size`
bytes per request. The value is rounded to a multiple of 256 KiB and capped at
the 2 MB default, so it can only be lowered, which trades more round trips per
object for less memory held per in-flight upload. No setting
raises it.

<SourceLink file="nativelink-store/src/gcs_store.rs" symbol="GcsStore" />

<Callout type="warn" title="Two timeout fields are documented in milliseconds but read in seconds">
  `connection_timeout_s` and `read_timeout_s` carry a documented default of
  `3000` and the words "in milliseconds" in the reference. The code reads
  both as seconds and uses 3 seconds when they are unset. If you set them,
  set them in seconds, and leave them unset unless you are actively debugging
  a timeout.
</Callout>

The remaining shared options behave as described in
[the shared options section](/how-to/stores/s3-and-compatible), with one
difference: here `multipart_max_concurrent_uploads` (default 10) caps how many
requests this store has in flight to GCS at once, across all operations, and
the chunks of one resumable upload are sent one after another.
`max_retry_buffer_per_request` is raised to at least the chunk size, and
`insecure_allow_http` and `disable_http2` are not consulted by the GCS
client.

## Steps [#steps]

<Steps>

<li>

**Create the bucket** in the region your workers run in. Uniform bucket-level
access is fine and simpler than per-object ACLs.

</li>

<li>

**Grant the service account** object read and write on that bucket, and
nothing wider.

</li>

<li>

**Set `authentication_required: true`** so a missing credential fails at
startup. A resumable upload session that a killed process leaves behind
expires on its own after a week; there is nothing to clean up.

</li>

<li>

**Add the store** as the `slow` half of a `fast_slow` pair whose `fast` half is
a local [filesystem store](/how-to/stores/filesystem).

</li>

<li>

**Run a build twice** (once cold, once after emptying the local fast tier)
and watch object counts in the bucket.

</li>

</Steps>

<VerifyBlock>

- Objects appear under `key_prefix` in the bucket during the first build.
- The second build is a cache hit even after `content_path` is emptied, which
  proves reads fall through to the bucket.
- Bucket request counts are far lower than blob counts, which proves the fast
  tier is absorbing repeat reads.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Every request fails with a permission error" id="every-request-fails-with-a-permission-error">

The credential chain found something, but it can't act on the bucket. Confirm
the service account attached to the instance or workload identity has object
read and write on it. If it can create but not overwrite, re-uploads of a
digest that already exists fail, because GCS needs the delete permission to
replace an object.

</Accordion>

<Accordion title="The store starts but no credential was ever found" id="the-store-starts-but-no-credential-was-ever-found">

That's `authentication_required` at its default of `false`. Set it to `true`
so the store errors at startup instead of quietly issuing unauthenticated
requests.

</Accordion>

<Accordion title="Large uploads are slow or memory-hungry" id="large-uploads-are-slow-or-memory-hungry">

The chunk size is at most 2 MB and each chunk is a round trip, so a large
object is many sequential requests; `resumable_chunk_size` can only make the
chunks smaller. Memory per in-flight upload is bounded by the chunk size plus
the retry buffer; the number of concurrent requests is bounded by
`multipart_max_concurrent_uploads`.

</Accordion>

<Accordion title="Builds are slow even though the cache is hitting" id="builds-are-slow-even-though-the-cache-is-hitting">

You're reaching the bucket on every read. Confirm the store is the `slow` half
of a `fast_slow` pair, and that the `fast` half's `eviction_policy` isn't so
small that everything is evicted before it's re-read.

</Accordion>

<Accordion title="Objects exist in the bucket but NativeLink reports them missing" id="objects-exist-in-the-bucket-but-nativelink-reports-them-missing">

Check `consider_expired_after_s`. A non-zero value makes objects older than
that window invisible on purpose, so they pair with a matching bucket lifecycle
rule. If you didn't intend that, set it back to `0`.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Deduplication, compression and existence caching in front of the bucket,
  the layers that decide what it actually costs.
</NextStep>

<NextStep href="/how-to/stores/s3-and-compatible" kind="aside" title="S3 and compatible">
  The same store type with a different provider, and the shared options in
  full.
</NextStep>

# https://docs.nativelink.com/how-to/stores/azure

**Azure Blob Storage**: Back the cache with an Azure Blob container: account and container naming, the endpoint override for Azurite, and when a SAS URL replaces everything else.

**Who this is for:** anyone running NativeLink on Azure who wants the durable
tier to be a Blob Storage container. **What you'll have at the end:** a
container-backed CAS behind a local fast tier. **Time:** twenty minutes, most
of it access configuration.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and an Azure storage account you can write to.
</Prerequisites>

Azure Blob is one `provider` value of the same cloud object store used by
[S3 and compatible](/how-to/stores/s3-and-compatible),
[GCS](/how-to/stores/gcs) and
[OCI Object Storage](/how-to/stores/oci-object-storage). The composition around
it is identical; only the provider block changes.

## The recipe [#the-recipe]

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        filesystem: {
          content_path: "/var/lib/nativelink/cas/content",
          temp_path: "/var/lib/nativelink/cas/tmp",
          eviction_policy: { max_bytes: "50gb" },
        },
      },
      slow: {
        experimental_cloud_object_store: {
          provider: "azure",
          account_name: "mystorageaccount",
          container: "nativelink-cas",
          key_prefix: "cas/",
          retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
        },
      },
    },
  },
],
```

`account_name` and `container` are the pair you have to set. From them the store
builds the endpoint `https://{account_name}.blob.core.windows.net/{container}`.
The container is a blob container inside the storage account. Create it before
starting NativeLink; the store does not create it for you.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalAzureSpec" />

## Three ways to point at a container [#three-ways-to-point-at-a-container]

**Account plus container.** The default above. The endpoint is derived, and
the store authenticates with Entra ID Workload Identity and nothing else: it
reads `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_FEDERATED_TOKEN_FILE`
(the variables AKS workload identity injects into a pod) and fails at startup
if they are missing. A VM managed identity, a storage account key or an
`az login` session are not tried. Nothing secret is in the config.

<SourceLink file="nativelink-store/src/azure_blob_store.rs" symbol="AzureBlobStore" />

**An endpoint override.** `endpoint` replaces the host portion only; the
container is still appended to whatever you give. This is how you point the
store at an emulator such as Azurite:

```json5
{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "azure",
    account_name: "devstoreaccount1",
    container: "nativelink-cas",
    endpoint: "http://127.0.0.1:10000/devstoreaccount1",
  },
},
```

That address is Azurite's default. Plain `http://` is accepted as-is; the
Azure store does not read `insecure_allow_http`. Workload Identity is still
the credential in this mode, so the three environment variables must be
present; if the emulator cannot accept such a token, use `sas_url` instead.

**A SAS URL.** `sas_url` takes precedence over all of the above: when it is
set, `account_name`, `container` and `endpoint` are ignored for URL
construction and no credential is attached, because the signature in the URL
*is* the credential.

```json5
{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "azure",
    account_name: "mystorageaccount",
    container: "nativelink-cas",
    sas_url: "${AZURE_CAS_SAS_URL}",
  },
},
```

<Callout type="warn" title="A SAS URL is a bearer credential with an expiry">
  Anyone holding the URL has whatever the signature grants, so keep it in the
  environment rather than the config file, and note that the store will start
  failing the moment the signature expires, with an authentication error.
  Prefer Workload Identity where you have the choice.
</Callout>

## Options worth setting [#options-worth-setting]

| Field | Default | What it decides |
| --- | --- | --- |
| `key_prefix` | none | Namespacing inside one container; give the CAS `cas/` and the AC `ac/` |
| `endpoint` | derived | Host override; ignored when `sas_url` is set |
| `sas_url` | none | Full pre-signed URL; overrides the derived endpoint and the credential |
| `retry` | | Exponential backoff with jitter, shared with every other provider |
| `consider_expired_after_s` | `0` (never) | How old a blob may be and still count as present |
| `max_retry_buffer_per_request` | 5 MiB | How much of a single-request upload is buffered so it can be retried |
| `multipart_max_concurrent_uploads` | 10 | Concurrent staged-block uploads within one large blob |

Blobs under 5 MiB with a known size are uploaded in one request; anything
larger, or of unknown size, is staged as blocks of at least 5 MiB and
committed with a block list. `retry` and `consider_expired_after_s` behave as
described in [the shared options section](/how-to/stores/s3-and-compatible);
`insecure_allow_http` and `disable_http2` are accepted but not read by the
Azure store.

## Steps [#steps]

<Steps>

<li>

**Create the storage account and container** in the region your workers run
in. A standard general-purpose v2 account on hot tier is the right default;
cool and archive tiers price retrieval in a way a build cache cannot afford.

</li>

<li>

**Grant access.** Assign the workload identity the Storage Blob Data
Contributor role scoped to the container, or mint a SAS with read and write.
The store never lists or deletes blobs.

</li>

<li>

**Set the Workload Identity variables** (`AZURE_TENANT_ID`, `AZURE_CLIENT_ID`,
`AZURE_FEDERATED_TOKEN_FILE`) in the process environment, unless you use
`sas_url`. Uncommitted blocks left by a process killed mid-upload are discarded
by Azure after a week on their own.

</li>

<li>

**Add the store** as the `slow` half of a `fast_slow` pair whose `fast` half is
a local [filesystem store](/how-to/stores/filesystem).

</li>

<li>

**Run a build twice**: once cold, once after emptying the local fast tier.

</li>

</Steps>

<VerifyBlock>

- Blobs appear under `key_prefix` in the container during the first build.
- The second build is a cache hit even after `content_path` is emptied, which
  proves reads fall through to the container.
- Container request counts are far lower than blob counts, which proves the
  fast tier is absorbing repeat reads.
- No uncommitted blocks accumulate between builds.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Every request fails with a permission error" id="every-request-fails-with-a-permission-error">

Either the workload identity lacks Storage Blob Data Contributor on the
container, or a `sas_url` has expired. A SAS expiry surfaces as an
authentication failure, so check the signature's window before assuming the
role assignment is wrong. If the store did not start at all with a message
about the Workload Identity credential, the `AZURE_*` variables are missing.

</Accordion>

<Accordion title="Requests go somewhere unexpected" id="requests-go-somewhere-unexpected">

`sas_url` wins over everything. If it is set, `account_name`, `container` and
`endpoint` play no part in building the URL. Unset it to go back to the derived
endpoint.

</Accordion>

<Accordion title="The container is not found" id="the-container-is-not-found">

The store does not create containers. Create it in the storage account first,
and check that `container` matches exactly; container names are lowercase.

</Accordion>

<Accordion title="Connecting to Azurite fails" id="connecting-to-azurite-fails">

`endpoint` must include the account path segment:
`http://127.0.0.1:10000/devstoreaccount1`, not just the host and port. The
container is appended to whatever you give. Without `sas_url` the store still
needs the Workload Identity variables to start, and the emulator has to accept
the resulting token; a `sas_url` minted for the emulator avoids both.

</Accordion>

<Accordion title="Blobs exist in the container but NativeLink reports them missing" id="blobs-exist-in-the-container-but-nativelink-reports-them-missing">

Check `consider_expired_after_s`. A non-zero value makes blobs older than that
window invisible on purpose, so they pair with a matching lifecycle rule. If
you didn't intend that, set it back to `0`.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Deduplication, compression and existence caching in front of the container,
  the layers that decide what it actually costs.
</NextStep>

<NextStep href="/how-to/stores/s3-and-compatible" kind="aside" title="S3 and compatible">
  The same store type with a different provider, and the shared options in
  full.
</NextStep>

# https://docs.nativelink.com/how-to/stores/redis

**Redis**: Back the cache with Redis (standalone, sentinel or cluster) and the concurrency, chunking and timeout settings that keep it from timing out under a real build.

**Who this is for:** anyone who wants several NativeLink processes to share one
low-latency store, or who needs the scheduler's pub/sub subscriptions.
**What you'll have at the end:** a Redis-backed store with concurrency and
timeouts set so it survives a burst. **Time:** twenty minutes.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and a reachable Redis instance.
</Prerequisites>

Redis sits between the filesystem and object storage: it is shared across
machines like a bucket, and roughly a millisecond away like a local disk. What
it is not is cheap per byte. Use it for small, hot, shared objects (the Action
Cache, scheduler state, the index side of a `dedup` store) and put the bulk
CAS somewhere that prices storage by the gigabyte.

## The recipe [#the-recipe]

```json5
stores: [
  {
    name: "AC_MAIN_STORE",
    redis_store: {
      addresses: ["redis://127.0.0.1:6379/"],
      mode: "standard",
      key_prefix: "nativelink:ac:",
    },
  },
],
```

`addresses` is the only required field. The URL carries everything Redis needs:
scheme, optional credentials, host, port and database index.
`redis://username:password@redis-server-url:6380/99` selects port 6380 and
database 99. Keep credentials out of the file by writing
`redis://${REDIS_USER}:${REDIS_PASSWORD}@redis:6379/` and exporting the
variables; see [The config file](/configuration/config-file).

<SourceLink file="nativelink-config/src/stores.rs" symbol="RedisSpec" />

## Standalone, sentinel or cluster [#standalone-sentinel-or-cluster]

`mode` takes `standard` (the default), `sentinel` or `cluster`.

| Mode | What goes in `addresses` |
| --- | --- |
| `standard` | Exactly one server URL |
| `sentinel` | Exactly one sentinel URL, not a data node; the master name comes from a `?sentinelServiceName=` query parameter (default `master`), and a `redis+sentinel://` scheme is accepted |
| `cluster` | Any subset of the cluster's nodes; the client discovers the rest |

The store connects to the sentinel, asks it for the current master, and
re-resolves the master when a write hits a demoted replica or the connection
drops. Passing more than one address in `standard` or `sentinel` mode is
rejected at startup.

<SourceLink file="nativelink-store/src/redis_store.rs" symbol="RedisStore" />

`key_prefix` namespaces one Redis instance across several stores, and matters
more here than with object storage: a Redis instance is much more likely to be
shared with something else you run. Give the CAS, the AC and the scheduler
distinct prefixes.

## Sizing it so it doesn't time out [#sizing-it-so-it-doesnt-time-out]

Redis is single-threaded per node, so the settings that matter are the ones
bounding how much work you can have in flight.

| Field | Default | What it decides |
| --- | --- | --- |
| `max_client_permits` | 500 | Ceiling on concurrent actions against the store |
| `connection_pool_size` | 3 | Accepted but not used by the current client, which holds one multiplexed connection |
| `read_chunk_size` | 64 KiB | Bytes pulled per chunk when streaming a blob out |
| `max_chunk_uploads_per_update` | 10 | Concurrent chunk uploads within one write |
| `command_timeout_ms` | 10000 | Per-command ceiling |
| `connection_timeout_ms` | 3000 | Ceiling on establishing a connection |
| `health_check_timeout_ms` | 4000 | Ceiling on the health-check `PING` |
| `scan_count` | 10000 | `COUNT` hint per `SCAN` iteration |
| `key_ttl_s` | 0 (never) | Expiry set on every key this store writes; must exceed the longest upload |

`max_client_permits` is the most useful lever. It exists precisely to stop
timeouts caused by many, many inflight actions: when the ceiling is reached,
work queues instead of piling onto Redis and blowing through
`command_timeout_ms`. Lower it if you see timeouts under load; that is almost
always the right first move.

Peak memory in the client process is roughly
`read_chunk_size × max_chunk_uploads_per_update × concurrent uploads`. Working
backwards from available memory, divide by about ten to leave room for
everything else the process is doing.

<Callout type="warn" title="Raising `read_chunk_size` can cause the timeouts it looks like it should fix">
  A larger chunk means a single command moves more data, so a chunk that is too
  large will exceed `command_timeout_ms` on a busy or distant Redis and fail the
  whole operation. If large blobs are timing out, lower the chunk size or raise
  the timeout, don't raise the chunk size.
</Callout>

`max_count_per_cursor` (default 1500) bounds how many entries one cursor
returns; it exists to reduce thundering-herd effects when many workers hit the
provisioner at once.

<Callout type="info" title="Three fields are deprecated">
  `response_timeout_s` is superseded by `command_timeout_ms` and is ignored
  with a warning; `connection_timeout_s` is superseded by
  `connection_timeout_ms` and is still honoured (converted to milliseconds)
  if the millisecond field is unset, but setting both is an error.
  `broadcast_channel_capacity` is unused by the current client.
  `experimental_pub_sub_channel` remains, and is what scheduler
  subscriptions ride on.
</Callout>

## Where Redis actually earns its place [#where-redis-actually-earns-its-place]

Rather than as a bulk CAS, Redis is usually the fast or index half of a
composition:

```json5
{
  name: "CAS_MAIN_STORE",
  dedup: {
    index_store: {
      redis_store: {
        addresses: ["redis://redis:6379/"],
        key_prefix: "nativelink:index:",
      },
    },
    content_store: {
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "my-nativelink-cas",
        key_prefix: "cas/",
      },
    },
  },
},
```

The index is small, hot and shared by every process; the content is large, cold
and cheap per byte. [Compose stores](/how-to/stores/compose-stores) covers the
rest of these shapes.

## Steps [#steps]

<Steps>

<li>

**Provision Redis** with enough memory for what you intend to keep, and decide
its eviction policy on the Redis side, because NativeLink does not manage it.

</li>

<li>

**Add the store** with `addresses`, the right `mode`, and a `key_prefix` that
won't collide with anything else using that instance.

</li>

<li>

**Reference credentials through the environment** rather than writing them into
the URL in the config file.

</li>

<li>

**Run a build**, then run it again from a second machine pointed at the same
Redis, to confirm the store really is shared.

</li>

<li>

**Load-test it** and watch for `command_timeout_ms` errors. If you see them,
lower `max_client_permits` before touching anything else.

</li>

</Steps>

<VerifyBlock>

- Keys appear under `key_prefix`: `SCAN` with a `MATCH` on the prefix returns
  entries while a build runs.
- A second machine pointed at the same Redis gets cache hits for objects the
  first one uploaded.
- No command timeouts in the logs under your normal peak load.
- Redis memory use plateaus where its own eviction policy says it should,
  rather than climbing to the instance limit.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Timeouts under load, fine when idle" id="timeouts-under-load-fine-when-idle">

Too much in flight. Lower `max_client_permits` so work queues in NativeLink
instead of piling onto a single-threaded Redis. Only after that, consider
raising `command_timeout_ms`.

</Accordion>

<Accordion title="Large blobs time out, small ones don't" id="large-blobs-time-out-small-ones-dont">

`read_chunk_size` is too large for the round-trip time to this Redis: one
command is moving more data than fits in `command_timeout_ms`. Lower the chunk
size.

</Accordion>

<Accordion title="Redis runs out of memory" id="redis-runs-out-of-memory">

NativeLink does not evict from Redis; that's the instance's `maxmemory` and
`maxmemory-policy`. The one knob on this side is `key_ttl_s`, which puts an
expiry on every key the store writes (meant for stores whose consumer can
stop, not for scheduler state). Set the Redis-side limits, and reconsider
whether the bulk CAS should live here at all. For large blobs, a filesystem or object store tier is both
cheaper and better suited.

</Accordion>

<Accordion title="Connection failures against a sentinel or cluster setup" id="connection-failures-against-a-sentinel-or-cluster-setup">

Check `mode`. For `sentinel`, `addresses` must hold exactly one sentinel URL,
not a data node, and the master name must match your sentinel configuration
(`?sentinelServiceName=mymaster`; the default is `master`); pointing at a
replica directly gives read-only errors on write. For `cluster`, any subset of
nodes works and the client discovers the rest.

</Accordion>

<Accordion title="Two deployments are seeing each other's entries" id="two-deployments-are-seeing-each-others-entries">

They share a `key_prefix`, or neither sets one. Give each deployment its own,
or its own database index in the URL.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Where a Redis store belongs inside a larger composition, usually as the
  index or fast half, not the bulk store.
</NextStep>

<NextStep href="/configuration/scheduler-and-workers" kind="aside" title="Schedulers and workers">
  Redis also backs scheduler state when more than one scheduler shares a
  queue.
</NextStep>

# https://docs.nativelink.com/how-to/stores/mongo

**MongoDB**: Back the cache with MongoDB: collections, write concern, timeouts, and what the change-streams flag is for.

**Who this is for:** anyone who already operates MongoDB. **What you'll have
at the end:** a Mongo-backed CAS behind a local fast tier. **Time:**
twenty-five minutes.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and a reachable MongoDB deployment.
</Prerequisites>

Mongo lands between [Redis](/how-to/stores/redis) and
[object storage](/how-to/stores/s3-and-compatible): durable, shared across
machines, single-digit milliseconds away, and priced by the cluster you run
rather than per gigabyte stored. [Storage backends](/how-to/stores) compares
all of them on the same axes.

## The recipe [#the-recipe]

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        memory: {
          eviction_policy: { max_bytes: "1gb" },
        },
      },
      slow: {
        experimental_mongo: {
          connection_string: "${MONGO_URI}",
          database: "nativelink",
          cas_collection: "cas",
          key_prefix: "prod:",
        },
      },
    },
  },
],
```

`connection_string` is the only required field and takes either form Mongo
accepts, `mongodb://localhost:27017` or
`mongodb+srv://cluster.mongodb.net` for Atlas. It carries the credentials, so
keep it in the environment and reference it with `${VAR}`; see
[The config file](/configuration/config-file).

Everything else has a default: `database` is `nativelink`, `cas_collection` is
`cas`, `scheduler_collection` is `scheduler`, and `key_prefix` is empty.

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalMongoSpec" />

## Change streams and the scheduler [#change-streams-and-the-scheduler]

The Mongo store implements the scheduler-store interface, and
`enable_change_streams: true` is what that interface needs: the store refuses
to hand out scheduler subscriptions without it. In v1.6.5, though, the `simple`
scheduler's `experimental_backend` accepts only `memory` and `redis`, so there
is no config path that puts a Mongo store behind the scheduler. Treat this
store as a CAS and Action Cache backend, and see
[Redis](/how-to/stores/redis) for a shared scheduler.

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="ExperimentalSimpleSchedulerBackend" />

Leave `enable_change_streams` off for a store used as a CAS. It buys nothing
there.

<Callout type="warn" title="Change streams require a replica set">
  Mongo only offers change streams on a replica set or sharded cluster; a
  standalone `mongod` cannot serve them, no matter what this flag says. Atlas
  is always a replica set; a local single-node deployment needs
  `--replSet` and an initiated set.
</Callout>

## Options worth setting [#options-worth-setting]

| Field | Default | What it decides |
| --- | --- | --- |
| `database` | `nativelink` | Database name |
| `cas_collection` | `cas` | Collection holding CAS objects |
| `scheduler_collection` | `scheduler` | Collection holding scheduler state |
| `key_prefix` | empty | Namespacing when several deployments share one database |
| `read_chunk_size` | 64 KB | Bytes pulled per chunk when streaming a blob out |
| `max_requests` | unlimited | Ceiling on concurrent requests to the deployment |
| `connection_timeout_ms` | 3000 | Applied as both the driver's connect timeout and its server-selection timeout |
| `command_timeout_ms` | 10000 | Accepted and defaulted, but not applied to any driver option in v1.6.5 |
| `enable_change_streams` | `false` | Required for scheduler subscriptions, which nothing uses yet (see above) |

Write concern is exposed as three separate fields mirroring Mongo's own:
`write_concern_w` (a number like `1`, or the string `"majority"`),
`write_concern_j` (whether to wait for the journal), and
`write_concern_timeout_ms`. All three are optional and unset means the
deployment's own default applies, but `write_concern_j` or
`write_concern_timeout_ms` without `write_concern_w` is rejected at startup.
For a CAS, `"majority"` is the honest
setting, because a cache entry acknowledged by one node and then lost to a failover is
a cache entry a client believes exists.

`max_requests` is the equivalent of Redis's permit ceiling and is unlimited by
default. Set it if bursts are overwhelming the deployment; queuing in
NativeLink beats timing out at Mongo. Zero is rejected at startup.

<SourceLink file="nativelink-store/src/mongo_store.rs" symbol="ExperimentalMongoStore::new" />

<Callout type="info" title="`max_concurrent_uploads` is deprecated and unused">
  It survives in the schema for compatibility. Setting it only logs a warning;
  use `max_requests` to bound concurrency.
</Callout>

## Steps [#steps]

<Steps>

<li>

**Provision the deployment**: a replica set if you want writes to survive a
failover with `"majority"` write concern.

</li>

<li>

**Create a user** with read and write on the database, and put the full
connection string in the environment rather than the config file.

</li>

<li>

**Add the store**, choosing a `key_prefix` if this database is shared with
another NativeLink deployment.

</li>

<li>

**Set write concern** to `"majority"` for anything you intend to trust.

</li>

<li>

**Run a build twice**, the second time from a different machine, to confirm the
store is genuinely shared.

</li>

</Steps>

<VerifyBlock>

- Documents appear in the CAS collection during the first build.
- A second machine pointed at the same deployment gets cache hits for objects
  the first one uploaded.
- No connection or server-selection timeouts in the logs at your normal peak
  load.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Can this store back the scheduler?" id="workers-never-pick-up-queued-actions">

Not in v1.6.5. The store implements the scheduler-store interface behind
`enable_change_streams`, but the `simple` scheduler's `experimental_backend`
only accepts `memory` and `redis`, so there is nothing to point at it. Use
[Redis](/how-to/stores/redis) for a shared scheduler.

</Accordion>

<Accordion title="Connection failures against Atlas" id="connection-failures-against-atlas">

Use the `mongodb+srv://` form, and check that the deployment's IP access list
includes wherever NativeLink runs. An access-list rejection often surfaces as a
connection timeout rather than as an authorization error.

</Accordion>

<Accordion title="Timeouts under load" id="timeouts-under-load">

Set `max_requests` so concurrency queues in NativeLink instead of arriving all
at once, then consider raising `connection_timeout_ms`, which also bounds
server selection. Also check the deployment's own connection limit; Atlas
tiers cap it low.

</Accordion>

<Accordion title="Cache entries disappear after a failover" id="cache-entries-disappear-after-a-failover">

Write concern. With `write_concern_w` unset or `1`, a write acknowledged by the
primary can be rolled back when that primary is replaced. Set `"majority"`.

</Accordion>

<Accordion title="Two deployments are seeing each other's entries" id="two-deployments-are-seeing-each-others-entries">

They share a database with no `key_prefix`, or the same one. Give each its own
prefix, or its own database.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  Putting a local fast tier in front of this one, and the rest of the
  composition vocabulary.
</NextStep>

<NextStep href="/how-to/stores/redis" kind="aside" title="Redis">
  The backend that can hold scheduler state in v1.6.5, if you need several
  schedulers to share one queue.
</NextStep>

# https://docs.nativelink.com/how-to/stores/oci-object-storage

**Oracle Cloud (OCI) Object Storage**: Back the cache with OCI Object Storage through its S3 Compatibility API: Customer Secret Keys, the two adjustments the store makes for you, and what has actually been verified.

**Who this is for:** anyone running NativeLink on Oracle Cloud who wants the
durable tier to be an OCI bucket. **What you'll have at the end:** a
bucket-backed CAS and Action Cache authenticated with a Customer Secret Key.
**Time:** thirty minutes, most of it namespace and policy setup.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and an OCI tenancy you can create a bucket in.
</Prerequisites>

OCI is one `provider` value of the same cloud object store used by
[S3 and compatible](/how-to/stores/s3-and-compatible),
[GCS](/how-to/stores/gcs) and [Azure Blob](/how-to/stores/azure): six
providers over one store type, sharing the same multipart, retry and streaming
code paths.

## How it works [#how-it-works]

[OCI Object Storage](https://www.oracle.com/cloud/storage/object-storage/)
exposes an [S3 Compatibility
API](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm),
and the `oci` provider is a thin adapter pointing the S3 store at it. Two
OCI-specific adjustments are applied automatically, and you do not configure
either:

- **Path-style addressing.** OCI requires `endpoint/bucket/key` rather than
  virtual-hosted `bucket.endpoint/key` URLs. The endpoint is derived as
  `https://{namespace}.compat.objectstorage.{region}.oci.customer-oci.com`.
- **Checksums downgraded to "when required".** The AWS SDK otherwise adds a
  default trailing checksum that forces `Content-Encoding: aws-chunked`, which
  OCI rejects with `501 NotImplemented: AWS chunked encoding not supported`.
  Without the downgrade, small single-`PUT` objects (notably Action Cache
  entries) fail.

<SourceLink file="nativelink-store/src/oci_store.rs" symbol="OciStore" />

## Before you start [#before-you-start]

You need three things.

**A bucket.** Create one in the Console under *Storage → Buckets*, or with
`oci os bucket create`. Note its region, for example `ap-mumbai-1`.

**Your Object Storage namespace.** A tenancy-wide identifier shown under
*Tenancy details*, or from `oci os ns get`. It goes into the derived endpoint.

**A Customer Secret Key.** See below.

## Authentication [#authentication]

The S3 Compatibility API authenticates with Customer Secret Keys, a static
access-key and secret-key pair used as `access_key_id` and
`secret_access_key`, signed with AWS SigV4.

Generate one in the Console under *Profile → User settings → Customer secret
keys → Generate secret key*. The secret is shown **only once** at generation time, so copy it
immediately; the access key stays listed in the table and can be copied
whenever you like. OCI has no way to retrieve a secret after generation, so if
you lose it you rotate rather than recover.

Grant the associated user read and write on the bucket with a least-privilege
policy:

```
Allow group <your-group> to manage object-family in compartment <compartment> where target.bucket.name='<bucket>'
```

Supply both keys through environment variables and reference them with `${VAR}`
rather than pasting them into the config; see
[The config file](/configuration/config-file).

<Callout type="warn" title="Only Customer Secret Keys work here">
  The S3 Compatibility API supports no other credential. OCI-native
  authentication (API signing keys, instance principals, resource principals)
  is not available through this store. If you run NativeLink on OCI compute and
  want keyless workload identity, that is not currently supported: you use
  static Customer Secret Keys and rotate them yourself.
</Callout>

## The recipe [#the-recipe]

A minimal CAS and Action Cache sharing one bucket, segmented by `key_prefix`:

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    experimental_cloud_object_store: {
      provider: "oci",
      namespace: "storagenamespace",
      region: "ap-mumbai-1",
      bucket: "nativelink-cas",
      access_key_id: "${OCI_ACCESS_KEY_ID}",
      secret_access_key: "${OCI_SECRET_ACCESS_KEY}",
      key_prefix: "cas/",
      retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
    },
  },
  {
    name: "AC_MAIN_STORE",
    experimental_cloud_object_store: {
      provider: "oci",
      namespace: "storagenamespace",
      region: "ap-mumbai-1",
      bucket: "nativelink-cas",
      access_key_id: "${OCI_ACCESS_KEY_ID}",
      secret_access_key: "${OCI_SECRET_ACCESS_KEY}",
      key_prefix: "ac/",
      retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
    },
  },
],
```

<SourceLink file="nativelink-config/examples/oci_backend.json5" />

That is the smallest thing that works, not the thing to run. In production,
wrap the CAS in `verify` for integrity and front it with a `fast_slow`
filesystem tier to cut round trips, exactly as for AWS.
[Compose stores](/how-to/stores/compose-stores) has the shapes.

### Fields [#fields]

| Field | Required | What it is |
| --- | --- | --- |
| `namespace` | yes | Object Storage namespace; used to derive the endpoint |
| `region` | yes | OCI region id, for example `ap-mumbai-1`; also the SigV4 signing region |
| `bucket` | yes | Target bucket |
| `access_key_id` | no | Customer Secret Key access key; if either key is unset the store falls back to the AWS default credential chain, which reads `AWS_*` names, not OCI ones |
| `secret_access_key` | no | Customer Secret Key secret |
| `key_prefix`, `retry`, … | no | The [shared cloud object options](/how-to/stores/s3-and-compatible) |

<SourceLink file="nativelink-config/src/stores.rs" symbol="ExperimentalOciSpec" />

OCI also accepts `us-east-1` as a `region` value to target the tenancy home
region, which is occasionally useful when a tool insists on an AWS region name.

## Steps [#steps]

<Steps>

<li>

**Create the bucket** and note its region and your tenancy's Object Storage
namespace.

</li>

<li>

**Generate a Customer Secret Key** and copy the secret immediately, because it is
shown once.

</li>

<li>

**Apply a least-privilege policy** scoped to that bucket, and export both keys
into the process environment.

</li>

<li>

**Add a lifecycle rule** aborting incomplete multipart uploads after a few days
as a backstop.

</li>

<li>

**Add the stores** with distinct `key_prefix` values, then run a build twice.

</li>

</Steps>

<VerifyBlock>

- Objects appear under both `cas/` and `ac/` in the bucket during the first
  build.
- The second build is a cache hit, which proves the Action Cache round trip
  works. That is the path the checksum adjustment exists for.
- A large artifact uploads and downloads intact, which exercises multipart.
- No incomplete multipart uploads accumulate between builds.

</VerifyBlock>

## Operational notes [#operational-notes]

**Reap incomplete multipart uploads.** When a multipart upload fails mid-flight
the store issues `AbortMultipartUpload` to clean up, and OCI honors both
`ListMultipartUploads` and `AbortMultipartUpload`. A process killed *before* it
can abort still leaves orphaned parts, which is what the lifecycle rule is for.

**Status.** This backend lives under `experimental_cloud_object_store`. When it
was added, its core paths (read, write, existence checks, ranged reads,
multipart and integrity) were verified against live OCI, including an
end-to-end Bazel remote cache round trip, a 2 GiB multipart upload of roughly 400 parts with full byte
verification, multipart abort and cleanup, and 200-way concurrent load. It has
not been benchmarked at sustained production scale or duration.

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Small objects fail with 501 NotImplemented: AWS chunked encoding not supported" id="small-objects-fail-with-501-notimplemented-aws-chunked-encoding-not-supported">

Something is re-adding the default trailing checksum the store removes. This is
the failure the checksum downgrade exists to prevent, and it shows up first on
Action Cache entries because they are small single-`PUT` objects. Confirm you
are on a NativeLink version that carries the `oci` provider rather than
pointing the `aws` provider at an OCI endpoint by hand.

</Accordion>

<Accordion title="Every request fails to authenticate" id="every-request-fails-to-authenticate">

The S3 Compatibility API takes Customer Secret Keys only. An API signing key,
an instance principal or a resource principal will not work here. Check also
that `region` matches the bucket's region. It is the SigV4 signing region as
well as part of the endpoint.

</Accordion>

<Accordion title="The endpoint cannot be resolved" id="the-endpoint-cannot-be-resolved">

`namespace` is wrong. It is the tenancy's Object Storage namespace from
`oci os ns get`, not the tenancy name and not the compartment.

</Accordion>

<Accordion title="You lost the secret key" id="you-lost-the-secret-key">

It cannot be retrieved; OCI shows it once. Generate a new Customer Secret Key
and delete the old one.

</Accordion>

<Accordion title="Can the CAS and Action Cache share one bucket?" id="can-the-cas-and-action-cache-share-one-bucket">

Yes, and they should. Segment them with `key_prefix` (`cas/` and `ac/` in the
example above) rather than provisioning two buckets. Lifecycle rules can still
treat the prefixes differently.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  The `verify` wrapper and the `fast_slow` tier that turn this into something
  you'd run in production.
</NextStep>

<NextStep href="/how-to/stores/s3-and-compatible" kind="aside" title="S3 and compatible">
  The shared cloud object options (retry, expiry, buffering) documented in
  full.
</NextStep>

# https://docs.nativelink.com/how-to/stores/compose-stores

**Compose stores**: The wrapper vocabulary (fast_slow, verify, dedup, existence_cache, size_partitioning, shard and the rest) and the order the layers have to go in.

**Who this is for:** anyone who has picked a backend and now needs it to be
fast, cheap and trustworthy. **What you'll have at the end:** a layered store
you can explain line by line, and the vocabulary to read anyone else's config.
**Time:** thirty minutes.

<Prerequisites>
  A config file with at least one backend in it. See [Storage backends](/how-to/stores).
</Prerequisites>

Most store types in NativeLink are not places to put bytes. They are wrappers:
each one takes another store and changes something about how it behaves. A real
CAS store is four or five of these nested inside each other, and the nesting
order is the design.

The things at the bottom of the nesting are the backends:
[filesystem](/how-to/stores/filesystem),
[S3 and compatible](/how-to/stores/s3-and-compatible),
[GCS](/how-to/stores/gcs), [Azure Blob](/how-to/stores/azure),
[OCI](/how-to/stores/oci-object-storage), [Redis](/how-to/stores/redis),
[Mongo](/how-to/stores/mongo), each with a page of its own under
[Storage backends](/how-to/stores). Everything on this page goes above them.

## The two ways stores refer to each other [#the-two-ways-stores-refer-to-each-other]

A wrapper can hold its inner store inline:

```json5
{
  name: "CAS_MAIN_STORE",
  verify: {
    verify_size: true,
    verify_hash: true,
    backend: {
      filesystem: {
        content_path: "/var/lib/nativelink/cas/content",
        temp_path: "/var/lib/nativelink/cas/tmp",
        eviction_policy: { max_bytes: "50gb" },
      },
    },
  },
},
```

Or it can name a store defined elsewhere in the `stores` array, with
`ref_store`:

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    verify: {
      verify_size: true,
      verify_hash: true,
      backend: { ref_store: { name: "CAS_DISK" } },
    },
  },
  {
    name: "CAS_DISK",
    filesystem: {
      content_path: "/var/lib/nativelink/cas/content",
      temp_path: "/var/lib/nativelink/cas/tmp",
      eviction_policy: { max_bytes: "50gb" },
    },
  },
],
```

Both configs describe the same thing. Order in the array does not matter, because
`ref_store` is resolved after every entry is read, so a store may reference one
defined below it. Reach for `ref_store` when two stores genuinely share one
backend, or when nesting has got deep enough to be unreadable.

<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

## The wrappers [#the-wrappers]

### `fast_slow`: a tier in front of a tier [#fast_slow--a-tier-in-front-of-a-tier]

The single most important wrapper. Reads try `fast` first and fall through to
`slow` on a miss, populating `fast` as they go. Writes go to both.

```json5
{
  name: "CAS_MAIN_STORE",
  fast_slow: {
    fast: {
      filesystem: {
        content_path: "/var/lib/nativelink/cas/content",
        temp_path: "/var/lib/nativelink/cas/tmp",
        eviction_policy: { max_bytes: "50gb", evict_bytes: "5gb" },
      },
    },
    slow: {
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "my-nativelink-cas",
        key_prefix: "cas/",
      },
    },
  },
},
```

Two knobs are worth knowing. `fast_direction` and `slow_direction` restrict
what each half participates in: `update` means writes land here but reads
never consult it, which only makes sense on the fast half; `get`, `read_only`
and the default `both` do what they sound like.

`bypass_dedup_threshold_bytes` makes reads at or above a given size skip the
fast tier entirely and stream straight from `slow`. It defaults to `0`, which
disables the bypass; 256 MiB is a reasonable starting point if huge artifacts
are evicting everything useful out of your fast tier.

### `verify`: refuse to store corruption [#verify--refuse-to-serve-corruption]

Checks the size and hash of an upload against the digest it is filed under,
and rejects the write on a mismatch. Reads pass straight through to the
backend unchecked, so this protects what enters the store, not what a backend
later hands back.

<SourceLink file="nativelink-store/src/verify_store.rs" symbol="VerifyStore" />

```json5
{
  name: "CAS_MAIN_STORE",
  verify: {
    verify_size: true,
    verify_hash: true,
    backend: { ref_store: { name: "CAS_DISK" } },
  },
},
```

<Callout type="warn" title="Verify the CAS, not the Action Cache">
  A CAS entry is addressed by the hash of its content, so both checks are
  meaningful. An Action Cache entry is not: its key is the action digest, and
  its value is a result message, so `verify_size` and `verify_hash` should both
  be `false` for an AC store. Setting them true there rejects every valid
  update.
</Callout>

### `completeness_checking`: the Action Cache wrapper [#completeness_checking--the-action-cache-wrapper]

The AC's counterpart. Before returning an action result as a hit, it confirms
every output digest that result references still exists in the CAS. Without it,
a client gets a cache hit pointing at blobs that have since been evicted.

```json5
{
  name: "AC_MAIN_STORE",
  completeness_checking: {
    backend: { ref_store: { name: "AC_DISK" } },
    cas_store: { ref_store: { name: "CAS_MAIN_STORE" } },
  },
},
```

This is why the AC and CAS eviction policies are coupled in practice: evicting
aggressively from the CAS quietly invalidates AC entries, and this wrapper is
what turns that from a confusing build failure into a clean cache miss.

### `existence_cache`: stop asking the same question [#existence_cache--stop-asking-the-same-question]

Caches the answer to "does this digest exist" so `FindMissingBlobs` doesn't hit
the backend for every blob of every build.

```json5
{
  name: "CAS_MAIN_STORE",
  existence_cache: {
    backend: { ref_store: { name: "CAS_SLOW" } },
    eviction_policy: { max_seconds: 3600 },
  },
},
```

The `eviction_policy` here governs the existence *index*, not the data. It is
a small map of digests, so it can hold far more entries than the store itself.
This is the highest-leverage wrapper in front of object storage, where every
existence check is a billed request.

### `compression`: smaller on the backend [#compression--smaller-on-the-backend]

Compresses blobs with LZ4 before they reach the wrapped store.

```json5
{
  name: "CAS_SLOW_STORE",
  compression: {
    compression_algorithm: { lz4: {} },
    backend: { ref_store: { name: "CAS_BUCKET" } },
  },
},
```

LZ4 is the only algorithm. `block_size` defaults to 64 KiB, and
`max_decode_block_size` defaults to match it. The latter bounds how much a
malformed block can ask the decoder to allocate, so raise it only alongside
`block_size`.

This is store-level compression, on the way to the backend. It is a different
thing from [remote cache compression](/how-to/stores/compression), which
compresses on the wire between the client and NativeLink. The two are
independent and can both be on.

### `dedup`: store the changed parts only [#dedup--store-the-changed-parts-only]

Splits blobs into content-defined chunks, stores each chunk once, and keeps a
per-blob index of which chunks it is made of.

```json5
{
  name: "CAS_MAIN_STORE",
  dedup: {
    index_store: { ref_store: { name: "FAST_INDEX" } },
    content_store: { ref_store: { name: "CAS_BUCKET" } },
    min_size: 65536,
    normal_size: 262144,
    max_size: 524288,
  },
},
```

`index_store` should be small and fast; `content_store` is the large slow one.
The three size fields bound the chunker: `normal_size` is the target average,
`min_size` and `max_size` are hard limits, and they default to 64 KiB, 256 KiB
and 512 KiB. Every blob gets an index entry, even one small enough to be a
single chunk; there is no size below which the store skips the index.

<SourceLink file="nativelink-store/src/dedup_store.rs" symbol="DedupStore" />

`max_concurrent_fetch_per_get` (default 10) bounds parallel chunk fetches.
Worst-case memory per `get()` is about
`max_concurrent_fetch_per_get × max_size`, so the defaults cost roughly 5 MB
per in-flight read.

### `size_partitioning`: route by blob size [#size_partitioning--route-by-blob-size]

Sends everything under `size` to one store and everything at or above it to
another.

```json5
{
  name: "CAS_MAIN_STORE",
  size_partitioning: {
    size: 262144,
    lower_store: { ref_store: { name: "SMALL_OBJECT_STORE" } },
    upper_store: { ref_store: { name: "DEDUPED_LARGE_STORE" } },
  },
},
```

The usual use is to keep small objects out of a `dedup` store, where the
per-blob index costs more than the deduplication saves. The other use is a hard
ceiling: point `upper_store` at `noop` and blobs above the threshold are
discarded rather than stored.

### `shard`: spread across several backends [#shard--spread-across-several-backends]

Distributes blobs across stores by digest, weighted.

```json5
{
  name: "CAS_MAIN_STORE",
  shard: {
    stores: [
      { store: { ref_store: { name: "SHARD_A" } }, weight: 1 },
      { store: { ref_store: { name: "SHARD_B" } }, weight: 1 },
      { store: { ref_store: { name: "SHARD_C" } }, weight: 2 },
    ],
  },
},
```

A store's share is its `weight` divided by the sum of all weights, so `SHARD_C`
above takes half. `weight` defaults to `1`, so an unweighted list is an even
split.

<Callout type="warn" title="Changing the shard set invalidates placement">
  Blobs are placed by hashing against the current set of stores and weights.
  Add, remove or reweight a shard and existing entries stop being looked for
  where they were put. Treat a shard change as a cache flush.
</Callout>

### `cache_metrics`: measure one layer [#cache_metrics--measure-one-layer]

Wraps a store to record hit rates and timings under a label:

```json5
{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: { ref_store: { name: "CAS_LAYERS" } },
  },
},
```

`cache_type` should be low-cardinality: `cas` or `ac`, not something per-user.
The wrapper is opt-in: stores not wrapped by it are built without it and pay
none of its hot-path timing cost. Wrap the layer whose behaviour you actually
want to see, not every layer.

### The three with nothing to configure [#the-three-with-nothing-to-configure]

`memory` takes an `eviction_policy` and nothing else. It is the right fast half
for a short-lived process and the wrong one for anything that must survive a
restart.

`grpc` points at another REAPI cache upstream, and can attach `headers` and
forward selected headers from the incoming request. Use it to front an existing
cache.

`noop` takes nothing at all and discards everything written to it, reporting
every read as a miss. It is how you deliberately turn a half of something off.

## The order the layers go in [#the-order-the-layers-go-in]

Reading a real config outside-in, each layer answers a different question:

<Mermaid>
{`flowchart TD
  A["cache_metrics<br/>what is this layer doing?"] --> B["verify<br/>is this data what it claims?"]
  B --> C["existence_cache<br/>do I have to ask the backend?"]
  C --> D["dedup<br/>have I stored these chunks already?"]
  D --> E["compression<br/>can each chunk be smaller?"]
  E --> F["fast_slow<br/>is it already nearby?"]
  F --> G["backend<br/>filesystem, bucket, redis, mongo"]`}
</Mermaid>

Four placements are load-bearing rather than stylistic.

**`verify` goes outermost.** It checks an upload against the digest the client
filed it under. Underneath compression or dedup, the bytes it sees are no
longer the bytes the digest describes.

**`existence_cache` goes above the expensive store, not below it.** Its whole
purpose is to avoid a round trip, so anything between it and the backend is
work it was meant to skip.

**`dedup` goes above `compression`, never below it.** Compressed bytes of
similar inputs are not similar, so a `dedup` store fed by a `compression`
wrapper finds nothing to share. Put `compression` on the dedup store's
`content_store`, as the full example below does.

**`fast_slow` goes innermost, just above the backends.** Layers above it apply
to both halves once, rather than being paid twice.

## A full example [#a-full-example]

The shipped GCS and Azure examples both use this shape, and it is a good
default for a cloud-backed CAS:

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    verify: {
      verify_size: true,
      verify_hash: true,
      backend: {
        dedup: {
          index_store: {
            fast_slow: {
              fast: {
                filesystem: {
                  content_path: "/data/index/content",
                  temp_path: "/data/index/tmp",
                  eviction_policy: { max_bytes: "500mb" },
                },
              },
              slow: {
                experimental_cloud_object_store: {
                  provider: "gcs",
                  bucket: "my-nativelink-cas",
                  key_prefix: "index/",
                },
              },
            },
          },
          content_store: {
            compression: {
              compression_algorithm: { lz4: {} },
              backend: {
                fast_slow: {
                  fast: {
                    filesystem: {
                      content_path: "/data/content/content",
                      temp_path: "/data/content/tmp",
                      eviction_policy: { max_bytes: "2gb" },
                    },
                  },
                  slow: {
                    experimental_cloud_object_store: {
                      provider: "gcs",
                      bucket: "my-nativelink-cas",
                      key_prefix: "cas/",
                    },
                  },
                },
              },
            },
          },
        },
      },
    },
  },
],
```

Read outside-in: verify what we serve; dedup so we store each chunk once; keep
the index on fast local disk backed by the bucket; compress the content;
tier that over the bucket too.

## Steps [#steps]

<Steps>

<li>

**Start with the backend alone** and confirm it works. Every layer you add
makes a failure harder to attribute.

</li>

<li>

**Add `fast_slow`** with a local filesystem fast half, and confirm the second
build is a hit with the fast tier emptied.

</li>

<li>

**Add `verify`** to the CAS, and *not* to the Action Cache.

</li>

<li>

**Add `completeness_checking`** to the Action Cache, pointing `cas_store` at
the CAS store by name.

</li>

<li>

**Add `existence_cache`** if the slow tier is object storage, and watch the
request count rather than the byte count to see whether it helped.

</li>

<li>

**Add `dedup` and `compression`** last, and only if artifact size is the
problem you actually have.

</li>

</Steps>

<VerifyBlock>

- The config loads. An unknown field anywhere in the nesting is rejected at
  startup, so a clean start means every layer name is right.
- A build is a cache hit on the second run with the fast tier emptied, which
  proves the layers pass through to the slow tier.
- Backend request counts are far below blob counts once `existence_cache` is
  in.
- Uploading a blob under a digest that does not match its content (for
  example with a hand-built ByteStream write) is rejected rather than stored.
  That's `verify` earning its place.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="The config is rejected at startup with an unknown field" id="the-config-is-rejected-at-startup-with-an-unknown-field">

Almost every store spec is strict about unknown fields, so this is usually a
field placed at the wrong nesting level: `eviction_policy` on the wrapper
rather than on the store, or a backend option one layer too high. Check what
the innermost brace you're inside actually is.

</Accordion>

<Accordion title="Every Action Cache lookup misses" id="every-action-cache-lookup-misses">

`verify` with `verify_hash` or `verify_size` true on an AC store. An AC entry's
key is the action digest and its value is a result message, so both checks
fail by construction and every AC update is rejected, leaving nothing to hit.
Set both false there, and use `completeness_checking` instead.

</Accordion>

<Accordion title="Cache hits point at blobs that no longer exist" id="cache-hits-point-at-blobs-that-no-longer-exist">

The Action Cache has no `completeness_checking` wrapper, so it hands out
results referencing outputs the CAS has since evicted. Add it with `cas_store`
pointing at the CAS store.

</Accordion>

<Accordion title="A ref_store name isn't found" id="a-ref_store-name-isnt-found">

The name is a store name from the `stores` array, matched exactly. Order does
not matter, so a forward reference is fine. A typo or a renamed store is the
usual cause.

</Accordion>

<Accordion title="Everything got slower after adding dedup" id="everything-got-slower-after-adding-dedup">

Small blobs. A blob that fits in one chunk gains nothing and still costs an
index write plus an extra existence check on every read. Put a
`size_partitioning` above the dedup store and route small objects straight to
the backend.

</Accordion>

<Accordion title="Entries went missing after changing the shard list" id="entries-went-missing-after-changing-the-shard-list">

Placement is derived from the set of stores and weights. Changing it moves
where blobs are looked for. Nothing rebalances, so treat it as a flush.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compression" title="Remote cache compression">
  The other kind of compression: on the wire, between the client and
  NativeLink.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  What these compositions look like once the three processes are separate.
</NextStep>

# https://docs.nativelink.com/how-to/stores/compression

**Remote cache compression**: Cut remote cache transfer bytes for compressible artifacts with REAPI zstd wire compression and Bazel's --remote_cache_compression.

**Who this is for:** anyone whose clients reach NativeLink across a real
network and whose artifacts compress well. **What you'll have at the end:**
zstd on the wire, with stores still holding raw bytes. **Time:** ten minutes.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and a Bazel client you can pass flags to.
</Prerequisites>

Build artifacts are often highly compressible (object files, archives, test
logs), but a conventional remote cache transfers them byte-for-byte. With wire
compression, clients and server exchange blobs as zstd-compressed streams and
only the wire bytes shrink: NativeLink decompresses on upload and compresses on
download at the gRPC boundary, so stores keep raw, uncompressed bytes and every
existing read path sees ordinary blobs.

NativeLink implements the server side of the REAPI
[`compressed-blobs`](https://github.com/bazelbuild/remote-apis/blob/main/build/bazel/remote/execution/v2/remote_execution.proto)
extension with `zstd` as the only supported non-identity compressor, matching
what Bazel's `--remote_cache_compression` flag speaks: ByteStream reads and
writes use `compressed-blobs/zstd/...` resource names, and the CAS batch RPCs
accept and serve zstd-compressed payloads.

## Requirements [#requirements]

- A Bazel client with `--remote_cache_compression` (available since Bazel 5).
- Wire compression is **optional and off until it is enabled somewhere in the
  NativeLink process**. Without the configuration below or an explicitly
  enabled CAS `grpc` store, NativeLink does not advertise zstd, rejects
  `compressed-blobs/zstd` requests with `InvalidArgument`, and keeps its own
  outgoing transfers on the identity path, so clients fall back to identity
  transfers.

## Enabling it [#enabling-it]

Set `remote_cache_compression: true` on the capabilities service for the
instance. The capabilities service advertises zstd to clients, and the
ByteStream and CAS services accept and serve compressed payloads for that
instance:

```json5
servers: [
  {
    // ...
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      capabilities: [
        {
          instance_name: "main",
          remote_cache_compression: true,
        },
      ],
      // ...
    },
  },
],
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CapabilitiesConfig" />

Then build with:

```sh
bazel build //... \
  --remote_cache=grpc://your-nativelink:50051 \
  --remote_cache_compression
```

Clients only compress when the server advertises it, so enabling the option is
safe with a mixed fleet of clients: anything that doesn't request
`compressed-blobs/zstd` keeps using identity transfers.

See [Servers and services](/configuration/servers-and-services) for where the
capabilities service sits in the rest of the config.

### NativeLink-to-NativeLink transfers [#nativelink-to-nativelink-transfers]

A `grpc` store with `store_type: "cas"` can compress its own uploads and
full-blob downloads to the upstream with `experimental_remote_cache_compression`.
Blobs of 64 KiB and above go as `compressed-blobs/zstd`; smaller blobs and
ranged reads stay on the identity path. The upstream instance must have
`remote_cache_compression` on its capabilities service, or compressed
requests fail with `InvalidArgument`.

Enabling zstd on any capabilities instance or any CAS `grpc` store expresses
process-wide intent. At config load, NativeLink then turns it on for every
other CAS `grpc` store whose `experimental_remote_cache_compression` is
omitted, so worker-to-CAS and proxy-to-upstream transfers pick it up without
the setting being repeated on each store. Declare the intent once and leave
the rest inherited:

```json5
servers: [
  {
    // ...
    services: {
      capabilities: [
        { instance_name: "main", remote_cache_compression: true },
      ],
    },
  },
],
stores: [
  {
    name: "CAS_PRIMARY",
    grpc: {
      instance_name: "main",
      endpoints: [{ address: "grpc://cas-primary:50051" }],
      store_type: "cas",
      // Inherits zstd from the process-wide intent.
    },
  },
  {
    name: "CAS_MIRROR",
    grpc: {
      instance_name: "main",
      endpoints: [{ address: "grpc://cas-mirror:50051" }],
      store_type: "cas",
      // Inherits zstd from the process-wide intent.
    },
  },
],
```

To keep one link uncompressed (an older NativeLink, or a third-party cache
without `compressed-blobs` support), set `experimental_remote_cache_compression:
false` on that store. NativeLink logs a startup warning for a CAS `grpc` store
that opts out while zstd is enabled elsewhere, and a warning for
`experimental_remote_cache_compression: true` on an `ac` store, where the
setting has no effect: `ac` stores are never eligible and never inherit.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig::apply_zstd_grpc_store_defaults" />

Compressed uploads from a `grpc` store do not resume mid-stream: a transport
failure part-way through surfaces to the caller, and outer retries re-send the
whole blob.

## Semantics worth knowing [#semantics-worth-knowing]

- Resource names and digests always refer to the **uncompressed** blob; only
  the bytes on the wire are compressed. ByteStream compressed writes report
  `committed_size` in compressed wire bytes, as the REAPI extension specifies.
- Uploads are verified: NativeLink hashes the *decoded* bytes and rejects a
  blob whose decompressed content doesn't match the digest or size it was
  addressed with, bounding decompression by the expected size.
- Batch reads fall back to identity when the zstd encoding would not be
  smaller, so incompressible blobs don't pay the CPU and size overhead.

## When it helps, and when it doesn't [#when-it-helps-and-when-it-doesnt]

Compression pays off when clients reach the cache across a real network and the
artifacts are compressible: uncompressed archives, binaries with debug info,
and text-heavy outputs commonly shrink 2-10x on the wire. It does little on
same-rack links (the zstd CPU cost buys nothing when the wire isn't the
bottleneck) and nothing for already-compressed artifacts.

Wire compression doesn't change what's stored: the CAS holds raw bytes. To also
shrink bytes at rest, wrap the backing store in a `compression` store; see
[Compose stores](/how-to/stores/compose-stores). The two compose, since one
covers the wire and the other covers storage.

## Steps [#steps]

<Steps>

<li>

**Add `remote_cache_compression: true`** to the capabilities service for the
instance, and restart NativeLink.

</li>

<li>

**Run a build with `--remote_cache_compression`** against that instance.

</li>

<li>

**Compare transfer bytes** against the same build without the flag. If they
don't move, your artifacts weren't compressible.

</li>

</Steps>

<VerifyBlock>

- The build succeeds with `--remote_cache_compression` and produces identical
  outputs to a build without it.
- Bazel's reported remote cache transfer bytes drop for compressible targets.
- Clients that don't pass the flag keep working against the same instance,
  unchanged.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="The client falls back to uncompressed transfers" id="the-client-falls-back-to-uncompressed-transfers">

NativeLink only advertises zstd when `remote_cache_compression` is true on the
capabilities service **for that instance name**. An instance-name mismatch
between the capabilities entry and the CAS or ByteStream entries is the usual
cause.

</Accordion>

<Accordion title="Transfer bytes barely moved" id="transfer-bytes-barely-moved">

Your artifacts are already compressed, or the wire wasn't the bottleneck. Both
are real answers; on a same-rack link there is nothing here to win.

</Accordion>

<Accordion title="Is it safe to enable with a mixed client fleet?" id="is-it-safe-to-enable-with-a-mixed-client-fleet">

Yes. Clients only compress when the server advertises support, so anything that
doesn't request `compressed-blobs/zstd` keeps using identity transfers against
the same instance.

</Accordion>

<Accordion title="Can I combine this with the dedup store?" id="can-i-combine-this-with-the-dedup-store">

Yes, but order matters at the store level: deduplication must operate on raw
content, because identical chunks stop being identical after compression, so a `dedup`
store fed compressed bytes finds no duplicates. Wire compression is fine either
way; it's undone at the gRPC boundary before any store sees the bytes.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/chunking-and-dedup" title="Content-defined chunking">
  The other transfer-bytes lever: re-send only the parts of a large blob that
  changed.
</NextStep>

<NextStep href="/how-to/stores/compose-stores" kind="aside" title="Compose stores">
  Shrinking bytes at rest instead of on the wire, with the `compression` store
  wrapper.
</NextStep>

# https://docs.nativelink.com/how-to/stores/chunking-and-dedup

**Content-defined chunking**: Cut remote cache transfer bytes by 80-90% for incrementally changing artifacts with the REAPI SplitBlob/SpliceBlob extension and Bazel's --experimental_remote_cache_chunking.

**Who this is for:** anyone re-transferring large artifacts that change a
little at a time: container layers, linked binaries, archives. **What you'll
have at the end:** clients uploading and downloading only the chunks that
changed. **Time:** twenty minutes.

<Prerequisites>
  A config file you can edit (see [Configuration](/configuration)), and Bazel 9.1.1+ or 8.7.0+.
</Prerequisites>

When a large build output changes slightly (a relinked binary, a container
layer with one file modified), its digest changes, and a conventional remote
cache re-transfers the whole blob. Content-defined chunking (CDC) splits blobs
into chunks at content-derived boundaries, so clients upload and download only
the chunks that actually changed. In our measurements a one-file change to a
16.8 MB tar layer re-uploaded 2.1 MB (87.5% less), and a one-source-file change
to a linked binary re-uploaded 80.8% less.

NativeLink implements the server side of the
[REAPI blob split/splice extension](https://github.com/bazelbuild/remote-apis/pull/282):
`SpliceBlob` re-assembles chunked uploads (verifying the digest before
committing anything), and `SplitBlob` serves chunk layouts for downloads,
chunking blobs on demand with FastCDC 2020 when they were uploaded whole, which
is what makes chunked downloads work for outputs produced by remote execution
workers.

## Requirements [#requirements]

- **Bazel 9.1.1+ or 8.7.0+** on the client, with
  `--experimental_remote_cache_chunking`. Avoid 9.1.0: it has a client bug that
  corrupts outputs when the chunking flag is combined with `--disk_cache`
  (fixed in 9.1.1).
- Chunking is **optional and off by default**. Without the configuration below,
  NativeLink behaves exactly as before and does not advertise chunking support,
  so clients fall back to regular transfers.

## Enabling it [#enabling-it]

Add an `experimental_chunking` block to the CAS service and give it a small
store for chunk layouts. The index store must not verify content digests and
must not be the CAS store itself (naming the same store for both is rejected
at startup):

```json5
stores: [
  {
    name: "CAS_MAIN_STORE",
    // ... your existing CAS store ...
  },
  {
    // Blob-to-chunks layouts: roughly 80-140 bytes per chunk.
    name: "CHUNK_INDEX_STORE",
    filesystem: {
      content_path: "/tmp/nativelink/data/content_path-chunk-index",
      temp_path: "/tmp/nativelink/data/tmp_path-chunk-index",
      eviction_policy: { max_bytes: 100000000 },
    },
  },
],
servers: [
  {
    // ...
    services: {
      cas: [
        {
          cas_store: "CAS_MAIN_STORE",
          experimental_chunking: {
            index_store: "CHUNK_INDEX_STORE",
            // Optional; the REAPI-recommended default. Must be between
            // 1 KiB and 1 MiB. Clients derive min and max chunk sizes
            // from it (a quarter and four times this value).
            avg_chunk_size_bytes: 524288,
            // Optional; blobs producing more chunks than this are served
            // without chunking (~25 GiB at the default average).
            max_chunk_count: 50000,
          },
        },
      ],
      // The capabilities service advertises chunking support; clients
      // only use it when advertised.
      capabilities: [{}],
      // ...
    },
  },
],
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasChunkingConfig" />

A complete runnable example lives at
<SourceLink file="nativelink-config/examples/chunking_cas.json5" />.
For instances whose `cas_store` is a grpc proxy store, omit `index_store`: the
chunking RPCs are forwarded to the backend, which owns the layouts.

Then build with:

```sh
bazel build //... \
  --remote_cache=grpc://your-nativelink:50051 \
  --experimental_remote_cache_chunking
```

## When it helps, and when it doesn't [#when-it-helps-and-when-it-doesnt]

Chunking pays off when clients reach the cache across a real network (WAN,
metered links, cross-region) and artifacts change incrementally: uncompressed
archives, linked binaries, and container layers typically save 80 to 90% of
transfer bytes per change. It does little on same-rack links (saved bytes only
save time when the wire is the bottleneck) and little for compressed
artifacts, where everything after the first changed byte re-transfers. Small
blobs, below the maximum chunk size (4x the average), are not worth chunking
and clients leave them alone, so hot small-object traffic is unaffected.

The server verifies every spliced blob's digest before committing it and
materializes the full blob, so non-chunking clients and every existing read
path see ordinary blobs.

<SourceLink file="nativelink-service/src/cas_server.rs" symbol="CasServer::inner_splice_blob" /> Storage grows by roughly the chunk bytes for
chunk-eligible blobs; pairing the CAS with a `dedup` or `compression` store
composes normally; see [Compose stores](/how-to/stores/compose-stores).

## Steps [#steps]

<Steps>

<li>

**Add a chunk index store**: a small `filesystem` store of its own, not the
CAS store, and not wrapped in `verify`.

</li>

<li>

**Add `experimental_chunking`** to the CAS service pointing at it, and make
sure the instance has a `capabilities` entry so support is advertised.

</li>

<li>

**Build with `--experimental_remote_cache_chunking`**, change one source file,
and build again.

</li>

<li>

**Compare the re-upload bytes** between the two builds. That difference is the
whole point.

</li>

</Steps>

<VerifyBlock>

- The chunk index store fills with small entries while a build runs, and stays
  far smaller than the CAS.
- A one-file change to a large artifact re-uploads a fraction of the artifact,
  not all of it.
- Outputs are byte-identical to a build without the flag.
- Clients that don't pass the flag keep working against the same instance.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="My outputs corrupt with --disk_cache" id="my-outputs-corrupt-with---disk_cache">

Bazel 9.1.0 has a client bug that corrupts outputs when the chunking flag is
combined with `--disk_cache`. Use 9.1.1+ or 8.7.0+.

</Accordion>

<Accordion title="Clients never use chunking" id="clients-never-use-chunking">

Support is advertised through the capabilities service, so the instance needs a
`capabilities` entry. Check the client version too: the flag exists only from
Bazel 9.1.1 / 8.7.0.

</Accordion>

<Accordion title="Nothing gets chunked" id="nothing-gets-chunked">

Clients do not chunk blobs below the maximum chunk size, 4x
`avg_chunk_size_bytes`. At the 512 KiB default that's everything under 2 MiB.
Blobs producing more chunks than `max_chunk_count` are also served whole:
`SplitBlob` answers `NOT_FOUND` and the client downloads normally.

</Accordion>

<Accordion title="Errors from the chunk index store" id="errors-from-the-chunk-index-store">

It must not be the CAS store, and must not sit under `verify`, because the layouts it
holds are not content-addressed, so digest verification rejects them. Give it
its own plain `filesystem` store.

</Accordion>

<Accordion title="Do I need to change anything for existing clients?" id="do-i-need-to-change-anything-for-existing-clients">

No. Chunking is opt-in and advertised through the capabilities service, so
clients that don't request it (or predate it) keep using regular transfers
against the same CAS.

</Accordion>

<Accordion title="Chunking or wire compression?" id="chunking-or-wire-compression">

They solve different transfer problems. Chunking re-transfers only the changed
parts of large, incrementally-changing blobs;
[wire compression](/how-to/stores/compression) shrinks compressible blobs
wholesale. Compressed artifacts chunk poorly (everything after the first
changed byte re-transfers), so lean on chunking for archives, binaries, and
layers, and on compression for text-heavy outputs.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compression" title="Remote cache compression">
  The complementary lever: zstd on the wire for artifacts that compress well.
</NextStep>

<NextStep href="/how-to/stores/compose-stores" kind="aside" title="Compose stores">
  The store-level `dedup` wrapper: deduplication at rest rather than on the
  wire.
</NextStep>

# https://docs.nativelink.com/how-to/run-multiple-workers

**Run multiple workers**: Add workers to a running NativeLink: how they share the CAS, how the scheduler matches them, and the two things about the shipped compose example that will mislead you.

**Who this is for:** anyone with remote execution working against one worker
who now wants several. **What you'll have at the end:** three workers pulling
from one queue, and a clear idea of how to make it thirty. **Time:** forty
minutes.

<Prerequisites>
  A working scheduler and at least one worker. See [Remote execution](/remote-execution).
</Prerequisites>

One worker is a demonstration. The reason to add more is that the scheduler
already assumes them: it holds a queue, and every worker that connects to the
worker API announces what it can run and starts taking work off that queue.
Nothing about adding the second worker is different from adding the tenth.

What does need thought is the part that goes wrong most often: workers do not
share a disk. They share the CAS, and they share it over gRPC.

## The topology [#the-topology]

Three roles, three kinds of process, on one network:

<Mermaid>
{`flowchart LR
  C["Bazel client"] -->|":50051 CAS + AC"| CAS["CAS server"]
  C -->|":50052 execution"| S["Scheduler"]
  S -->|"grpc store"| CAS
  W1["worker-1"] -->|":50061 worker API"| S
  W2["worker-2"] -->|":50061 worker API"| S
  W3["worker-3"] -->|":50061 worker API"| S
  W1 -->|"grpc store"| CAS
  W2 -->|"grpc store"| CAS
  W3 -->|"grpc store"| CAS`}
</Mermaid>

The CAS server owns the bytes. The scheduler owns the queue. Workers own
nothing durable. They are the disposable tier, which is exactly why they are
the tier you scale.

Each worker keeps a local filesystem cache in front of the shared CAS so that
an input it has fetched once does not cross the network again. That is a
`fast_slow` store whose `slow` half is a `grpc` store pointing at the CAS
server. [Compose stores](/how-to/stores/compose-stores) covers the wrapper in
general; this is its most load-bearing use.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

## What a worker config looks like [#what-a-worker-config-looks-like]

```json5
{
  stores: [
    {
      name: "GRPC_CAS",
      grpc: {
        instance_name: "",
        store_type: "cas",
        endpoints: [
          { address: "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051" },
        ],
      },
    },
    {
      name: "GRPC_AC",
      grpc: {
        instance_name: "",
        store_type: "ac",
        endpoints: [
          { address: "grpc://${CAS_ENDPOINT:-127.0.0.1}:50051" },
        ],
      },
    },
    {
      name: "WORKER_FAST_SLOW_STORE",
      fast_slow: {
        fast: {
          filesystem: {
            content_path: "/root/.cache/nativelink/content_path-cas",
            temp_path: "/root/.cache/nativelink/tmp_path-cas",
            eviction_policy: { max_bytes: 10000000000 },
          },
        },
        slow: { ref_store: { name: "GRPC_CAS" } },
      },
    },
  ],
  workers: [
    {
      local: {
        worker_api_endpoint: {
          uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
        },
        cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
        upload_action_result: { ac_store: "GRPC_AC" },
        work_directory: "/root/.cache/nativelink/work",
        platform_properties: {
          cpu_count: { query_cmd: "nproc" },
          OSFamily: { values: [""] },
          "container-image": { values: [""] },
          ISA: { values: ["x86-64"] },
        },
      },
    },
  ],
  servers: [],
}
```

`servers: []` is not an oversight. A worker listens on nothing; it dials out to
the scheduler's worker API and to the CAS. That is why the worker tier needs no
ingress, no certificates and no load balancer, and why it is safe to run many
of them on machines that are otherwise unreachable.

Every worker can run this identical file. The only things that must differ per
worker are the paths, and only because two processes on the same host would
otherwise collide.

<Callout type="warn" title="`work_directory` is deleted at startup">
  A worker removes its `work_directory` and recreates it every time it starts.
  Point it somewhere that exists only for this purpose, never at a directory
  that holds anything you want to keep, and never at a path two workers on the
  same host both use.
</Callout>

## How the scheduler picks a worker [#how-the-scheduler-picks-a-worker]

The scheduler will only send an action to a worker whose advertised
`platform_properties` satisfy the action's platform requirements, and it will
only consider property keys listed in its own
`supported_platform_properties`:

```json5
schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: {
        cpu_count: "minimum",
        OSFamily: "priority",
        "container-image": "priority",
        ISA: "exact",
      },
    },
  },
],
```

The value is the matching rule. `exact` means the strings must be equal.
`minimum` means the worker's number must be at least what the action asks for,
which is what makes `cpu_count` usable as a sizing hint. `priority` does not
restrict matching at all: the worker only has to advertise the key, and the
value is passed to the worker as information. Despite the name, the scheduler
does not currently use it to prefer one worker over another. `ignore` lets an
action request a key without requiring the worker to have it.

A key an action requests that the scheduler does not list is a key the
scheduler cannot match on: matching that action fails with `Unknown platform
property`, and the usual symptom is an action that queues forever while idle
workers sit next to it. A worker that advertises a key the scheduler does not
list is rejected when it connects.

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="PropertyType" />

[Platform properties](/remote-execution/platform-properties) is the page for
that contract; this page only needs you to know that every worker in a pool
should advertise the same keys.

## Scaling the pool [#scaling-the-pool]

Workers are stateless, so growing the pool is adding processes. Three things
scale with them and are worth watching before you double the count.

The CAS server is now serving every input to every worker. It is the first
thing to saturate, and [Tuning](/operate/tuning) has the storage levers for it.
Give each worker a large enough local `fast` tier and most reads never reach
it.

The scheduler holds one queue and matches against every connected worker. It is
CPU-cheap but not free, and it is a single point of failure until you give it a
shared store.

Disk on each worker host is consumed by the `fast` tier plus the work
directory of every action in flight. `max_bytes` bounds the first and nothing
bounds the second, so leave headroom.

## Steps [#steps]

<Steps>

<li>

**Split the processes** if you have not already: one CAS server, one
scheduler, workers separately. A single all-in-one config cannot become a
multi-worker deployment without this.

</li>

<li>

**Put the scheduler's worker API on its own listener**, conventionally
`:50061`, separate from the client-facing `:50052`. It is a backend API with a
different trust level.

</li>

<li>

**Write one worker config** with a `fast_slow` store over a `grpc` store, and
confirm it works with a single worker before adding any others.

</li>

<li>

**Start two more workers** from the same config, giving each its own
`content_path`, `temp_path` and `work_directory` if they share a host.

</li>

<li>

**Run a build with high parallelism** (`--jobs=50` or so) and confirm the
actions land across all three.

</li>

</Steps>

<VerifyBlock>

- Each worker logs `Worker registered with scheduler` at startup, and the
  scheduler's `worker.connected.count` metric reads three rather than one.
- A build with high `--jobs` shows actions executing on more than one worker.
- Stopping one worker mid-build does not fail the build. Its actions are
  retried elsewhere.
- Each worker's local `content_path` grows and then plateaus at its
  `max_bytes`, rather than growing without bound.

</VerifyBlock>

## The shipped example, and where its README misleads [#the-shipped-example-and-where-its-readme-misleads]

`deployment-examples/docker-compose/` contains a working three-worker
deployment: `docker-compose-multi-worker.yml` with
`cas-server-multi-worker.json5`, `scheduler-multi-worker.json5` and
`worker-shared-cas.json5`. Start there. Two things in and around it will send
you the wrong way.

**The `MULTI_WORKER.md` advice to share a CAS volume is not what the shipped
config does.** That document tells you all workers must mount the same
`cas-data` volume and share a `content_path`. The worker config next to it does
not do that. It uses a worker-local filesystem tier over a `grpc` store, which
is the correct design, and it writes to a path the shared volume does not even
cover. Workers share the CAS by talking to the CAS server. A shared filesystem
between workers is neither required nor sufficient, and on anything other than
one host it is not available.

You will also see this line in every worker's log at startup (the quoted name
is the worker's `name`, which the shipped config leaves empty):

```
Starting worker ''. IMPORTANT: If running multiple workers, all workers must
share the same CAS storage path to avoid 'Object not found' errors.
```

It is unconditional (every worker prints it, including a single worker) and
it is describing the same stale advice. If your workers reach one CAS server
over gRPC, you have satisfied the requirement it is gesturing at.

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="new_local_worker" />

**The healthchecks in the compose file probe a path that isn't served.** Both
services healthcheck `http://localhost:PORT/status`, but `/status` only exists
when that server's `services` block includes `health`. Neither the CAS config
nor the scheduler's client-facing listener has one. Add `health: {}` to the
services on the port you're probing, or drop the healthcheck.

<Callout type="info" title="`--scale` needs a differently shaped compose file">
  `docker compose up -d --scale worker=5` only works against a service literally
  named `worker`. The shipped file declares `worker-1`, `worker-2` and
  `worker-3` as three separate services, so scaling it means collapsing them
  into one service without per-worker volume mounts, at which point each
  replica needs its paths derived from something unique rather than hardcoded.
</Callout>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Workers connect but never receive actions" id="workers-connect-but-never-receive-actions">

Almost always the platform properties. Compare the keys the worker advertises
against `supported_platform_properties` on the scheduler and against what the
client requests: a key the scheduler doesn't list can't be matched on, and a
key the worker doesn't advertise excludes it from anything requiring that key.
Check the scheduler log for the queue depth: actions queuing while workers
idle is the signature.

</Accordion>

<Accordion title="`Object not found in either fast or slow store`" id="object-not-found-in-either-fast-or-slow-store">

The worker cannot reach the blob through its `slow` store. Check that the
`grpc` store's address resolves from inside the worker (a container name that
works on the host network usually doesn't) and that the CAS server is the
same one the client uploaded to. This error also appears when the scheduler and
the workers point at different CAS servers.

</Accordion>

<Accordion title="Two workers on one host interfere with each other" id="two-workers-on-one-host-interfere-with-each-other">

They are sharing `work_directory`, `content_path` or `temp_path`. The work
directory is the worst of the three, because it is deleted and recreated at
startup: the second worker to start wipes the first one's in-flight actions.
Give each process its own paths.

</Accordion>

<Accordion title="Adding workers made builds slower" id="adding-workers-made-builds-slower">

You have moved the bottleneck to the CAS server, which is now serving inputs to
every worker. Confirm by watching its network and disk while a build runs.
Raising each worker's `fast` tier `max_bytes` cuts the read traffic; past that
it is a CAS scaling problem.

</Accordion>

<Accordion title="Workers keep dropping and reconnecting" id="workers-keep-dropping-and-reconnecting">

Check `worker_api_endpoint.timeout` on the worker against `worker_timeout_s`
on the scheduler, and any proxy or load balancer between the two. The worker
sends a `KeepAlive` RPC every `timeout / 2` (default timeout 5 seconds), and
the scheduler evicts a worker it has not heard from within `worker_timeout_s`
(default 5 seconds), so a hop that delays or drops those RPCs looks like a
flapping worker. HTTP/2 keepalive pings on this listener come from
`advanced_http.http2_keep_alive_interval` on the scheduler's worker API
`servers` entry. The worker API is a long-lived streaming connection, not a
series of requests.

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="start_keep_alive" />

</Accordion>

<Accordion title="Can workers run on different architectures?" id="can-workers-run-on-different-architectures">

Yes, and this is what `ISA` and `OSFamily` are for. Advertise the real values
on each worker and set `ISA` to `exact` on the scheduler, so an action built
for `x86-64` never lands on an `aarch64` worker. A pool of mixed architectures
with everything advertising the same values is a way to get silent, confusing
build failures.

</Accordion>

</Accordions>

<NextStep href="/operate/tuning" title="Tuning">
  The worker and storage levers that matter once the pool is large enough to
  push something to its limit.
</NextStep>

<NextStep href="/remote-execution/platform-properties" kind="aside" title="Platform properties">
  The matching contract in full, and the single most common reason an action never
  gets scheduled.
</NextStep>

# https://docs.nativelink.com/how-to/tls-and-auth

**TLS and authentication**: Put TLS on the listeners clients reach, use mutual TLS as the access control, and understand exactly what NativeLink does and does not authenticate.

**Who this is for:** anyone about to expose NativeLink beyond a single trusted
network. **What you'll have at the end:** encrypted listeners, client
certificates as the gate, and an accurate picture of what is still unguarded.
**Time:** an hour, most of it spent on certificates.

<Prerequisites>
  A running deployment you can restart. See [Production configuration](/operate/production-config).
</Prerequisites>

Start with the honest version, because it changes what you build.

NativeLink has no user accounts, no password, no token check and no
authorization rules. Nothing in the binary decides whether a
particular caller is allowed to read a particular blob. The access control you
get is **mutual TLS** (the listener refuses connections from clients that
cannot present a certificate signed by a CA you name) plus whatever your
network already does. Everything past the handshake is trusted equally.

That is a coherent security model, and it is the one to build. It is not the
model you would get from a system with a user table, so it pays to be explicit
about it before you plan around a feature that does not exist.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="TlsConfig" />

## The three listeners have different trust levels [#the-three-listeners-have-different-trust-levels]

A production deployment exposes three surfaces, and they do not want the same
treatment.

The **CAS and AC** listener, conventionally `:50051`, is what every developer's
Bazel talks to. It is the widest surface and the one that needs TLS.

The **execution** listener, conventionally `:50052`, carries the same client
traffic for remote execution. Treat it identically to the CAS.

The **worker API**, conventionally `:50061`, is not a client API. Workers dial
it, and a caller who reaches it can register as a worker and be handed your
builds to execute. It belongs on a private interface or inside a private
network, and no amount of TLS makes it safe to expose publicly. Bind it to an
internal address, and keep it on a separate `servers` entry from anything a
client touches so that the two cannot be confused.
[Run multiple workers](/how-to/run-multiple-workers) covers why that listener
exists and who dials it.

[Servers and services](/configuration/servers-and-services) covers how those
listeners are declared; this page is about what to put on them.

## Turn on TLS [#turn-on-tls]

Every entry in `servers` may carry a `tls` block on its `listener.http`:

```json5
servers: [
  {
    name: "public_cas",
    listener: {
      http: {
        socket_address: "0.0.0.0:50051",
        tls: {
          cert_file: "/etc/nativelink/tls/server.crt",
          key_file: "/etc/nativelink/tls/server.key",
        },
      },
    },
    services: {
      cas: [{ cas_store: "CAS_MAIN_STORE" }],
      ac: [{ ac_store: "AC_MAIN_STORE" }],
      capabilities: [{}],
      bytestream: [{ cas_store: "CAS_MAIN_STORE" }],
    },
  },
],
```

Both paths are PEM files read at startup. The certificate file may hold a full
chain. The key may be PKCS8, SEC1 or PKCS1 (all three are accepted), but it
must contain **exactly one** key, and a file with two will fail startup rather
than pick one.

Once TLS is on, the listener negotiates ALPN `h2`, and clients must address it
as `grpcs://` rather than `grpc://`. That scheme change is the single most
common reason a first TLS rollout fails: a `grpc://` client sends plaintext
into the handshake, the server logs `Failed to accept tls stream` and drops the
connection, and the client sees a connection error that says nothing about
TLS.

<SourceLink file="src/bin/nativelink.rs" symbol="TlsAcceptor" />

<Callout type="warn" title="The example certificates are examples">
  `deployment-examples/docker-compose/` ships
  `example-do-not-use-in-prod-key.pem` and
  `example-do-not-use-in-prod-rootca.crt`. The names are the documentation.
  Their private key is in the public repository, so anything they protect is
  protected from nobody.
</Callout>

## Make the client prove who it is [#make-the-client-prove-who-it-is]

Adding `client_ca_file` changes the listener from encrypted to gated:

```json5
tls: {
  cert_file: "/etc/nativelink/tls/server.crt",
  key_file: "/etc/nativelink/tls/server.key",
  client_ca_file: "/etc/nativelink/tls/clients-ca.crt",
  client_crl_file: "/etc/nativelink/tls/clients.crl",
},
```

Now every connecting client must present a certificate signed by that CA. This
is the access control. No optional or opportunistic mode exists: set
`client_ca_file` and client certificates are required, leave it unset and no
client certificate is ever examined.

`client_crl_file` is a PEM certificate revocation list, and it is how a leaked
client key stops working before its expiry. It is worth setting up on day one
even if the list is empty, because the moment you need it is the moment you do
not want to be learning how it is wired.

What you get from this is coarse: a client either holds a valid certificate or
it does not. NativeLink does not read the subject, does not map it to an
identity, and does not vary permissions by it. Issuing per-team or per-developer
certificates is still worth doing, because it lets you revoke one without
reissuing all of them, but the difference is visible to your CA, not to
NativeLink.

## Point clients at the new scheme [#point-clients-at-the-new-scheme]

Bazel needs the CA that signed the server certificate, and its own certificate
if you turned on mutual TLS:

```sh
build --remote_cache=grpcs://cache.example.com:50051
build --remote_executor=grpcs://cache.example.com:50052
build --tls_certificate=/etc/nativelink/tls/ca.crt
build --tls_client_certificate=/etc/nativelink/tls/client.crt
build --tls_client_key=/etc/nativelink/tls/client.key
```

NativeLink is also a client of itself: a worker dials the scheduler, and a
`grpc` store dials an upstream CAS. Those connections carry their own
`tls_config`, with a different shape from the listener's:

```json5
{
  name: "GRPC_CAS",
  grpc: {
    instance_name: "",
    store_type: "cas",
    endpoints: [
      {
        address: "grpcs://cas.internal.example.com:50051",
        tls_config: {
          ca_file: "/etc/nativelink/tls/ca.crt",
          cert_file: "/etc/nativelink/tls/client.crt",
          key_file: "/etc/nativelink/tls/client.key",
        },
      },
    ],
  },
},
```

`ca_file` validates the remote. `cert_file` and `key_file` are what the remote
sees when it demands a client certificate, so they are only needed against a
listener with `client_ca_file` set (setting one without the other is a startup
error). Setting `use_native_roots: true` instead of `ca_file` uses the
operating system's trust store, which is what you want against a public CA and
not what you want against your own; when it is set, `ca_file` is ignored with
a warning. The scheme and the block go together: a `grpcs://` address with no
`tls_config` is rejected at startup, and so is a `tls_config` on a `grpc://`
address.

<SourceLink file="nativelink-util/src/tls_utils.rs" symbol="endpoint_from" />

The worker's `worker_api_endpoint` takes the same block:

```json5
worker_api_endpoint: {
  uri: "grpcs://scheduler.internal.example.com:50061",
  tls_config: {
    ca_file: "/etc/nativelink/tls/ca.crt",
    cert_file: "/etc/nativelink/tls/worker.crt",
    key_file: "/etc/nativelink/tls/worker.key",
  },
},
```

## What `experimental_identity_header` actually does [#what-experimental_identity_header-actually-does]

You will find this on `ServerConfig`, and its name invites a wrong conclusion:

```json5
experimental_identity_header: {
  header_name: "x-identity",
  required: true,
},
```

Two things are true about it. The first is that it is not authentication: the
value it gates on is caller-supplied and never verified against anything, so any
client can claim any identity. The second is that it does not read the header
you name. `required` is passed to the telemetry layer, which looks for an
OpenTelemetry **Baggage** entry named `enduser.id`, and rejects the request with
`FAILED_PRECONDITION` when that entry is absent:

```
NativeLink instance configured to require this OpenTelemetry Baggage header:

    `Baggage: enduser.id=YOUR_IDENTITY`
```

`header_name` is not read anywhere in the codebase.

`required: true` is useful for exactly one thing: making sure your traces and
logs can attribute a request to a person or a CI job, by refusing traffic that
would be unattributable. That is a real operational need. Just do not put it in
the column marked access control.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="IdentityHeaderSpec" />

## Credentials to an upstream service [#credentials-to-an-upstream-service]

When NativeLink proxies to something that does want a token (a hosted REAPI
cache, a gateway in front of one), a `grpc` store can attach headers:

```json5
grpc: {
  instance_name: "",
  store_type: "cas",
  endpoints: [
    {
      address: "grpcs://upstream.example.com:443",
      tls_config: { use_native_roots: true },
    },
  ],
  headers: { authorization: "Bearer my-static-token" },
},
```

`headers` are static and sent on every request. Note that, unlike most string
fields in the config, header values are not run through `${VAR}` expansion:
the value in the file is sent verbatim, so a secret placed here lives in the
file and the file needs to be protected accordingly. `forward_headers` takes a
list of header names to copy from the incoming client request instead, which is
how a client's own credential reaches the upstream without NativeLink holding
one. Forwarding is incompatible with `experimental_read_batching`, and a store
that configures both is rejected at startup.

<SourceLink file="nativelink-config/src/stores.rs" symbol="GrpcSpec" />

## Steps [#steps]

<Steps>

<li>

**Decide the boundary.** Which listeners are reachable from where. If the
worker API is reachable from anywhere a developer is, stop and fix that first.
It is a bigger hole than plaintext.

</li>

<li>

**Issue certificates** from a CA you control: one server certificate per
listener hostname, one client certificate per team or per CI system. Keep the
CA key somewhere the build system cannot read.

</li>

<li>

**Add `tls` to the client-facing listeners** with `cert_file` and `key_file`
only. Restart, and confirm a `grpcs://` client works before adding anything
else.

</li>

<li>

**Add `client_ca_file` and `client_crl_file`** to those same listeners, and
distribute client certificates. Every client breaks at this step until it has
one, so do it deliberately rather than at the end of the day.

</li>

<li>

**Update the internal hops** (worker to scheduler, `grpc` stores to upstream)
to `grpcs://` with a matching `tls_config`.

</li>

</Steps>

<VerifyBlock>

- A build against `grpcs://` succeeds and populates the cache.
- The same build against `grpc://` on the same port fails rather than silently
  falling back.
- With `client_ca_file` set, a client with no certificate is rejected at the
  handshake, not after.
- Revoking a client certificate and adding it to the CRL locks that client out
  after a restart.
- `openssl s_client -connect host:50051 -alpn h2` shows your chain and
  negotiates `h2`.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="`InvalidArgument: Expected 1 key in file`" id="invalidargument-expected-1-key-in-file">

The key file holds more than one key (a file with none fails earlier, with
`Could not extract key(s) from file`). A concatenated bundle is the usual cause.
Split it, and leave the chain in `cert_file` where multiple PEM blocks are fine.

</Accordion>

<Accordion title="Clients get connection errors and the server logs `Failed to accept tls stream`" id="clients-hang-instead-of-failing">

Almost always `grpc://` against a TLS listener. The client sends plaintext HTTP/2
into a socket waiting for a handshake, the handshake fails, and the server drops
the connection. Check the scheme everywhere: Bazel flags, `grpc` store
addresses, `worker_api_endpoint.uri`, and any healthcheck.

</Accordion>

<Accordion title="The client rejects the server certificate" id="certificate-signature-verify-failed-from-a-client">

The client does not trust the CA that signed the server certificate. Point it at
the right `ca_file`, or `--tls_certificate` for Bazel. When the client is
NativeLink itself (a `grpc` store or a worker) the error reads `invalid peer
certificate: UnknownIssuer`. If the server certificate was issued by an
intermediate, `cert_file` must contain the full chain, leaf first, because
clients will not fetch a missing intermediate for you.

</Accordion>

<Accordion title="`peer sent no certificates` after adding `client_ca_file`" id="peer-did-not-return-a-certificate-after-adding-client_ca_file">

The listener now requires client certificates and this client has none. The
server logs it as `Failed to accept tls stream` with that reason, and the client
sees the handshake aborted. That includes clients you may have forgotten are
clients: workers dialing the scheduler, `grpc` stores, and anything probing a
health endpoint.

</Accordion>

<Accordion title="Certificates renewed but the old ones are still in use" id="certificates-renewed-but-the-old-ones-are-still-in-use">

Server certificates and a `grpc` store's client certificates are read once at
startup, and a worker re-reads its client files only when it reconnects to the
scheduler. Renewal means restarting the process. Plan for it, because a
cert-manager rotation that nothing restarts will quietly expire.

</Accordion>

<Accordion title="How do I give different teams different access?" id="how-do-i-give-different-teams-different-access">

You can't, inside NativeLink. It has no per-caller permissions. The
approaches that work are running separate instances with separate stores for
separate trust domains, or putting an authorizing proxy in front and letting
NativeLink trust everything the proxy passes through. If you take the second
route, the worker API must not be behind the same proxy.

</Accordion>

</Accordions>

<NextStep href="/operate/production-config" title="Production configuration">
  The rest of what changes between the example configs and something you run for
  other people.
</NextStep>

<NextStep href="/configuration/servers-and-services" kind="aside" title="Servers and services">
  How listeners and service blocks fit together, and why splitting them by trust
  level is the shape to start from.
</NextStep>

# https://docs.nativelink.com/how-to/upgrade-versions

**Upgrade to a newer version**: Read the changelog for the things that actually break, upgrade the three process types in the right order, and know what happens to the cache you already have.

**Who this is for:** anyone running a pinned version who wants a newer one.
**What you'll have at the end:** a rehearsed upgrade, a rollback that works, and
your existing cache intact. **Time:** thirty minutes for the rehearsal, minutes
for the upgrade itself.

<Prerequisites>
  A running deployment with a pinned version.
</Prerequisites>

NativeLink releases every week or two. Most releases change nothing you depend on,
which makes it tempting to upgrade casually and equally tempting to never
upgrade at all. Both go wrong in the same way: the thing that breaks an upgrade
is almost never the code, it is the config.

Nearly every block of the config is parsed with `deny_unknown_fields`. A field
that was renamed or removed in the new version is not ignored: it is a startup
failure with a message naming the field (renamed fields sometimes keep the old
name as a serde alias, in which case both spellings parse). That is the
behaviour you want, because the alternative is a silently ignored setting and a
deployment that is quietly not doing what the file says. But it means an
upgrade is a config compatibility exercise first and a binary swap second.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig" />

<SourceLink file="CHANGELOG.md" />

## What is stable and what isn't [#what-is-stable-and-what-isnt]

The **client-facing API** is the Remote Execution API. It is a published,
versioned protocol, and Bazel talking to NativeLink over `:50051` and `:50052`
is the most stable surface here. You do not generally need to upgrade clients in
step with the server.

The **config schema** is the surface that moves. Fields get added constantly,
which is harmless; occasionally one is renamed or removed, which is not.

The **worker API** between a worker and the scheduler is internal. Nothing
promises that a worker from one release talks to a scheduler from another, so
treat scheduler and workers as a unit that upgrades together.

Stored **data** is content-addressed, which does most of the work for you: a
blob's key is its digest, so bytes written by an old version are found by a new
one. What is not automatically portable is the on-disk or in-Redis *layout* a
store uses to organize those bytes. Layout changes are rare (the filesystem
store still migrates its old single-folder layout in place at startup), and they
are one of the things to look for when you read the changelog.

<SourceLink file="nativelink-store/src/filesystem_store.rs" symbol="read_files" />

## Read the changelog for four things [#read-the-changelog-for-four-things]

`CHANGELOG.md` is generated per release, grouped into features, bug fixes,
documentation, testing and CI, miscellaneous, and bumps and version updates,
with a `compare/v1.6.4..v1.6.5` link in each release heading that shows every
commit between the two.

Skimming the whole thing is not the point. Read it for:

Renamed, removed or newly-required **config fields**: these are the ones that
stop your process from starting.

Changes to **default values**, which do not fail loudly and are the reason to
set explicitly anything you actually care about.

**Store layout or serialization** changes, which determine whether existing
cached data is still reachable.

Anything about the **worker protocol**, which decides whether you can upgrade
workers and scheduler independently.

Skip several versions and you must read every intervening release's entries, not
just the newest, because the breaking change may be three releases back.

<Callout type="info" title="The config reference is versioned">
  [The configuration reference](/reference/nativelink-config) is published per
  release. Open the page for the version you are moving to and diff it against
  the one you are on. It is the fastest way to see field-level changes without
  reading commits.
</Callout>

## Rehearse the config before you touch production [#rehearse-the-config-before-you-touch-production]

No `--validate` flag exists. The binary takes one positional argument, the
config file (plus the usual `--help` and `--version`), and parses it at
startup, which is exactly what you want, because the check is just "does the
new binary accept this file":

```sh
docker run --rm \
  -v $(pwd)/cas.json5:/config.json5 \
  ghcr.io/tracemachina/nativelink:v1.6.5 /config.json5
```

A config error fails within a second, before any port is bound, with a message
naming the offending field. If instead it starts up and begins listening, the
config parsed and you can stop the container.

Run this for every config file you have (the CAS, the scheduler, and each
distinct worker) against the version you are moving to. A parse failure here is
a five-minute fix. The same failure during a rolling restart is an outage.

## Upgrade in the right order [#upgrade-in-the-right-order]

<Steps>

<li>

**Pin, don't float.** If anything in your deployment says `:latest`, fix that
first. You cannot control an upgrade you did not choose, and you cannot roll
back to a tag that means something different now.

</li>

<li>

**Rehearse every config file** against the new image as above, and fix whatever
fails.

</li>

<li>

**Upgrade the CAS servers first.** They hold the data, they serve the most
traffic, and they are the piece other processes depend on. A filesystem store
rescans its `content_path` at startup to rebuild its index, so a restart with a
large cache is not instant. Expect a pause proportional to the number of
objects, and do not interpret it as a hang.

</li>

<li>

**Upgrade the scheduler and the workers together.** The worker API between them
is internal. With a `simple` scheduler the queue is in memory, so restarting it
drops queued actions. Clients retry, but do it when a retry storm is cheap.

</li>

<li>

**Run a real build**, then run it again. The first proves the new version works;
the second proves it can still read what the old version wrote, which is the
question that actually matters.

</li>

</Steps>

<VerifyBlock>

- Every process is running the intended tag, and you can prove it rather than
  assume it.
- A build run before the upgrade gets cache hits after it. This is the
  data-compatibility check.
- Remote execution actions complete on upgraded workers.
- Nothing in any process log mentions an unknown or deprecated config field.
- The previous image tag plus the previous config file are still somewhere you
  can reach in under a minute.

</VerifyBlock>

## Rolling back [#rolling-back]

Rollback is the same operation in reverse, with one asymmetry worth internalizing.

Config is not backward compatible in the direction you need it to be. If you
added a field that only exists in the new version, the old binary rejects that
file, because `deny_unknown_fields` cuts both ways. **Roll back the config and the
image together, as a pair.** Keeping the two versions of each config file next
to each other for a release or two costs nothing and turns a rollback into a
single decision.

Data written by the newer version stays readable by the older one in the normal
case, because content addressing does not change. If the changelog mentioned a
store layout change, that is the case where it does not, and it is also the case
where rolling back means accepting a cold cache. Reading for that in advance is
what makes the difference between a rollback and an incident.

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="`unknown field` at startup" id="unknown-field-at-startup">

Exactly the failure this design intends. The message names the field. Find it in
the new version's config reference. Usually it was renamed, occasionally it
moved under a different parent, rarely it is gone because the feature it
controlled is. Fix the file, do not try to make the parser lenient.

</Accordion>

<Accordion title="Everything starts but the cache is cold" id="everything-starts-but-the-cache-is-cold">

The new version is not finding the old data. Check first that `content_path`,
the bucket, or the Redis instance is genuinely the same one. A changed volume
mount or a changed key prefix looks exactly like a format incompatibility. If
the paths are identical, re-read the changelog for the intervening releases with
store layout in mind.

</Accordion>

<Accordion title="Workers connect but actions fail after upgrading only the scheduler" id="workers-connect-but-actions-fail-after-upgrading-only-the-scheduler">

Version skew across the worker API. Bring the workers to the same version. This
is why the two upgrade together rather than independently.

</Accordion>

<Accordion title="The CAS takes a long time to start" id="the-cas-takes-a-long-time-to-start">

A filesystem store walks `content_path` at startup to rebuild its in-memory
index, and that scales with object count. It is not stuck. If the pause is long
enough to matter operationally, that is a signal about cache size rather than
about the upgrade, and [Tuning](/operate/tuning) has the storage levers.

</Accordion>

<Accordion title="How far behind can I safely get?" id="how-far-behind-can-i-safely-get">

Being several releases behind is not itself dangerous. The risk is that the
upgrade becomes several config migrations at once with no way to tell which one
broke. If you are many versions back, step through in two or three hops and run
a build at each, rather than making one jump you cannot bisect.

</Accordion>

<Accordion title="Do clients need to upgrade too?" id="do-clients-need-to-upgrade-too">

Usually no. Bazel and NativeLink talk the Remote Execution API, which is stable
across these releases. Client upgrades matter when you want a client-side
feature, not because the server moved.

</Accordion>

</Accordions>

<NextStep href="/reference/nativelink-config" title="Configuration reference">
  The per-version field reference, which is what you diff when a release note is
  ambiguous.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  The settings worth pinning explicitly so a changed default never surprises you.
</NextStep>

# https://docs.nativelink.com/how-to/migrate-an-existing-cache

**Migrate from another REAPI cache**: Put NativeLink in front of your existing cache as a read-only diode, let real builds warm it, and cut over when the hit rate says you can.

**Who this is for:** anyone running Buildbarn, Buildfarm, bazel-remote or a
hosted REAPI cache who wants to move to NativeLink without a flag day. **What
you'll have at the end:** NativeLink serving your builds, backed by the old
cache, with a cutover you choose rather than one you're forced into. **Time:**
an hour to set up; days to weeks of running both, at your pace.

<Prerequisites>
  A working NativeLink CAS. See [Getting started](/getting-started).
</Prerequisites>

The instinct is to copy the data across. Don't. A remote cache is not a
database: every entry in it is derived, reproducible, and has a key that is a
hash of its own contents. Copying terabytes to preserve something your build can
regenerate is work you can skip.

The move that works instead is to put NativeLink in front of the old cache and
let it read through. Builds hit NativeLink; anything it doesn't have, it fetches
from the old cache and keeps. Your hit rate never drops, the new cache warms
itself on real traffic, and the old one is never written to again, so if you
need to go back, it is exactly as you left it.

<SourceLink file="nativelink-config/src/stores.rs" symbol="GrpcSpec" />

## The diode [#the-diode]

A `fast_slow` store whose `slow` half points at the old cache, marked read-only:

```json5
stores: [
  {
    name: "OLD_CACHE",
    grpc: {
      instance_name: "",
      store_type: "cas",
      endpoints: [
        {
          address: "grpcs://old-cache.example.com:443",
          tls_config: { use_native_roots: true },
        },
      ],
      rpc_timeout_s: 0,
    },
  },
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        filesystem: {
          content_path: "/data/cas/content",
          temp_path: "/data/cas/tmp",
          eviction_policy: { max_bytes: 500000000000 },
        },
      },
      slow: { ref_store: { name: "OLD_CACHE" } },
      slow_direction: "read_only",
    },
  },
],
```

`slow_direction: "read_only"` is the whole trick, and it is a documented use of
the field rather than a clever misuse: the source calls this shape a diode.
Reads fall through to the old cache when NativeLink misses. Writes stop at
NativeLink. The old cache is a source and never a destination.

The `tls_config` block is required on any `grpcs://` address; the
`use_native_roots` form is right for a hosted cache with a public certificate,
and `ca_file` is the alternative for a private CA
([TLS and authentication](/how-to/tls-and-auth) has both).

<SourceLink file="nativelink-config/src/stores.rs" symbol="FastSlowSpec" />

Do the same for the action cache with a second `grpc` store at
`store_type: "ac"`. Action results are the entries that actually decide your hit
rate; the CAS blobs they reference come along behind them.

Leave `rpc_timeout_s` at `0`. It disables the per-RPC deadline, which matters
because pulling a multi-gigabyte blob through the diode on a cold cache can take
longer than any timeout you would have guessed. Dead connections are still
caught by the HTTP/2 and TCP keepalives.

## What crosses the boundary and what doesn't [#what-crosses-the-boundary-and-what-doesnt]

**Digests must match.** If your old cache holds SHA-256 entries and your
builds now ask for BLAKE3, nothing is found, not because of a bug but because
the key of a blob is its hash, and you are asking a different question. The
digest function is chosen by the client (Bazel's `--digest_function`, SHA-256
unless you changed it); NativeLink's `global.default_digest_hash_function`,
described in [Stores](/configuration/stores), only fills in when a client does
not say. Keep the client, NativeLink and the old cache on the same function.
This is the one incompatibility that cannot be worked around.

**Instance names may need rewriting.** `instance_name` on the `grpc` store
replaces the instance name on every proxied call, so an old cache that expects
`main` while your clients send an empty instance is a one-field fix rather than
a client migration.

**Older backends want older resource names.** Modern REAPI resource names look
like `{instance}/blobs/{digest_function}/{hash}/{size}`. Buildbarn before v0.3
and other older servers expect the original `{instance}/blobs/{hash}/{size}`, and
sending them the modern form produces `InvalidArgument: Unsupported digest
function`. Set `use_legacy_resource_names: true` when you see that.

**Credentials come with you.** A hosted cache that wants a token takes it via
`headers`, and one that wants each developer's own token takes `forward_headers`
to pass the client's credential upstream:

```json5
grpc: {
  instance_name: "",
  store_type: "cas",
  endpoints: [
    {
      address: "grpcs://old-cache.example.com:443",
      tls_config: { use_native_roots: true },
    },
  ],
  headers: { authorization: "Bearer my-static-token" },
},
```

Header values are sent verbatim: they are not `${VAR}`-expanded like most
string fields, so the token lives in the config file and the file must be
protected accordingly.

**Remote execution does not transfer.** Workers, platform properties and queue
state are per-system. A read-through migration moves your cache; the execution
side is a separate build-out, and
[Remote execution](/remote-execution) is where it happens.

**Anything vendor-specific doesn't transfer either.** If you depend on a
feature of your current system outside the REAPI surface (a custom HTTP
interface, a proprietary authorization model, a bespoke admin API), that
dependency is what you should be examining before the cache bytes.

## Run both while you cut over [#run-both-while-you-cut-over]

Nothing forces a single switch date. Give the two caches separate config groups
in `.bazelrc`, so a build can pick:

```
build:oldcache --remote_cache=grpcs://old-cache.example.com:443
build:nativelink --remote_cache=grpcs://nativelink.example.com:50051
build --config=nativelink
```

Move CI first, because it is the highest-volume traffic and warms the new cache
fastest, and because its failures are visible immediately rather than reported
by a person mid-afternoon. Developers follow once CI has been steady for a week.

<Callout type="info" title="The diode is not permanent">
  Once NativeLink's own hit rate matches what the old cache was giving you, the
  `slow` tier is doing nothing but adding a network hop on misses. Remove it,
  and the `fast_slow` wrapper with it, leaving the filesystem store on its own.
  Deciding to remove it is the actual end of the migration.
</Callout>

## Steps [#steps]

<Steps>

<li>

**Confirm the digest function** your builds use, and set NativeLink's
`default_digest_hash_function` to match. Everything downstream depends on this
being right.

</li>

<li>

**Stand up NativeLink normally**, with its own filesystem or object store, and
check that a build against it works before the old cache is involved at all.

</li>

<li>

**Add the diode**: `grpc` stores for CAS and AC pointing at the old cache,
wrapped as the `slow` half of `fast_slow` with `slow_direction: "read_only"`.

</li>

<li>

**Point CI at NativeLink** and watch the hit rate. It should start near what the
old cache gave, because misses are being served through the diode, and it should
stay there.

</li>

<li>

**Move developers**, then wait. Give it long enough that a full build's worth of
actions has passed through, typically a week or two of normal work.

</li>

<li>

**Remove the diode** once misses are no longer being answered by the old cache,
then decommission it.

</li>

</Steps>

<VerifyBlock>

- A build that got hits from the old cache gets hits from NativeLink on the
  first run against it.
- NativeLink's `content_path` grows during that run, because it is keeping what it
  reads.
- A second identical build is served entirely by NativeLink, with no traffic to
  the old cache.
- The old cache's storage stops growing, confirming nothing is writing to it.
- Turning the old cache off entirely, at the end, changes nothing.

</VerifyBlock>

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Everything misses: the diode returns nothing" id="everything-misses--the-diode-returns-nothing">

Check the digest function first; a mismatch produces exactly this and no error.
Then check `instance_name`, since an old cache scoped to a named instance
returns nothing for a differently-named one. Then confirm the address is
reachable from the NativeLink process rather than from your laptop.

</Accordion>

<Accordion title="`InvalidArgument: Unsupported digest function`" id="invalidargument-unsupported-digest-function">

The upstream is an older server that does not understand modern resource names.
Set `use_legacy_resource_names: true` on the `grpc` store.

</Accordion>

<Accordion title="The old cache is getting writes" id="the-old-cache-is-getting-writes">

`slow_direction` is not set to `read_only`, or it is set on the wrong store. It
belongs on the `fast_slow` wrapper next to `slow`, not inside the `grpc` spec.
Its default is `both`, which is a normal read-write tier.

</Accordion>

<Accordion title="Large blobs fail partway through the diode" id="large-blobs-fail-partway-through-the-diode">

A per-RPC timeout. Set `rpc_timeout_s: 0` on the `grpc` store (the default),
or raise it well past the worst-case transfer time. Multi-gigabyte blobs
pulled cross-region are slower than any guess.

</Accordion>

<Accordion title="Hit rate is good but builds got slower" id="hit-rate-is-good-but-builds-got-slower">

Every miss now costs a round trip to the old cache on top of the local lookup.
That is the diode working as designed and it is temporary. If it is painful,
raise the local store's `max_bytes` so fewer reads fall through, and check
whether the old cache is in another region.

</Accordion>

<Accordion title="Should I ever actually copy the data?" id="should-i-ever-actually-copy-the-data">

Rarely, and only when the old cache holds artifacts that are expensive or
impossible to reproduce: a toolchain nobody can rebuild, or outputs from a
source revision that no longer exists. That is a one-off copy of specific blobs,
not a migration strategy. For everything a build can regenerate, the diode is
both faster and safer.

</Accordion>

</Accordions>

<NextStep href="/how-to/stores/compose-stores" title="Compose stores">
  The wrapper vocabulary the diode is built from, and what else it can do.
</NextStep>

<NextStep href="/operate/production-config" kind="aside" title="Production configuration">
  What to change once NativeLink is the cache rather than a proxy in front of
  one.
</NextStep>

# https://docs.nativelink.com/operate

**Operate**: What to watch, what to tune, and what to do at 3am, once other people depend on your cluster.

**Who this is for:** whoever gets paged when the build farm is slow. **What
you'll have at the end of this section:** a production-shaped deployment, a
metrics pipeline that tells you when it's unhealthy, and a written answer for
each of the ways it fails. **Time:** ongoing.

<Prerequisites>
  A NativeLink deployment you configured yourself. See [Your first full config](/configuration/your-first-full-config).
</Prerequisites>

## The shift this section represents [#the-shift-this-section-represents]

Everything up to the How-to guides is about making NativeLink do what you want.
This section is about making it keep doing that while you aren't looking.
The questions change shape: not "how do I configure a filesystem store" but
"what happens when that filesystem fills up overnight, how will I know, and what
do I do about it."

That's a different reader moment, so it's a different section, even though
some of the same subsystems come up.

## The arc [#the-arc]

**Get the shape right.** The production configuration is not the quickstart
configuration with bigger numbers. It has a fast tier over a durable one,
eviction policies that match your disk, and a deliberate split between the
public listeners and the worker API.

**Deploy it somewhere real.** Docker Compose or bare metal, each with the
topology, the resource sizing, and the failure domains that go with it. For
Kubernetes, the Helm charts ship with [NativeLink Enterprise](https://enterprise.nativelink.com).

**See it.** OTLP to Prometheus to Grafana, and the specific metrics worth
alerting on rather than all of them.

**Scale it.** Workers scale on queue depth and action duration; the CAS
scales on throughput and eviction pressure. They are separate problems with
separate signals, and conflating them is the most common scaling mistake.

**Tune it.** A small number of levers move the needle. Most of the config
surface does not, and knowing which is which saves a lot of afternoons.

**Harden it.** Port exposure, mTLS, and keeping the worker API private,
which matters more than it sounds, because a worker API is an arbitrary code
execution endpoint by design.

**Have runbooks.** Redis failover, worker OOM, disk full, backpressure. Each
one written down before it happens.

## Start here, or skip ahead [#start-here-or-skip-ahead]

**Start here** if you're moving from a working setup to one people depend on.

**Skip back** to [How-to guides](/how-to) if what you have isn't working the
way you want yet. Operate assumes the shape is right and the question is
whether it stays right.

## Current pages [#current-pages]

- [Production configuration](/operate/production-config): the three-process
  shape, the port surface, and the pre-launch checklist.
- [Tuning](/operate/tuning): the lever table (the signal you see, the knob
  that moves it, and what moving it costs).
- [Deploy with Docker Compose](/operate/deploy-docker-compose) or
  [on bare metal](/operate/deploy-bare-metal): the two deployment shapes,
  and what separates each from a production one.
- [Observability](/operate/observability): the OTLP to Prometheus to
  Grafana pipeline, and why skipping the collector silently breaks every
  shipped dashboard.
- [Security hardening](/operate/security-hardening): there is no inbound
  authentication anywhere, so the network is the authorization layer.
- [Scaling workers](/operate/scaling-workers): sizing one worker, the
  queue-depth signal, and wiring an autoscaler to it.
- [Scaling the CAS](/operate/scaling-cas): composing the store graph, sizing
  eviction, and what a second CAS process actually costs.
- [An autoscaling reference deployment](/operate/autoscaling-reference): the
  manifests that make the worker pool size itself, and the five gaps in the
  shipped examples you close first.
- [Runbooks](/operate/runbooks): the four incidents NativeLink actually
  produces: a worker OOM-killed, the disk full, Redis failing over, and a
  queue that backpressures and never recovers.
- [Troubleshooting](/operate/troubleshooting): symptom to cause to fix,
  indexed on the error strings the system really emits.

<NextStep href="/operate/production-config" title="Production configuration">
  The production shape: three processes, a split port surface, and what to turn
  on before other people depend on it.
</NextStep>

# https://docs.nativelink.com/operate/production-config

**Production configuration**: The shape a real NativeLink cluster runs in: three processes, a split port surface, and what to turn on before other people depend on it.

**Who this is for:** you have NativeLink working and now other people are going
to depend on it. **What you'll have at the end:** a three-process deployment
whose ports, state, and failure behaviour you can explain to someone else.
**Time:** an afternoon, plus however long your change process takes.

<Prerequisites>
  A working deployment that only you depend on.
</Prerequisites>

A production configuration is not the quickstart configuration with bigger
numbers. It differs in one structural way: **the CAS, the scheduler, and the
workers become three separate processes.** Everything else on this page (the
port split, the store composition, the hardening checklist) follows from that
one decision.

Every config on this page is adapted (paths and hostnames changed) from
<SourceLink file="deployment-examples/docker-compose" dir symbol="deployment-examples/docker-compose/" />,
which CI builds and runs end to end on every pull request: `run_integration_tests.sh`
brings the three processes up with `docker compose` and then drives real Bazel
builds through them.

## The shape [#the-shape]

<Mermaid>
{`flowchart LR
    B["Bazel / Buck2 / recc"] -->|CAS + AC + ByteStream :50051| CAS
    B -->|Execution + Capabilities :50052| SCH
    CAS[["CAS process<br/>CAS + AC + ByteStream"]]
    SCH[["Scheduler process<br/>queue + matching"]]
    W[["Worker process<br/>runs the actions"]]
    SCH -->|reads/writes blobs over gRPC| CAS
    W -->|worker_api :50061| SCH
    W -->|blobs over gRPC :50051| CAS
    CAS --> D[("filesystem or object store")]`}
</Mermaid>

Three facts about that picture are the whole design:

**The scheduler owns no storage.** It reaches the CAS through `grpc` stores, the
same way a client would. That is why you can restart it without losing blobs, and
why it needs no volume.

**The worker also owns no durable storage.** Its local disk is a read cache in
front of the CAS, not a source of truth. A worker can be deleted mid-action; the
scheduler retries the action elsewhere.

**Only the CAS process is stateful.** It is the one thing you back up, size for
growth, and think carefully about before restarting.

## Process 1: the CAS and Action Cache [#process-1--the-cas-and-action-cache]

This is the process that holds bytes. Two stores, one for content and one for
action results, each with its own eviction budget:

```json5
{
  stores: [
    {
      name: "CAS_MAIN_STORE",
      compression: {
        compression_algorithm: { lz4: {} },
        backend: {
          filesystem: {
            content_path: "/var/lib/nativelink/content_path-cas",
            temp_path: "/var/lib/nativelink/tmp_path-cas",
            eviction_policy: {
              // 10gb. This is a cache: size it to your working set,
              // not to your total build output.
              max_bytes: 10000000000,
            },
          },
        },
      },
    },
    {
      name: "AC_MAIN_STORE",
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-ac",
        temp_path: "/var/lib/nativelink/tmp_path-ac",
        eviction_policy: {
          // 500mb. The AC is tiny next to the CAS; it stores
          // result metadata, not outputs.
          max_bytes: 500000000,
        },
      },
    },
  ],
  servers: [
    {
      listener: { http: { socket_address: "0.0.0.0:50051" } },
      services: {
        cas: [{ cas_store: "CAS_MAIN_STORE" }],
        ac: [{ ac_store: "AC_MAIN_STORE" }],
        capabilities: [],
        bytestream: [{ cas_store: "CAS_MAIN_STORE" }],
      },
    },
  ],
}
```

`stores` is an **array of named stores**, not an object map; every store carries
its own `name`, and other parts of the config refer to it by that name.
`filesystem` requires both `content_path` and `temp_path`; the config parser uses
`deny_unknown_fields`, so a typo or an invented key is a startup failure rather
than a silently ignored line.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig" />

Each entry in `cas`, `ac`, `bytestream`, and `capabilities` carries an optional
`instance_name` (default: the empty string, which is also what Bazel sends by
default). If you set a non-empty one, every client has to pass it too
(`--remote_instance_name=` for Bazel). An empty `capabilities: []` still answers
`GetCapabilities` with cache capabilities for any instance name; an entry with
`remote_execution` is only needed to advertise remote execution.

### Moving off local disk [#moving-off-local-disk]

`filesystem` is the right terminal store for a single CAS node with a real disk.
For a CAS you intend to replicate, put an object store underneath instead and keep
the local disk as the fast tier:

```json5
{
  name: "CAS_MAIN_STORE",
  fast_slow: {
    fast: {
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-cas",
        temp_path: "/var/lib/nativelink/tmp_path-cas",
        eviction_policy: { max_bytes: 100000000000 },
      },
    },
    slow: {
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "nativelink-prod-cas",
        key_prefix: "cas/",
        retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
      },
    },
  },
}
```

The store key is `experimental_cloud_object_store` with a `provider` discriminator:
`aws`, `gcs`, `azure`, `ontap`, `r2`, or `oci`. No `s3_store` key exists.

<Callout type="warn" title="fast_slow does not guarantee the slow tier has everything">
  `fast_slow` never verifies that a blob present in `fast` also exists in `slow`.
  If the fast tier is wiped, anything that only ever landed there is gone. Size
  the fast tier as a cache of the working set and treat the slow tier as the
  thing you can lose data from.
</Callout>

The full backend list and the tradeoffs between them are in
[Storage backends](/how-to/stores/oci-object-storage) and
[Store overview](/reference/nativelink-config/store-overview).

## Process 2: the scheduler [#process-2--the-scheduler]

The scheduler holds the queue and matches actions to workers. It reaches the CAS
over gRPC rather than owning a store:

```json5
{
  stores: [
    {
      name: "GRPC_CAS_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "cas",
      },
    },
    {
      name: "GRPC_AC_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "ac",
      },
    },
  ],
  schedulers: [
    {
      name: "MAIN_SCHEDULER",
      simple: {
        supported_platform_properties: {
          cpu_count: "minimum",
          OSFamily: "priority",
          "container-image": "priority",
          ISA: "exact",
        },
      },
    },
  ],
  servers: [
    {
      // Client-facing. Anyone who can reach this can queue work.
      listener: { http: { socket_address: "0.0.0.0:50052" } },
      services: {
        ac: [{ ac_store: "GRPC_AC_STORE" }],
        execution: [{ cas_store: "GRPC_CAS_STORE", scheduler: "MAIN_SCHEDULER" }],
        capabilities: [{ remote_execution: { scheduler: "MAIN_SCHEDULER" } }],
      },
    },
    {
      // Worker-facing. Never expose this outside the cluster.
      listener: { http: { socket_address: "0.0.0.0:50061" } },
      services: {
        worker_api: { scheduler: "MAIN_SCHEDULER" },
        health: {},
      },
    },
  ],
}
```

`schedulers`, like `stores`, is an **array of named entries**.

`supported_platform_properties` is the matching contract between this scheduler
and its workers. Each value is a match rule (`exact`, `minimum`, `priority`, or
`ignore`). A worker that advertises a property the scheduler does not list is
rejected at registration (`Unknown platform property`), and an action that
requests one is rejected the same way. Getting this out of sync with the
workers is the single most common reason actions sit in the queue forever; see
[platform properties](/remote-execution/platform-properties) for how the two
halves line up.

<SourceLink file="nativelink-scheduler/src/platform_property_manager.rs" symbol="make_prop_value" />

<Callout type="warn" title="The queue is in memory">
  The default scheduler keeps its queue in process memory. Restarting it loses the
  in-flight queue; clients retry, but work in progress is repeated. No
  mechanism reconstructs the queue from the Action Cache. If you need the
  queue to survive a restart, the scheduler has an experimental Redis backend
  (`experimental_backend: { redis: { redis_store: "..." } }`). The only example
  of it in this repository is
  `nativelink-config/examples/worker_with_redis_scheduler.json5`; there is no
  Compose or Kubernetes deployment of it, so treat it as something to validate
  yourself rather than something to copy.
</Callout>

## Process 3: the workers [#process-3--the-workers]

Workers are the fleet you scale. Note `servers: []`: a worker serves nothing and
needs no inbound port at all. It dials out to the scheduler and the CAS.

```json5
{
  stores: [
    {
      name: "GRPC_CAS_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "cas",
      },
    },
    {
      name: "GRPC_AC_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "ac",
      },
    },
    {
      name: "WORKER_FAST_SLOW_STORE",
      fast_slow: {
        fast: {
          filesystem: {
            content_path: "/var/lib/nativelink/worker/content_path-cas",
            temp_path: "/var/lib/nativelink/worker/tmp_path-cas",
            eviction_policy: { max_bytes: 10000000000 },
          },
        },
        // "get" means the local disk serves reads but is not written
        // through on upload; inputs get cached, outputs go straight out.
        fast_direction: "get",
        slow: { ref_store: { name: "GRPC_CAS_STORE" } },
      },
    },
  ],
  workers: [
    {
      local: {
        worker_api_endpoint: { uri: "grpc://scheduler.internal:50061" },
        cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
        upload_action_result: { ac_store: "GRPC_AC_STORE" },
        work_directory: "/var/lib/nativelink/worker/work",
        platform_properties: {
          cpu_count: { query_cmd: "nproc" },
          OSFamily: { values: [""] },
          "container-image": { values: [""] },
          ISA: { values: ["x86-64"] },
        },
        use_namespaces: true,
        use_mount_namespace: true,
      },
    },
  ],
  servers: [],
}
```

`worker_api_endpoint`, `cas_fast_slow_store`, `work_directory`, and
`platform_properties` are all required. `upload_action_result` takes `ac_store`
directly; it is not nested twice.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

`work_directory` is where actions are staged and executed. It sees every input
file and every output file of every action, so put it on the fastest local disk
the machine has, and size it for the largest action tree you run rather than the
average one.

## Which ports are public [#which-ports-are-public]

| Port | Process | Serves | Who should reach it |
| --- | --- | --- | --- |
| `50051` | CAS | `cas`, `ac`, `bytestream`, `capabilities` | Build clients and, internally, schedulers and workers |
| `50052` | Scheduler | `execution`, `capabilities`, `ac` | Build clients |
| `50061` | Scheduler | `worker_api`, `health` | **Workers only** |

<Callout type="warn" title="The worker API is a remote code execution endpoint">
  `worker_api` is how a process claims to be a worker and receives commands to
  run. It is on its own listener specifically so that it can be on its own
  network policy. Anything that can reach `:50061` can register as a worker,
  receive your actions, and return whatever outputs it likes, which then land in
  your Action Cache and get served to everyone. Keep it inside the cluster.
  [Security hardening](/operate/security-hardening#the-worker-api-has-no-authentication)
  covers the full consequence and what to do about it.
</Callout>

`health` is served alongside `worker_api` in the reference config. It is the only
readiness signal the binary exposes; there is no metrics port to probe.

## What's durable and what's disposable [#whats-durable-and-whats-disposable]

| Thing | Lose it and… | So |
| --- | --- | --- |
| CAS content store | Builds recompile from source until it warms back up | Back it up or accept the warm-up; this is the one stateful component |
| Action Cache | Same, but cheaper; results re-derive from the CAS | Size it small; it is result metadata, not outputs |
| Scheduler queue | In-flight actions are re-run by clients | Restart freely, off-peak |
| Worker local disk | Worker re-fetches inputs from the CAS | Delete workers at will; this is the property that makes autoscaling safe |
| `work_directory` | Nothing; it is scratch space | Never put it on network storage |

The rule this table encodes: **exactly one component is stateful.** If you find
yourself needing to protect a scheduler's disk or a worker's disk, something has
drifted from this shape.

## Before you go live [#before-you-go-live]

<Steps>

### Raise the file-descriptor limit [#raise-the-file-descriptor-limit]

Every open blob is a file descriptor. The binary gates file opens on a semaphore
sized to 80% of the limit it actually achieved, so a CAS at the default limit
does not error under load; it waits.

```json5
global: {
  // The reference Kubernetes worker sets 524288. The binary's own
  // default is 24576 when `global` is omitted or this is 0.
  max_open_files: 524288,
},
```

`global` accepts exactly three keys: `max_open_files` (required whenever the
`global` block is present), `default_digest_hash_function`, and
`default_digest_size_health_check`. It has no `metrics` or `tracing` section;
telemetry is configured by environment variable, below.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="GlobalConfig" />
<SourceLink file="nativelink-util/src/fs.rs" symbol="set_open_file_limit" />

### Turn on namespaces on Linux workers [#turn-on-namespaces-on-linux-workers]

Bazel in particular can leave zombie processes behind on a worker. Two opt-in
flags contain that:

```json5
workers: [{
  local: {
    use_namespaces: true,
    use_mount_namespace: true,
    // ...the rest of the worker config
  },
}],
```

`use_namespaces` puts each action in its own process namespace;
`use_mount_namespace` additionally isolates the worker root in a new mount
namespace, and only works when `use_namespaces` is on. Both are off by default
and both are recommended in production. The container
running the worker needs the privileges to create namespaces; the reference
Compose stack marks the executor service `privileged: true` for exactly this.

### Put TLS on the client-facing listeners [#put-tls-on-the-client-facing-listeners]

```json5
servers: [{
  listener: {
    http: {
      socket_address: "0.0.0.0:50071",
      tls: {
        cert_file: "/etc/nativelink/tls/server.crt",
        key_file: "/etc/nativelink/tls/server.key",
        // Optional: require client certificates (mTLS).
        client_ca_file: "/etc/nativelink/tls/clients-ca.crt",
        client_crl_file: "/etc/nativelink/tls/clients.crl",
      },
    },
  },
  services: { /* ... */ },
}],
```

Setting `client_ca_file` makes the listener require a client certificate signed by
that CA, and `client_crl_file` lets you revoke one. That is the whole
authorization model on the listener: **any client with a valid, unrevoked
certificate gets full access to that listener's services.** No per-identity
allow-list exists and no mapping from certificate subject to permissions. If you need
finer-grained access than "trusted or not," put a proxy in front.

The certificates checked into `deployment-examples/` are self-signed and named
`do-not-use-in-prod` for a reason. Generate your own.

### Point telemetry somewhere [#point-telemetry-somewhere]

The config file has no metrics section, and the binary has no Prometheus scrape endpoint
in the binary. Telemetry is OTLP over gRPC, configured by environment:

```bash
# Where to send traces, metrics, and logs. Must include an explicit
# host AND port; the client panics on a URL without both.
export NL_OTEL_ENDPOINT="http://otel-collector.internal:4317"

# Log format: "compact", "json", or anything else for pretty.
export NL_LOG=json

# Standard tracing filter.
export RUST_LOG=info
```

Prometheus sees NativeLink through an OpenTelemetry Collector that exports to it,
or through Prometheus' own OTLP receiver. [Observability](/operate/observability)
has the collector config and the Grafana dashboard.

### Opt in to cache metrics, if you want them [#opt-in-to-cache-metrics-if-you-want-them]

Execution metrics are emitted unconditionally. **Cache metrics are not**: a store
only reports hits, misses, and errors on its reads, writes, and deletes if you
wrap it:

```json5
{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: {
      // ...the store you would otherwise have written here
    },
  },
}
```

Without the wrapper there are no `nativelink_cache_*` series at all, and any
dashboard panel or alert built on them stays empty forever. The wrapper is opt-in
because it costs hot-path timing on every operation.

</Steps>

<VerifyBlock>

Each process starts and stays up: `nativelink /etc/nativelink/cas.json5` exits
non-zero immediately on a config error rather than starting degraded, so a
process that is still running has a config the parser fully accepted.

A build against `grpc://cas.internal:50051` gets cache hits on a second run, and
a build against `grpc://scheduler.internal:50052` reports remote execution rather
than falling back to local.

`:50061` is refused from outside the cluster.

</VerifyBlock>

## What to watch [#what-to-watch]

Four signals cover most of what goes wrong. All four are recording rules in
<SourceLink file="deployment-examples/metrics/prometheus-recording-rules.yml" symbol="deployment-examples/metrics/prometheus-recording-rules.yml" />:

- `nativelink:queue_depth`: actions waiting, by priority. Rising and staying
  risen means you need workers; spiking and draining is normal.
- `nativelink:worker_utilization`: the fraction of workers actually executing.
  Low utilization with a deep queue means a matching problem, not a capacity
  problem.
- `nativelink:execution_queue_time_p95`: how long an action waits before a
  worker picks it up. This is the number your users experience as "the build farm
  is slow."
- `nativelink:cache_hit_rate`: requires the `cache_metrics` wrapper above.

[Observability](/operate/observability) covers the pipeline that produces these.

## What to tune [#what-to-tune]

Once the shape is right, [Tuning](/operate/tuning) is the table of levers: the
signal you observe, the knob that moves it, and what it costs you.

## What to do when it breaks [#what-to-do-when-it-breaks]

[Troubleshooting](/operate/troubleshooting) indexes symptoms to causes, and
[Runbooks](/operate/runbooks) has the procedure for each of the four incidents
this shape actually produces.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="What happens to in-flight actions when a scheduler restarts?" id="what-happens-to-in-flight-actions-when-a-scheduler-restarts">
    They are lost from the scheduler's point of view. The default scheduler holds
    its queue in process memory and nothing reconstructs it on startup. Clients
    re-submit, so no build fails, but work in progress is repeated. Restart
    schedulers when the queue is shallow.
  </Accordion>
  <Accordion title="How do I route GPU or other special-hardware actions?" id="how-do-i-route-gpu-or-other-special-hardware-actions">
    Advertise the property on the workers that have the hardware
    (`gpu: { values: ["nvidia-a100"] }`) and declare it as `"exact"` in the
    scheduler's `supported_platform_properties`. An action requesting that value
    is then only ever dispatched to a matching worker. Both halves are required:
    a worker advertising a property the scheduler does not list is rejected at
    registration.
  </Accordion>
  <Accordion title="Does fast_slow guarantee everything reaches the slow tier?" id="does-fast-slow-guarantee-everything-reaches-the-slow-tier">
    No. It never verifies that an object in `fast` also exists in `slow`. For
    artifacts that must survive a fast-tier wipe (remote execution outputs
    especially), make sure the write path covers both tiers deliberately. More
    store gotchas:
    [Store overview](/reference/nativelink-config/store-overview#faq).
  </Accordion>
  <Accordion title="Can I run this as one process instead of three?" id="can-i-run-this-as-one-process-instead-of-three">
    Yes, and for a single-machine deployment you should: put the stores,
    schedulers, workers, and servers in one file and run one binary. The split
    exists so the three roles can be scaled and firewalled independently. If you
    have one machine, you have neither problem.
  </Accordion>
</Accordions>

<NextStep href="/operate/tuning" title="Tuning">
  The lever table: which knob moves which signal, in which direction, and what it
  costs.
</NextStep>

<NextStep href="/operate/security-hardening" title="Security hardening" kind="aside">
  No service authenticates inbound callers. Plan the network before
  anyone else can reach it.
</NextStep>

# https://docs.nativelink.com/operate/tuning

**Tuning**: The lever table: the signal you observe, the knob that moves it, the direction to turn it, and what turning it costs.

**Who this is for:** you have the
[production shape](/operate/production-config) running and a number you don't
like. **What you'll have at the end:** the specific knob for that number, and an
honest account of what moving it costs.

<Prerequisites>
  The production shape running (see [Production configuration](/operate/production-config)), and a number you don't like.
</Prerequisites>

Tuning NativeLink is not a matter of turning up every dial. A small number of
levers move real workloads; most of the config surface does not. Each row below
starts from **a signal you can actually observe** (a metric, an error, a
complaint), because starting from a knob and hoping is how afternoons disappear.

<Callout type="info" title="Change one thing">
  Every lever here has a cost column, and several of them trade against each
  other. Move one, watch the signal it targets for a full build cycle, then
  decide whether to move the next.
</Callout>

## Storage and the CAS path [#storage-and-the-cas-path]

| Signal you see | Knob | Direction | Expected effect | Cost / risk |
| --- | --- | --- | --- | --- |
| Cache hit rate falling, disk has headroom | `max_bytes` in `eviction_policy` | Up | More of the working set stays resident; hits go up | Disk. Nothing else; it's a cache |
| CAS read latency high, disk is the bottleneck | `content_path` / `temp_path` | Move to NVMe | Direct latency win on every blob | Cost of the disk; `temp_path` must be on the same filesystem as `content_path` |
| Egress or storage bill dominated by blob volume | `compression` with `lz4` | Wrap the terminal store | Fewer bytes stored and sent | CPU on every read and write through that store |
| Per-request object-store cost dominated by tiny blobs | `size_partitioning` | Split at ~64k to 1MB | Small blobs go to a cheap fast store, large ones to the object store | Two stores to size instead of one |
| Existence checks (`FindMissingBlobs`) dominating CAS load | `existence_cache` | Wrap the backend | Repeat existence checks answered locally | Stale negatives if the backend is mutated out of band; needs its own `eviction_policy` or it grows without bound |
| The same large artifacts uploaded repeatedly with small deltas | `dedup` | Wrap with an index store | Only changed chunks are stored | Chunk index store to run and size; more round trips per `get()` |
| One backend at its throughput ceiling | `shard` | Add shards | Load spreads by digest across backends | Resharding changes which backend holds what; plan it as a migration |
| Read latency high but the working set is small | `fast_slow` hot tier | Size `fast` to the 24 to 48h working set | Hot reads served locally, cold reads fall through | `fast_slow` never verifies the slow tier has everything; see the warning on [production config](/operate/production-config) |
| Worker disk filling with blobs it only reads | `fast_direction: "get"` | Set it | Local tier caches inputs without being written through on upload | Outputs are not locally cached; they go straight to the CAS |
| Page cache growth on a co-located filesystem-only deployment | `evict_page_cache` | **Almost always leave off** | Kernel drops this store's pages after each read and write | Severe on many-core hosts; see below |

<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

### The `evict_page_cache` lever [#the-evict_page_cache-lever]

The `filesystem` store has an opt-in `evict_page_cache` flag (default `false`).
When enabled, after every blob it writes or reads it asks the kernel to drop that
file's pages from the page cache (`posix_fadvise(POSIX_FADV_DONTNEED)`).

Most production deployments should leave it **off**. The common enterprise
topology (an object store with multiple CAS nodes) never benefits from it, and
enabling it carries two costs.

```json5
stores: [{
  name: "CAS_MAIN_STORE",
  filesystem: {
    content_path: "/var/lib/nativelink/content_path-cas",
    temp_path: "/var/lib/nativelink/tmp_path-cas",
    // Leave this off unless the narrow case below applies.
    evict_page_cache: false,
  },
}],
```

<Callout type="warn" title="The cost grows with core count">
  On a real filesystem (xfs/ext4), `POSIX_FADV_DONTNEED` takes the kernel's
  `lru_add_drain_all()` path: it schedules and waits on a drain across **every
  online CPU**, and concurrent callers serialize on a global lock. The per-call
  cost therefore grows with the number of cores, and **adding workers or
  concurrent actions cannot overcome it**: every caller converges on the same
  serialized kernel path, so throughput is capped no matter how much parallelism
  you add. On a many-core host this turns a busy store into a bottleneck while
  CPU and disk sit near idle. (On `tmpfs` the call is a no-op, so this cost is
  invisible in tmpfs-backed tests.)
</Callout>

The second cost: it evicts the page cache. A store kept on a fast local disk that
relies on the page cache as its hot read tier loses that cache after every read
and write.

**When to enable it:** only for a filesystem-backed store on a low-core host
where keeping this store's I/O out of the page cache is specifically desired (for
example, to bound page-cache growth on a co-located filesystem-only deployment).
It was originally added to relieve page-cache pressure that contributed to worker
OOM-kills on such a deployment; note that page cache is reclaimable and the
dominant OOM driver is a worker's anonymous memory (action processes and
allocator retention), so this flag is a narrow mitigation, not a general OOM fix.
Do **not** enable it on many-core hosts or object-store / multi-CAS deployments.

<SourceLink file="nativelink-config/src/stores.rs" symbol="FilesystemSpec" />

## Workers [#workers]

| Signal you see | Knob | Direction | Expected effect | Cost / risk |
| --- | --- | --- | --- | --- |
| Worker CPU idle while the queue is deep | `max_inflight_tasks` | Up, toward core count | More actions run concurrently per worker | Memory. Actions are real processes; oversubscribing invites the OOM killer. Default `0` means unlimited |
| Workers OOM-killed under load | `max_inflight_tasks` | Down | Fewer concurrent action processes | Lower per-worker throughput; you need more workers |
| Long link or EDA steps failing with a timeout | `max_action_timeout_s` | Up | Long actions get to finish | A genuinely hung action now holds a slot for longer. Default is 20 minutes |
| Zombie or leaked processes after builds | `use_namespaces` + `use_mount_namespace` | On | Each action is contained and cleaned up | Linux only; the container needs privileges to create namespaces |
| `work_directory` filling or slow | `work_directory` | Move to fast local disk, size for the largest action | Faster staging on every action | Never put it on network storage; it is the hot path of every action |
| Actions never dispatched to the right hardware | worker `platform_properties` | Add the property | The scheduler can route on it | The scheduler must list it too, or it rejects the worker's registration with `Unknown platform property`; see below |

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

Worker count itself is not a lever on this page; it is a
[scaling](/operate) decision driven by `nativelink:queue_depth`, not by
worker CPU.

## Scheduler [#scheduler]

| Signal you see | Knob | Direction | Expected effect | Cost / risk |
| --- | --- | --- | --- | --- |
| Actions queued forever while workers sit idle | `supported_platform_properties` | Add the property the workers advertise | The scheduler can match them | A property listed here but absent on workers matches nothing; the two halves must agree |
| Spot / preemptible workers causing failed builds | `max_job_retries` | Up from the default of 3 | Interrupted actions are retried elsewhere | A deterministically failing action is now retried that many times before the client sees it |
| Healthy workers being dropped and re-registering | `worker_timeout_s` | Up from the default of 5 | Fewer false drops on a slow or busy network | A genuinely dead worker is noticed later, so its actions are re-queued later |
| Actions stuck in `executing` on a wedged worker | `max_action_executing_timeout_s` | Set it (default `0` = disabled) | Wedged actions are reclaimed | Too low and long legitimate actions get killed mid-flight |
| Clients giving up before the scheduler does | `client_action_timeout_s` | Align with client timeouts | Scheduler and client agree on how long is too long | Default is 60 seconds |

<SourceLink file="nativelink-config/src/schedulers.rs" symbol="SimpleSpec" />

## Process-level [#process-level]

| Signal you see | Knob | Direction | Expected effect | Cost / risk |
| --- | --- | --- | --- | --- |
| Blob reads and writes stall under load, or `Too many open files` in the log | `global.max_open_files` | Up | More concurrent blobs in flight (the binary gates file opens on a semaphore sized to 80% of the achieved limit) | Capped by the hard `ulimit -n`; the binary defaults to 24576 when `global` is omitted |
| Redis connection saturation | `connection_pool_size` on `redis_store` | Up from the default of 3 | More concurrent Redis commands | More connections against the Redis server's own limit |
| Redis commands timing out | `command_timeout_ms` / `connection_timeout_ms` on `redis_store` | Up from 10000 / 3000 | Fewer spurious failures on a loaded Redis | Slower failure detection. Use these, not the deprecated `response_timeout_s` / `connection_timeout_s` |
| High-latency links between clients and CAS | `advanced_http` HTTP/2 window and stream settings | Up | Better throughput on fat, long pipes | These are hyper's knobs and mostly should be left alone; the reference config sets them only under a `TODO` |

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="GlobalConfig" />
<SourceLink file="nativelink-config/src/stores.rs" symbol="RedisSpec" />

## What is not a lever [#what-is-not-a-lever]

Worth stating, because each of these gets reached for and none of them helps:

**Worker CPU utilization as a scaling signal.** A worker waiting on the CAS looks
idle. Scale on `nativelink:queue_depth` instead.

**Adding parallelism to work around `evict_page_cache`.** It cannot work; see the
callout above.

**A metrics port.** There isn't one. Telemetry is OTLP over gRPC via
`NL_OTEL_ENDPOINT`, and cache metrics additionally require the `cache_metrics`
store wrapper. [Observability](/operate/observability) has the pipeline.

<NextStep href="/operate/runbooks" title="Runbooks" kind="aside">
  Tuning is for a healthy system. When one of these signals is a symptom rather
  than a preference, start here instead.
</NextStep>

<NextStep href="/operate/observability" title="Observability">
  The OTLP to Prometheus to Grafana pipeline that produces every signal in the
  left-hand column above.
</NextStep>

# https://docs.nativelink.com/operate/deploy-docker-compose

**Deploy with Docker Compose**: Bring up a complete CAS, scheduler and worker on one host with the compose files in the repository: what each service is, which ports it publishes, and the four things that make this a development setup rather than a production one.

**Who this is for:** you want the whole system (cache *and* remote execution)
running on one machine, either to try it or to give a small team something
shared. **What you'll have at the end:** three containers serving a real CAS,
scheduler and worker, a build pointed at them, and a clear picture of what
separates this from production. **Time:** 20 minutes, most of it spent
compiling.

<Prerequisites>
  A config you wrote (see [Your first full config](/configuration/your-first-full-config)), and a host with Docker on it.
</Prerequisites>

Of the deployment shapes NativeLink ships, Compose is the one that runs
end-to-end out of the box. The Kubernetes building blocks need image tags and
a Gateway you supply; the Compose files need nothing but Docker.

## The configuration [#the-configuration]

```sh
git clone https://github.com/TraceMachina/nativelink
cd nativelink/deployment-examples/docker-compose
docker compose up -d
```

The first run builds the image from source with Bazel, which takes a while;
see [the build is not fast](#the-build-is-not-fast) below.

Three services come up:

| Service | Config file | Role | Published ports |
| --- | --- | --- | --- |
| `nativelink_local_cas` | `local-storage-cas.json5` | CAS, Action Cache, ByteStream | `50051` (plain), `50071` (TLS), and a `127.0.0.1:50061` mapping that nothing in the CAS config listens on |
| `nativelink_scheduler` | `scheduler.json5` | Execution, Capabilities, AC frontend; worker API | `50052` |
| `nativelink_executor` | `worker.json5` | Runs actions. `privileged: true` | none |

<Mermaid>
{`flowchart LR
    BZ["Bazel / Buck2"]
    CAS["nativelink_local_cas<br/>50051 plain · 50071 TLS"]
    SCH["nativelink_scheduler<br/>50052 frontend · 50061 worker API"]
    EX["nativelink_executor<br/>privileged, no published port"]
    D[("~/.cache/nativelink<br/>on the host")]

    BZ -->|"--remote_cache"| CAS
    BZ -->|"--remote_executor"| SCH
    EX -->|"registers on 50061"| SCH
    EX -->|"blobs on 50051"| CAS
    SCH -->|"AC and CAS on 50051"| CAS
    CAS --- D
    EX --- D`}
</Mermaid>

They talk to each other by service name. The worker config reaches the
scheduler at `grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061` and the CAS at
`grpc://${CAS_ENDPOINT:-127.0.0.1}:50051`; Compose sets both variables, and
NativeLink expands them at config-parse time; see
[CLI and environment](/reference/cli-and-env).

### Point a build at it [#point-a-build-at-it]

```sh
bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --remote_default_exec_properties=cpu_count=2
```

Cache and executor are **different endpoints**. That is not an accident of the
example: the CAS process and the scheduler process are genuinely separate
here, exactly as they are in the
[production shape](/operate/production-config).

`cpu_count` is what the shipped README passes. The scheduler declares it as a
`minimum` platform property and the worker advertises it from `nproc`, so an
action is only matched to a worker with at least that many cores; ask for more
than the worker has and the action queues silently forever. Omitting the
property is fine: an action with no platform properties matches any worker. See
[platform properties](/remote-execution/platform-properties).

## What to change [#what-to-change]

Only the fields carrying a decision. Everything else in the shipped files is
already correct for a single host.

| Field | File | Why you'd change it |
| --- | --- | --- |
| `eviction_policy.max_bytes` | `local-storage-cas.json5`, `worker.json5` | Ships at 10 GB for the CAS and 500 MB for the AC, plus a separate 10 GB fast tier for the worker, all under the same host directory. Size them to your disk. The worker's tier can reach roughly **twice** its `max_bytes`, because the executable-variant directory the worker creates next to `content_path` is not counted against it. See [Runbooks](/operate/runbooks#the-disk-fills-up). |
| `NATIVELINK_DIR` | environment | Where the CAS and worker caches land on the host. Defaults to `~/.cache/nativelink`. |
| `RUST_LOG` | environment | Ships at `warn`, which hides the scheduler's `INFO`-level matching diagnostics. Set `info` the first time something does not run. |
| `platform_properties` | `worker.json5` | The worker's advertised capabilities. Whatever you add here must also appear in the scheduler's `supported_platform_properties`, or the scheduler rejects the worker's registration with `Unknown platform property`. |
| `ADDITIONAL_SETUP_WORKER_CMD` | build arg | A bash string run inside the final image. This is the hook for installing whatever toolchain your actions need. |
| `experimental_chunking` | `local-storage-cas.json5` | Enabled and pointed at a separate `CHUNK_INDEX_STORE`, for Bazel's `--experimental_remote_cache_chunking`. Remove both if you are not using that flag. |

<Callout type="info" title="The chunk index must not be the CAS">
  `CHUNK_INDEX_STORE` holds blob-to-chunk layouts, not blobs. It must be a
  separate store that does not verify digests; pointing `index_store` at
  `CAS_MAIN_STORE` produces verification failures on every split.
</Callout>

## The multi-worker variant [#the-multi-worker-variant]

`docker-compose-multi-worker.yml` runs one CAS, one scheduler and three
workers sharing a `cas-data` volume:

```sh
docker compose -f docker-compose-multi-worker.yml build
docker compose -f docker-compose-multi-worker.yml up -d
```

The shared volume is the entire point, and the README says so more loudly than
anything else in it: **all workers must share the same CAS storage path.**
When they do not, actions fail with

```text
Object <digest> not found in either fast or slow store. If using multiple
workers, ensure all workers share the same CAS storage path.
```

which the worker reports to the scheduler as `FAILED_PRECONDITION` with that
message intact, so that is the text your developers will see.
[Troubleshooting](/operate/troubleshooting#actions-fail-with-a-cas-miss)
has the full path.

<Callout type="warn" title="Three defects in the multi-worker files">

  The first two are cosmetic if you know about them and confusing if you do not.

  The `healthcheck` on `cas-server` and `scheduler` curls `/status` on ports
  50051 and 50052. **Neither of those listeners has the `health` service
  enabled**; only the 50061 listener does. Both healthchecks therefore fail
  permanently. Nothing depends on them (`depends_on` uses
  `condition: service_started`), so the stack works anyway, but
  `docker compose ps` will show the containers as unhealthy forever.

  It publishes `50061:50061` on **all interfaces**, unlike
  `docker-compose.yml`, which does not publish the scheduler's 50061 at all.
  Port 50061 is the worker API. Read
  [Security hardening](/operate/security-hardening#the-worker-api-has-no-authentication)
  before running this on anything with a routable address.

  The shipped `worker-shared-cas.json5` keeps the worker's fast tier under
  `/root/.cache/nativelink/`, not under the `/data/cas` mount, so the three
  workers do not actually share the volume the compose file mounts for them.
  Point `content_path` and `temp_path` at `/data/cas/...` if you want the
  sharing the README describes.

</Callout>

## What makes this a development setup [#what-makes-this-a-development-setup]

Four things, all of them deliberate in the example and all of them
disqualifying for production.

### The executor runs privileged [#the-executor-runs-privileged]

```yaml
nativelink_executor:
  privileged: true # So we can do use_namespaces
```

`worker.json5` sets `use_namespaces: true` and `use_mount_namespace: true`,
which need `CLONE_NEWUSER`/`CLONE_NEWNS`, hence the flag. Note what this buys
and what it does not: namespaces here are a hermeticity and process-reaping
mechanism, not a security boundary. Actions still run as the worker's own uid
through an identity uid map, still see the host filesystem, and still have
network access to every NativeLink port. A privileged container running
arbitrary build actions is a lateral-movement primitive.
[Security hardening](/operate/security-hardening#the-sandbox-is-not-a-security-boundary)
covers what the sandbox actually does.

### The TLS certificates are expired examples [#the-tls-certificates-are-expired-examples]

The 50071 listener loads
`example-do-not-use-in-prod-rootca.crt` and `example-do-not-use-in-prod-key.pem`
from the same directory. They are committed to the repository, self-signed,
`CN=localhost`, and **expired in October 2024**. Every client must pass
`--insecure` or equivalent, which is what the repository's own TLS integration
test does. No certificate-generation script exists anywhere in the tree; you
supply real material yourself.

### Nothing is authenticated [#there-is-no-authentication-on-anything]

Not a Compose limitation: open-source NativeLink has no inbound
authentication on any service, in any deployment. `docker-compose.yml` does not
publish the scheduler's port 50061 at all; the worker reaches it over the
Compose network by service name. That is the only thing standing between an
attacker on your network and the ability to register as a worker. Keep it
that way.

### The build is not fast [#the-build-is-not-fast]

The `Dockerfile` compiles NativeLink from the repository root with Bazel, in a
three-stage build, and the final image is a fresh Ubuntu with the binary copied
in and `curl` installed. The Ubuntu base is digest-pinned and the bazelisk
download is checksum-verified, so the build is reproducible, but it is a full
from-source build every time the layer cache misses. For repeated use, build once and push the image to a
registry your hosts can pull from.

<SourceLink file="deployment-examples/docker-compose/docker-compose.yml" symbol="docker-compose.yml" />
<SourceLink file="deployment-examples/docker-compose/local-storage-cas.json5" symbol="CAS config" />
<SourceLink file="deployment-examples/docker-compose/worker.json5" symbol="worker config" />

<VerifyBlock>

  `docker compose ps` shows all three services `Up`.

  A `bazel build` with the two flags above completes. Run it twice; the second
  run reports a high `remote cache hit` count in the summary line.

  `docker compose logs nativelink_executor` contains
  `Worker registered with scheduler`. If it instead repeats
  `Could not connect to endpoint grpc://nativelink_scheduler:50061` roughly
  twice a second, the worker never reached port 50061; reconnect is a hard-coded
  0.5-second retry with no backoff, so that cadence is the signature.

  `ls ~/.cache/nativelink` shows `content_path-cas` growing as you build.

</VerifyBlock>

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| Build hangs with no action ever starting | The scheduler has no worker matching your platform properties, and a value mismatch queues silently | Set `RUST_LOG=info` and look for `No workers matched!`; see [Troubleshooting](/operate/troubleshooting#actions-queue-forever-and-workers-sit-idle) |
| `Unknown platform property` at once | The property *name* is not in the scheduler's `supported_platform_properties` | Add it to `scheduler.json5` and restart the scheduler |
| Actions fail with `FAILED_PRECONDITION` about fast or slow store | Workers do not share CAS storage | Use the shared volume in the multi-worker file |
| Executor container exits at startup | `use_namespaces: true` without `privileged: true` | Keep the flag, or set both namespace options to `false` |
| TLS client refuses port 50071 | The committed certificate expired in 2024 | Use port 50051, or supply your own certificate and key |
| `docker compose ps` shows containers unhealthy but everything works | The multi-worker healthchecks probe a port with no health service | Ignore it, or move the healthcheck to 50061 |
| Disk fills, then every write errors | The 10 GB eviction policy is smaller than what the store actually consumes | [Runbooks](/operate/runbooks#the-disk-fills-up) |

## What's next [#whats-next]

<NextStep href="/operate/production-config" title="Production configuration">
  The same three processes, configured for people who depend on them.
</NextStep>

<NextStep href="/operate/deploy-bare-metal" title="Move it to real hosts" kind="aside">
  One binary per role under systemd, with a load balancer in front.
</NextStep>

<NextStep href="/operate/security-hardening" title="Before anyone else can reach it" kind="aside">
  Nothing is authenticated. Plan the network accordingly.
</NextStep>

# https://docs.nativelink.com/operate/deploy-bare-metal

**Deploy on bare metal**: Run NativeLink as systemd services on hosts you own: writing the units, sizing the machines, placing the disks, and rolling an upgrade without dropping in-flight actions.

**Who this is for:** you have VMs or physical machines and no interest in
running a cluster orchestrator to get a build cache. **What you'll have at the
end:** three systemd units, a disk layout that will not surprise you, host
sizing you can defend, and a restart procedure. **Time:** an afternoon.

<Prerequisites>
  A config you wrote (see [Your first full config](/configuration/your-first-full-config)), and hosts you can install a systemd unit on.
</Prerequisites>

No systemd unit exists in the NativeLink repository. That is the honest
starting point for this page: unlike Compose and Kubernetes, where you can
point at shipped files, here you write everything. The good news is that there
is very little to write: the binary takes one positional argument and reads
nothing else from its command line.

If what you actually want is a cache for a handful of people rather than a
production deployment, [Shared cache](/getting-started/shared-cache) is the
shorter road and ends somewhere useful.

## The configuration [#the-configuration]

Three units, one per process. They are near-identical; the differences are the
config file and, for the worker, the security posture.

### The service account and directories [#the-service-account-and-directories]

```sh
sudo useradd --system --home-dir /var/lib/nativelink --shell /usr/sbin/nologin nativelink
sudo install -d -o nativelink -g nativelink /var/lib/nativelink
sudo install -d -o nativelink -g nativelink /etc/nativelink
sudo install -m 0755 nativelink /usr/local/bin/nativelink
```

Put the configuration files in `/etc/nativelink/{cas,scheduler,worker}.json5`.

### The CAS unit [#the-cas-unit]

```ini
# /etc/systemd/system/nativelink-cas.service
[Unit]
Description=NativeLink CAS and Action Cache
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/cas.json5
Restart=always
RestartSec=2s

# The binary raises its own nofile limit at startup, but only up to the hard
# limit systemd gives it. See "File descriptors" below.
LimitNOFILE=65536

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

# The CAS touches only its own data directory.
StateDirectory=nativelink
ReadWritePaths=/var/lib/nativelink
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
```

### The scheduler unit [#the-scheduler-unit]

Identical apart from the config file and the fact that it needs no writable
state at all when the scheduler backend is the default in-memory one:

```ini
# /etc/systemd/system/nativelink-scheduler.service
[Unit]
Description=NativeLink scheduler
After=network-online.target nativelink-cas.service
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/scheduler.json5
Restart=always
RestartSec=2s
LimitNOFILE=65536

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target
```

### The worker unit [#the-worker-unit]

This one is different, and the differences matter.

```ini
# /etc/systemd/system/nativelink-worker.service
[Unit]
Description=NativeLink worker
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/worker.json5
Restart=always
RestartSec=2s

# Workers open far more files than the other two processes: every input in
# every action tree, plus the CAS entries backing them.
LimitNOFILE=131072

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

# Actions run as this user, in this unit's cgroup. Everything the unit can
# reach, an action can reach.
ReadWritePaths=/var/lib/nativelink
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

# Bound the blast radius of an action that allocates without limit.
MemoryMax=48G
MemoryAccounting=true

# On SIGTERM the worker stops taking new actions and waits for in-flight
# ones to finish. Set this longer than your longest action (the worker's
# max_action_timeout_s defaults to 20 minutes), or systemd SIGKILLs it
# mid-action.
KillMode=mixed
TimeoutStopSec=25min

[Install]
WantedBy=multi-user.target
```

<Callout type="warn" title="NoNewPrivileges is absent from the worker unit on purpose">

  If your worker configuration sets `use_namespaces: true`, the process calls
  `unshare(CLONE_NEWUSER | CLONE_NEWPID | ...)`, which `NoNewPrivileges=true`
  can block depending on kernel and distribution policy. If you are **not**
  using namespaces, add `NoNewPrivileges=true` back; it costs nothing.

  Either way, understand what you are relying on: NativeLink's namespace
  support is a hermeticity and process-reaping mechanism, not a sandbox.
  Actions run as the worker's own uid, see the whole filesystem the unit can
  see, and can reach every NativeLink port on the network.
  [Security hardening](/operate/security-hardening#the-sandbox-is-not-a-security-boundary)
  is the full picture, and it is the reason `MemoryMax` and `ProtectSystem`
  are in this unit rather than left to chance.

</Callout>

Then:

```sh
sudo systemctl daemon-reload
sudo systemctl enable --now nativelink-cas nativelink-scheduler nativelink-worker
```

## What to change [#what-to-change]

| Setting | Where | Why you'd change it |
| --- | --- | --- |
| `LimitNOFILE` | unit | NativeLink raises its own soft limit at startup toward `max_open_files` (default 24,576) but cannot exceed the hard limit systemd grants. Set the unit's limit **above** the config value, or the config value is capped (the startup log line `set_open_file_limit() assigns new open file limit N` shows what it got). |
| `max_open_files` | `global` in the config | The binary reserves 80% of the *achieved* limit as a concurrency semaphore. Exhausting it does not error; it blocks. |
| `MemoryMax` | worker unit | NativeLink's configuration has no memory limit anywhere. The cgroup is the only bound that exists. |
| `RestartSec` | worker unit | The worker's own scheduler-reconnect retry is a hard-coded 0.5 s with no backoff. A restart loop plus that retry is a lot of connection attempts; keep `RestartSec` at seconds, not milliseconds. |
| `TimeoutStopSec` | worker unit | On `SIGTERM` the worker rejects new actions, waits for in-flight ones to finish, then tells the scheduler it is going away and exits. If this timer fires first, systemd `SIGKILL`s it and every in-flight action fails and re-queues. |
| `NL_LOG` | environment | `json` if anything is ingesting these logs. Any unrecognised value silently falls back to `pretty`; see [CLI and environment](/reference/cli-and-env). |

## Disk layout [#disk-layout]

The CAS host is the one where placement matters.

| Path | Requirement |
| --- | --- |
| `content_path` | The bulk of the data. On the CAS host, provision `max_bytes` plus headroom. On a worker host, provision **twice** the fast tier's `max_bytes`: the worker creates an executable-variant directory next to `content_path` that is invisible to the eviction accounting, so that store can reach roughly `2 × max_bytes` in the worst case. |
| `temp_path` | Must be on the **same block device** as `content_path`. Otherwise the atomic rename that completes every write degrades to a copy: slower, and it doubles peak space during writes. Everything here is deleted on every startup. |
| Worker `work_directory` | Scratch. Purged at worker startup, which means **two workers must never share one.** A second worker starting will delete the first worker's in-flight action trees. |

<Callout type="warn" title="Eviction never looks at free space">
  NativeLink evicts on the size it has accounted for, not on the size the
  filesystem reports, and it evicts only when the store is mutated, never in
  response to a failed write. An `eviction_policy` is optional and its
  `max_bytes` defaults to zero, which means *never evict*. Leaving it unset on
  a `filesystem` store is the single most common way to fill a disk.
  [Runbooks](/operate/runbooks#the-disk-fills-up) has the recovery.
</Callout>

## Host sizing [#host-sizing]

No sizing numbers are published for NativeLink; measure your own workload
before buying anything. What the code does fix:

| Resource | Starting point | Notes |
| --- | --- | --- |
| CAS storage | The `max_bytes` you set, plus headroom | It is a cache of the working set, not an archive; the eviction policy is the only bound. |
| Worker vCPU | 1 per concurrent action | Set `max_inflight_tasks` to match; it defaults to **0, meaning unlimited**. |
| Worker memory | The sum of your largest concurrent actions | The configuration has no per-action memory limit; `MemoryMax` on the unit is the only bound. |
| Scheduler | Small | It holds queue state in memory and owns no storage. |

CAS storage and I/O dominate the bill; the scheduler is the cheapest process in
the system. If you are choosing where to spend, spend on CAS I/O.

## Restarts and upgrades [#restarts-and-upgrades]

The three processes have genuinely different restart characteristics, and
treating them the same is how a routine upgrade turns into a stack of failed
builds.

**The scheduler** is the disruptive one. When it goes away, every worker
notices within seconds and reconnects at a flat 0.5-second interval with no
backoff and no jitter; a fleet of 200 workers produces up to 400 connection
attempts per second at the socket for the whole outage. Worse, each worker's
reconnect path calls `kill_all()` first, destroying every action it was
running. Restart the scheduler when the queue is quiet, and restart it once.

**Workers** drain themselves on `SIGTERM`: the process rejects new actions,
waits for the ones it is running to finish, sends the scheduler a
`GoingAway`, and exits. `systemctl stop` is therefore safe as long as
`TimeoutStopSec` outlasts your longest action. To take a worker out of
rotation without stopping it, and if your scheduler configuration enables the
`admin` service on the worker API listener:

```sh
curl -X POST \
  "http://127.0.0.1:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/${WORKER_ID}/1"
```

The path segment after `scheduler/` is the scheduler's `name` from the config,
not a REAPI instance name. The worker finishes what it has and takes nothing
new. What loses work is a `SIGKILL` (the stop timer firing, or the OOM
killer): every in-flight action fails and re-queues, and in this tree a
disconnect **consumes a retry attempt**; enough of them in a row and the action
fails outright with a message naming OOM-kill or eviction as the likely cause.
Note that the admin endpoint is unauthenticated;
[Security hardening](/operate/security-hardening#the-admin-endpoint-has-no-authentication)
explains why it must not be routable.

**The CAS** is the safe one. It holds no scheduling state; a restart is a
window of `UNAVAILABLE` responses and nothing more. Its `temp_path` is pruned
on the way back up, so a crash mid-write reclaims itself.

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="CONNECTION_RETRY_DELAY_S" />
<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="GoingAwayRequest" />
<SourceLink file="nativelink-scheduler/src/simple_scheduler_state_manager.rs" symbol="UpdateWithDisconnect" />
<SourceLink file="nativelink-util/src/fs.rs" symbol="set_open_file_limit" />
<SourceLink file="src/bin/nativelink.rs" symbol="set_drain_worker" />

<VerifyBlock>

  `systemctl is-active nativelink-cas nativelink-scheduler nativelink-worker`
  prints `active` three times.

  `journalctl -u nativelink-worker | grep 'Worker registered with scheduler'`
  finds a line. If instead you find `Could not connect to endpoint` repeating
  about twice a second, the worker cannot reach the scheduler's worker API
  port.

  `journalctl -u nativelink-cas | grep 'open file limit'` shows the limit the
  process actually achieved, not the one you asked for. A `warn` about the
  limit being below the recommended value means `LimitNOFILE` is too low.

  A build run twice against the cache reports hits on the second run.

</VerifyBlock>

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| Unit restarts in a loop, journal says `unknown field` | Config is parsed with `deny_unknown_fields`; one unrecognised key is fatal | Check the key in the [configuration reference](/reference/nativelink-config) |
| Process exits during startup, no error about ports | `NL_OTEL_ENDPOINT` without a scheme or explicit port panics before binding | Use `http://host:4317`; see [CLI and environment](/reference/cli-and-env) |
| Everything slows down under load, nothing errors | File-descriptor semaphore exhaustion blocks rather than failing | Raise `LimitNOFILE` **and** `max_open_files` together |
| Two workers on one host behave erratically | They share a `work_directory`, which is purged at startup | Give each worker its own |
| Disk full, then every request fails and keeps retrying | ENOSPC surfaces as a generic error code that clients and upstream stores retry, and eviction is not triggered by write failures | [Runbooks](/operate/runbooks#the-disk-fills-up) |
| Actions fail with exit code 9 and no error message | The kernel OOM-killer took the action's child process | [Runbooks](/operate/runbooks#a-worker-is-oom-killed) |
| A rolling restart produces a burst of failed builds | Workers were `SIGKILL`ed before their in-flight actions finished (`TimeoutStopSec` too short) | Raise `TimeoutStopSec`, or drain via the admin endpoint before stopping |

## What's next [#whats-next]

<NextStep href="/operate/runbooks" title="Runbooks">
  The four incidents that actually page you, each with the log line that
  identifies it and the sequence that resolves it.
</NextStep>

<NextStep href="/operate/security-hardening" title="Harden it" kind="aside">
  Port exposure, TLS, and why the worker API must never be routable.
</NextStep>

<NextStep href="/operate/scaling-workers" title="Add more workers" kind="aside">
  Sizing one worker, then deciding how many.
</NextStep>

# https://docs.nativelink.com/operate/observability

**Observability**: NativeLink pushes OTLP and nothing else. Wire it to a collector, get the series into Prometheus under the names the shipped rules expect, and know which environment variables actually do something.

**Who this is for:** you are running NativeLink for other people and need to
see queue times, cache hit rates and worker behaviour on a dashboard.
**What you'll have at the end:** a collector receiving OTLP from every
NativeLink process, Prometheus holding the series under the names the shipped
recording rules query, and a Grafana dashboard drawing them. **Time:** 30
minutes.

<Prerequisites>
  A running deployment you can set environment variables on.
</Prerequisites>

## NativeLink pushes; it is never scraped [#nativelink-pushes-it-is-never-scraped]

The binary has no `/metrics` endpoint and no configuration that creates
one. NativeLink builds three OTLP exporters at startup (logs, traces and
metrics) and pushes to whatever endpoint the environment names. Every
Prometheus-shaped thing in this pipeline happens downstream of the process.

<Mermaid>
{`flowchart LR
    NL["nativelink<br/>CAS · scheduler · worker"]
    COL["OTel Collector<br/>otlp receiver :4317"]
    PEX["prometheus exporter<br/>:9090 · namespace: nativelink"]
    PROM["Prometheus<br/>published on :9091"]
    GRAF["Grafana :3000"]

    NL -->|"OTLP/gRPC push<br/>every 60 s"| COL
    COL --> PEX
    PEX -->|"Prometheus scrapes"| PROM
    PROM --> GRAF`}
</Mermaid>

That `namespace: nativelink` on the collector's Prometheus exporter is
load-bearing, and it is the single most misunderstood part of this pipeline.
The instruments NativeLink declares are named `cache.operations`,
`execution.queue.time` and so on, with **no `nativelink` prefix anywhere in the
binary**. The prefix is added by the collector. Bypass the collector and you
lose it, along with every shipped recording rule and dashboard panel. See
[what happens if you skip the collector](#what-happens-if-you-skip-the-collector).

<Callout type="info" title="Metrics are Business Source Licensed">
  Individual developer cache use does not need a commercial licence. Teams
  using metrics in shared, production or commercial settings can use NativeLink
  Enterprise or a separate licence. Note that instrumentation cannot be
  switched off (see
  [Open source and Enterprise](/reference/oss-and-enterprise#metrics-are-always-on)), and
  the operative text is the [licence page](https://nativelink.com/license).
</Callout>

## The configuration [#the-configuration]

Point every NativeLink process at a collector. Two variables carry the whole
decision:

```sh
export NL_OTEL_ENDPOINT=http://otel-collector:4317
export RUST_LOG=info
```

<Callout type="warn" title="NL_OTEL_ENDPOINT needs a scheme and an explicit port">

  The value is parsed as a URL and the host and port are pulled out
  individually, and each step `unwrap`s. A value with no scheme, or with no
  explicit port, yields `None` for the port and **panics the process during
  startup**, before it binds a socket.

  `http://otel-collector:4317` is correct. `otel-collector:4317` and
  `http://otel-collector` both crash the binary.

</Callout>

`NL_OTEL_ENDPOINT` builds one client-side load-balanced gRPC channel shared by
all three exporters. Leave it unset and each exporter falls back to the
OpenTelemetry SDK's own resolution: `OTEL_EXPORTER_OTLP_ENDPOINT` (or the
per-signal `OTEL_EXPORTER_OTLP_METRICS_ENDPOINT` and friends), defaulting
to `http://localhost:4317`. Both work; the explicit channel is what you want in
a cluster, because it re-resolves the collector's DNS as pods move. Either way
the transport is OTLP over gRPC: the binary is built with only the gRPC
exporter, so `OTEL_EXPORTER_OTLP_PROTOCOL` cannot switch it to HTTP.

The rest of the surface is small enough to state in full:

| Variable | Default | What it does here |
| --- | --- | --- |
| `NL_OTEL_ENDPOINT` | unset | One shared gRPC channel for logs, traces and metrics. Panics on a malformed value. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | Used only when `NL_OTEL_ENDPOINT` is unset. |
| `OTEL_EXPORTER_OTLP_COMPRESSION` | unset | `gzip` or `zstd` on the gRPC exporters; the shipped Kubernetes manifests set `zstd`. |
| `OTEL_METRIC_EXPORT_INTERVAL` | `60000` (ms) | How often metrics flush. NativeLink sets no interval, so the SDK default of one minute applies. This is the first hop in the [reaction-time budget](/operate/autoscaling-reference#the-reaction-time-budget); lower it to 10000 before you scale on anything. |
| `RUST_LOG` | `info` | Applies to the stdout layer **and** to all three OTLP layers. `hyper`, `tonic`, `h2`, `reqwest` and `tower` are forced off afterwards and cannot be re-enabled. |
| `NL_LOG` | `pretty` | Stdout log format. `compact` and `json` are recognised; any other value silently falls back to `pretty`. |

The complete list, including the variables the config file expands and the ones
a worker forwards to actions, is in
[CLI and environment](/reference/cli-and-env).

<Callout type="warn" title="OTEL_SERVICE_NAME does nothing">

  `Resource::builder().with_service_name("nativelink")` is called
  unconditionally, so the service name is a compile-time constant. Setting
  `OTEL_SERVICE_NAME`, as several examples in the wild do, changes nothing.

  `service.instance.id` is a fresh UUIDv4 per process, so it changes on every
  restart. It is useful for telling replicas apart within one incident and
  useless as a stable series identity across restarts. Add your own stable
  identity through `OTEL_RESOURCE_ATTRIBUTES` if you need one.

</Callout>

<SourceLink file="nativelink-util/src/telemetry.rs" symbol="init_tracing" />

## Bring up the pipeline [#bring-up-the-pipeline]

<Tabs items={["Docker Compose", "Kubernetes", "Direct to Prometheus"]}>

  <Tab value="Docker Compose">

    ```sh
    git clone https://github.com/TraceMachina/nativelink
    cd nativelink/deployment-examples/metrics
    docker compose up -d
    ```

    Then point NativeLink at the collector and restart it:

    ```sh
    export NL_OTEL_ENDPOINT=http://localhost:4317
    export OTEL_METRIC_EXPORT_INTERVAL=10000
    nativelink /path/to/config.json5
    ```

    What comes up, and on which host port:

    | Service | Host port | Notes |
    | --- | --- | --- |
    | OTel Collector, OTLP gRPC | `4317` | Where NativeLink pushes. |
    | OTel Collector, OTLP HTTP | `4318` | Unused by NativeLink's gRPC exporters. |
    | OTel Collector, Prometheus exporter | `9090` | **This is not Prometheus.** It is the collector's scrape target. |
    | OTel Collector, own metrics | `8888` | `otelcol_*` series about the collector itself. |
    | OTel Collector, health | `13133` | `GET /health`. |
    | Prometheus UI | **`9091`** | Published as `9091:9090` to avoid colliding with the exporter above. |
    | Grafana | `3000` | `admin` / `admin`. |
    | Alertmanager | `9093` | Optional. |
    | Jaeger UI | `16686` | Traces; the collector's traces pipeline forwards to it. |
    | node-exporter | `9100` | Optional host metrics. |

    <Callout type="warn" title="Prometheus is on 9091, not 9090">
      Port 9090 belongs to the collector's Prometheus *exporter*, the thing
      Prometheus scrapes. Opening `http://localhost:9090` gets you a page of raw
      OpenMetrics text, not a query UI. The UI is `http://localhost:9091`.
    </Callout>

    The shipped collector config also runs a second metrics pipeline that
    pushes the same stream into Prometheus's own OTLP receiver over HTTP
    (`otlphttp/prometheus`). Those series arrive without the `nativelink_`
    prefix, so do not be surprised to find both spellings in the same
    Prometheus; the rules and dashboard use the prefixed one.

  </Tab>

  <Tab value="Kubernetes">

    ```sh
    kubectl apply -f deployment-examples/metrics/kubernetes/prometheus.yaml
    kubectl apply -f deployment-examples/metrics/kubernetes/otel-collector.yaml
    ```

    Both manifests live in a `nativelink` namespace; `prometheus.yaml` is the
    one that creates it, so apply it first. The collector's `ConfigMap`
    carries the same metrics pipelines as the Compose file (no traces
    pipeline), with a 1 GiB memory limiter instead of 512 MiB.

    Then add the endpoint to every NativeLink container, CAS, scheduler and
    worker alike:

    ```yaml
    env:
      - name: NL_OTEL_ENDPOINT
        value: "http://otel-collector:4317"
      - name: OTEL_METRIC_EXPORT_INTERVAL
        value: "10000"
      - name: OTEL_RESOURCE_ATTRIBUTES
        value: "deployment.environment=prod,k8s.cluster.name=main"
    ```

    On the scrape path, the collector's Prometheus exporter has
    `resource_to_telemetry_conversion` enabled, so every resource attribute,
    including anything you set in `OTEL_RESOURCE_ATTRIBUTES`, becomes a label
    (`deployment_environment`, `k8s_cluster_name`, `service_instance_id`).
    The `promote_resource_attributes` list in the shipped Prometheus config
    (`k8s.pod.name`, `k8s.namespace.name`, `k8s.deployment.name` and a dozen
    more) only governs the collector's second pipeline into Prometheus's own
    OTLP receiver; on that path attributes not on the list do not become
    labels on the series (they survive only on `target_info`).

  </Tab>

  <Tab value="Direct to Prometheus">

    Prometheus can receive OTLP itself (`--web.enable-otlp-receiver`), but its
    receiver is OTLP over HTTP at `/api/v1/otlp/v1/metrics`, and NativeLink's
    exporters are built with the gRPC transport only. Setting
    `OTEL_EXPORTER_OTLP_PROTOCOL=http/protobuf` does nothing, so there is no
    working configuration that sends the binary's metrics straight into
    Prometheus. Something that speaks OTLP/gRPC has to sit in between; the
    shipped collector is that something, and its `otlphttp/prometheus`
    exporter is exactly this hop.

    Read [what happens if you skip the
    collector](#what-happens-if-you-skip-the-collector) before pointing the
    binary at any OTLP/gRPC receiver other than the shipped collector: it
    renames every metric.

  </Tab>

</Tabs>

## What happens if you skip the collector [#what-happens-if-you-skip-the-collector]

The `namespace: nativelink` setting on the collector's Prometheus exporter is
what produces every metric name you have ever seen in a NativeLink dashboard.
Send the OTLP stream anywhere else (another collector without that setting,
an OTLP-native backend, or the shipped collector's own `otlphttp/prometheus`
pipeline into Prometheus's receiver) and the names change:

| Path | Series name for the `cache.operations` counter |
| --- | --- |
| NativeLink → collector → Prometheus scrapes the `prometheus` exporter | `nativelink_cache_operations_total` |
| NativeLink → OTLP receiver that adds no namespace | `cache_operations_total` (or `cache.operations_total`, depending on the receiver's name translation) |

Everything shipped under `deployment-examples/metrics/` (the recording rules,
the Grafana dashboard, the alert expressions) is written against the first
spelling. Against the second they all evaluate to an empty vector, which on a
dashboard renders as a flat healthy zero rather than as a missing signal.

If you take the un-prefixed path anyway, either rewrite the rules without the
prefix or add a `metric_relabel_config` that puts it back. The collector path
is the one the repository supports.

<Callout type="info" title="Counters gain _total, histograms fan out">
  The exporter also rewrites OTLP names into Prometheus form: dots become
  underscores, monotonic counters gain a `_total` suffix, and each histogram
  becomes `_bucket`, `_sum` and `_count` series. The
  [metrics reference](/reference/metrics#how-an-instrument-becomes-a-prometheus-series)
  records the exact result for every instrument.
</Callout>

## Turn on cache metrics [#turn-on-cache-metrics]

Every family except cache metrics is emitted with no configuration. Cache
metrics exist only for stores you explicitly wrap:

```json5
{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: {
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-cas",
        temp_path: "/var/lib/nativelink/tmp_path-cas",
      },
    },
  },
}
```

With `cache_metrics` absent, NativeLink constructs the same store graph without
a wrapper, a timer, an attribute allocation or an OpenTelemetry call on the hot
path. The feature costs nothing when it is off, which is why it is off.

Wrap the stores you actually want to measure, usually `CAS_MAIN_STORE` and the
AC. Wrapping every layer of a `fast_slow` composition gives you one series per
layer and a hit-rate query that no longer means what you think it means.

## Which metrics exist [#which-metrics-exist]

The authority is the [metrics reference](/reference/metrics), which is
generated from `nativelink-util/src/metrics.rs` and its call sites. It records
every declared instrument, its type and unit, its histogram buckets, and (the
part no hand-written table stays honest about) **whether anything in the
binary ever emits it**.

As of v1.6.5 the 30 declared instruments fall into eight families. Which process emits
each one decides where you have to point the collector:

| Family | Series prefix | Emitted by | What it tells you |
| --- | --- | --- | --- |
| Cache | `nativelink_cache_*` | Any process with a store wrapped in `cache_metrics` | Hit rate, operation latency, bytes in and out, per cache type |
| Execution | `nativelink_execution_*` | The scheduler | Queue time, stage durations, active and completed counts, retries, and per-action CPU time and peak memory as reported by the worker that ran it |
| Worker fleet | `nativelink_worker_*` | The scheduler | Workers connected, joins and departures by reason, keepalives, and how many are paused or draining |
| Scheduler | `nativelink_scheduler_matching_*` | The scheduler | How long each matching pass takes and whether it succeeded |
| gRPC server | `nativelink_rpc_server_duration_*` | Every process, for every gRPC service it serves | Request latency by service, method and status code |
| Store tier | `nativelink_store_tier_*` | Any process with a `fast_slow` store | Fast and slow tier hits, misses and stale reads, and bytes read from each tier |
| Health | `nativelink_health_checks_*` | Every process, on each run of the health endpoint | Health check outcomes per component: `ok`, `initializing`, `warning`, `failed` or `timeout` |
| Connection pool | `nativelink_connection_*` | Any process with a gRPC connection pool or a Redis store | Pool headroom, acquisitions that had to queue, and reconnects |

Three things worth knowing before you build a dashboard:

`nativelink_cache_size` and `nativelink_cache_entries` are declared and
[never emitted](/reference/metrics#declared-but-never-emitted). The shipped
recording rules and the metrics README query both. Those rules have always
returned nothing.

Execution and worker fleet metrics come from the scheduler process only, and
they describe the fleet as that scheduler instance sees it. With several
scheduler replicas, aggregate across instances before reading a fleet total.

The shipped recording rules and dashboard predate the worker, gRPC, scheduler,
store tier, health and connection pool families. Nothing under
`deployment-examples/metrics/` queries them yet, so the queries below are the
starting point.

## Queries worth having [#queries-worth-having]

Cache hit rate, per cache type:

```text
sum(rate(nativelink_cache_operations_total{cache_operation_result="hit"}[5m])) by (cache_type) /
sum(rate(nativelink_cache_operations_total{cache_operation_name="read"}[5m])) by (cache_type)
```

How long actions wait before a worker picks them up, the number that tells you
whether to add workers:

```text
histogram_quantile(0.95,
  sum(rate(nativelink_execution_queue_time_bucket[5m])) by (le, instance_name)
)
```

Actions currently queued, which is the closest thing to a queue depth:

```text
sum(nativelink_execution_active_count{execution_stage="queued"})
```

Success rate:

```text
sum(rate(nativelink_execution_completed_count_total{execution_result="success"}[5m])) /
sum(rate(nativelink_execution_completed_count_total[5m]))
```

Workers connected to each scheduler instance, and how many of them are paused
or draining:

```text
sum(nativelink_worker_connected_count) by (service_instance_id)
sum(nativelink_worker_state_count) by (worker_state)
```

Why workers are leaving, which separates a scaling event from a fleet that is
being evicted for timeouts:

```text
sum(rate(nativelink_worker_disconnections_total[5m])) by (worker_disconnect_reason)
```

Matching pass latency, which is the scheduler's own cost per pass rather than
how long actions wait:

```text
histogram_quantile(0.95, sum(rate(nativelink_scheduler_matching_duration_bucket[5m])) by (le))
```

gRPC error rate per method, across every process:

```text
sum(rate(nativelink_rpc_server_duration_count{rpc_grpc_status_code!="0"}[5m])) by (rpc_service, rpc_method) /
sum(rate(nativelink_rpc_server_duration_count[5m])) by (rpc_service, rpc_method)
```

Fast tier hit rate of a `fast_slow` store, which is what tells you whether the
local disk in front of a bucket is earning its keep:

```text
sum(rate(nativelink_store_tier_operations_total{store_tier="fast",store_result="hit"}[5m])) /
sum(rate(nativelink_store_tier_operations_total{store_tier="fast"}[5m]))
```

Peak memory per action at the 95th percentile, which is the number to size
worker pods from (the scheduler records this sample with an empty
`execution_instance`, so there is nothing useful to group by):

```text
histogram_quantile(0.95, sum(rate(nativelink_execution_peak_memory_bucket[5m])) by (le))
```

The repository ships 34 more as recording rules in
`prometheus-recording-rules.yml`, grouped into `nativelink_execution`,
`nativelink_cache`, `nativelink_performance` and `nativelink_slo`. Loading that
file gives you `nativelink:execution_queue_time_p95` and friends as
pre-computed series. The shipped dashboard does not query them; its panels
read the raw `nativelink_execution_*` series directly.

## The Grafana dashboard [#the-grafana-dashboard]

A reference dashboard lives at
<SourceLink file="deployment-examples/metrics/grafana/dashboards/nativelink-overview.json" />.
The Compose stack provisions it automatically. Elsewhere: Dashboards → New →
Import, paste the JSON, pick your Prometheus data source. It draws execution
throughput, success rate, active actions by stage and stage transitions from
`nativelink_execution_*`; it has no cache, worker or gRPC panels.

If you add panels bound to `nativelink_cache_size` or
`nativelink_cache_entries`, they will be empty forever. That is the metric,
not your pipeline.

<SourceLink file="nativelink-util/src/telemetry.rs" symbol="init_tracing" />
<SourceLink file="nativelink-util/src/metrics.rs" />
<SourceLink file="deployment-examples/metrics/otel-collector-config.yaml" />

<VerifyBlock>

  `curl http://localhost:13133/health` returns healthy. The collector's
  pipeline check is on, so this also fails when its exporters are failing.

  `curl -s http://localhost:8888/metrics | grep otelcol_receiver_accepted_metric_points`
  is non-zero and climbing. If it is zero, NativeLink is not reaching the
  collector.

  `curl -s http://localhost:9090/metrics | grep nativelink_execution` shows
  series on the collector's exporter, with the `nativelink_` prefix present.

  In Prometheus at `http://localhost:9091`, `nativelink_execution_queue_time_bucket`
  returns data after one build.

  If the `nativelink_cache_*` series are missing but `nativelink_execution_*`
  are present, no store is wrapped in `cache_metrics`.

</VerifyBlock>

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| Process panics at startup with a URL error | `NL_OTEL_ENDPOINT` has no scheme or no explicit port | Use `http://host:4317` |
| No metrics anywhere, process healthy | Nothing is pushing; check `otelcol_receiver_accepted_metric_points` on `:8888` | Verify the endpoint and that it is set on the process you are looking for |
| Metrics appear a minute late | `OTEL_METRIC_EXPORT_INTERVAL` defaults to 60 s | Set `10000` |
| `nativelink_execution_*` present, `nativelink_cache_*` missing | No store is wrapped | [Turn on cache metrics](#turn-on-cache-metrics) |
| Metrics exist under names with no `nativelink_` prefix | They came through an OTLP receiver that adds no namespace (the shipped collector's `otlphttp/prometheus` pipeline does this alongside the scrape path) | [Skip the collector](#what-happens-if-you-skip-the-collector) |
| No execution or worker metrics, cache metrics fine | The collector endpoint is set on the CAS but not on the scheduler process | Set `NL_OTEL_ENDPOINT` on the scheduler; those families are emitted from it alone |
| `nativelink_worker_*` or `nativelink_rpc_*` series missing entirely | The binary is older than the release these families were added in | Upgrade to v1.6.5; the [metrics reference](/reference/metrics#every-instrument-at-a-glance) is generated from the version it names |
| Cache-size panels flat at zero | Those instruments are never emitted | [Declared but never emitted](/reference/metrics#declared-but-never-emitted) |
| `out of order sample` in Prometheus logs | Multiple processes with restart-fresh `service.instance.id` | Raise `storage.tsdb.out_of_order_time_window` in `prometheus.yml` (the shipped Kubernetes config sets 30m) |
| Metrics stop when a worker OOMs and returns | Expected; the new process gets a new `service.instance.id` | Aggregate away from `service_instance_id` in your queries |

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Can Prometheus scrape NativeLink directly?" id="can-prometheus-scrape-nativelink-directly">
    No. The binary has no scrape endpoint, at any configuration, and
    the binary cannot push to Prometheus's OTLP receiver either, because that
    receiver is HTTP-only and NativeLink's exporters are gRPC-only. Run a
    collector and scrape its Prometheus exporter; the shipped collector also
    forwards into Prometheus's OTLP receiver, with the renaming described in
    [what happens if you skip the collector](#what-happens-if-you-skip-the-collector).
  </Accordion>
  <Accordion title="Do metrics require a commercial licence?" id="do-metrics-require-a-commercial-licence">
    Metrics are one of the Business Source Licensed modules. Individual
    developer use is fine; shared, production or commercial settings need
    NativeLink Enterprise or a separate licence; see the
    [licence page](https://nativelink.com/license). No configuration
    turns instrumentation off, which
    [Open source and Enterprise](/reference/oss-and-enterprise#metrics-are-always-on)
    explains and which matters when you assess your own position.
  </Accordion>
  <Accordion title="Which metric should drive worker autoscaling?" id="which-metric-should-drive-worker-autoscaling">
    Queue time, not queue depth. `nativelink_execution_queue_time` is a real
    histogram emitted by the scheduler; a p95 above your tolerance means add
    workers. No `nativelink_scheduler_queue_depth` series exists; the
    closest thing is `nativelink_execution_active_count{execution_stage="queued"}`,
    a gauge, and `nativelink_worker_connected_count` tells you how many workers
    that queue is waiting on. Before you build an HPA on either, read the
    [autoscaling reference](/operate/autoscaling-reference#the-reaction-time-budget):
    the default one-minute export interval alone puts your reaction time in
    minutes, and CPU-based scaling is worse because workers are mostly I/O-bound.
  </Accordion>
  <Accordion title="Why does my dashboard show a new instance every restart?" id="why-does-my-dashboard-show-a-new-instance-every-restart">
    `service.instance.id` is a fresh UUIDv4 per process and `OTEL_SERVICE_NAME`
    is ignored. Set a stable identity of your own through
    `OTEL_RESOURCE_ATTRIBUTES`, for example `nativelink.instance_name`, and
    aggregate on that. The shipped collector config copies that attribute onto
    every datapoint as `instance_name`, which is the label the shipped
    recording rules group by; nothing in the binary sets it for you.
  </Accordion>
</Accordions>

## What's next [#whats-next]

<NextStep href="/reference/metrics" title="The metrics reference">
  Every instrument, its buckets, its attributes, and whether anything emits it,
  generated from the source.
</NextStep>

<NextStep href="/operate/autoscaling-reference" title="Scale on these signals" kind="aside">
  The reaction-time budget, and the five gaps between a metric and a working
  HPA.
</NextStep>

<NextStep href="/operate/runbooks" title="What to do when a signal fires" kind="aside">
  The incidents that page you, and the sequence that resolves each.
</NextStep>

# https://docs.nativelink.com/operate/security-hardening

**Security hardening**: Open-source NativeLink has no inbound authentication on any service. What that means for your network, which ports must never be routable, how to configure mTLS, and why the worker sandbox is not a security boundary.

**Who this is for:** you are about to put NativeLink somewhere more than one
person can reach it. **What you'll have at the end:** a network layout where
the dangerous ports are unreachable, mTLS on the ports that must be reachable,
and an accurate picture of the trust you are extending to anyone who can submit
an action. **Time:** 45 minutes, plus however long your certificate authority
takes.

<Prerequisites>
  A deployment reachable by something other than your laptop.
</Prerequisites>

## Start here: there is no authentication [#start-here-there-is-no-authentication]

Open-source NativeLink authenticates no inbound request, on any service, in any
deployment. Not the CAS. Not the Action Cache. Not the Execution service. Not
the worker API. Not the admin endpoint. No bearer-token check, no API
key, no mTLS-identity-to-permission mapping; the code that would do it does
not exist in the tree.

This is not a gap you close in the configuration file. The one server option
that looks like it might, `experimental_identity_header` with `required: true`,
only rejects requests that arrive without an OpenTelemetry `Baggage:
enduser.id=...` header; the value is a telemetry label and is never checked
against anything. **Your network is the authorization layer.** Everything below
follows from that one fact.

<SourceLink file="nativelink-util/src/telemetry.rs" symbol="OtlpLayer" />

<Mermaid>
{`flowchart TD
    subgraph PUB["Reachable by developers and CI"]
      CAS["CAS / AC / ByteStream / Capabilities<br/>:50051 plain · :50071 TLS"]
      SCH["Execution frontend<br/>:50052"]
    end

    subgraph PRIV["Must never be routable from outside"]
      WAPI["worker API<br/>:50061"]
      ADM["admin REST<br/>/admin/*"]
      HLTH["health<br/>/status"]
    end

    subgraph W["Worker hosts: same blast radius as an action"]
      WK["nativelink worker"]
    end

    DEV["developer / CI"] -->|"mTLS"| CAS
    DEV -->|"mTLS"| SCH
    WK -->|"registers, no authn"| WAPI
    WK -->|"reads and writes blobs"| CAS
    ADM -.->|"drains workers, no authn"| WAPI`}
</Mermaid>

## The port inventory [#the-port-inventory]

Every listener NativeLink opens, what it serves, and who may reach it. Ports
are the ones the shipped examples use; nothing in the binary reserves them;
[Production configuration](/operate/production-config) covers how the three
processes divide these listeners between them.

| Port | Services on it | Who should reach it |
| --- | --- | --- |
| `50051` | CAS, Action Cache, ByteStream, Capabilities | Developers and CI. Everything on it is a cache read or write. |
| `50052` | Execution, Capabilities, AC frontend | Developers and CI. Submitting an action means running code on a worker. |
| `50061` | **worker API**, `admin`, `health` | Workers and your operators. **Never routable from a developer network.** |
| `50071` | The TLS variant of `50051` in the shipped examples | Developers and CI. |

The configuration crate says this itself, in the doc comment on the field:

> This is the service used for workers to connect and communicate through.
> NOTE: This service should be served on a different, non-public port. In other
> words, `worker_api` configuration should not have any other services that are
> served on the same port. Doing so is a security risk, as workers have a
> different permission set than a client that makes the remote execution/cache
> requests.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="ServicesConfig::worker_api" />

## The worker API has no authentication [#the-worker-api-has-no-authentication]

Port 50061 is the one that turns a cache into a compromise. A process that can
open a gRPC connection to it can call `ConnectWorker`, be added to the
scheduler's pool, and start receiving actions, including the compiler
invocations, the secrets in your action environments, and the outputs that get
written back into the CAS under digests your developers will later trust.

No handshake exists beyond the protocol's own. The scheduler does not check
who is connecting, because there is nothing to check with.

Three consequences, in order of how often they bite:

An attacker who can reach 50061 can **register a malicious worker** and be
handed real actions. Whatever those actions can see, they can see.

An attacker who can reach 50061 can **poison the cache**, because a worker
uploads action results and the CAS accepts them. A poisoned result for a
widely-cached action propagates to every developer who gets a cache hit on it.

An attacker who can reach 50061 can **denial-of-service the scheduler** by
registering and disconnecting, since each disconnect requeues that worker's
in-flight actions and, in this tree, consumes one of their retries.

What to do:

Bind it to an address only workers can reach. The single-host Compose file gets
this right (the scheduler's 50061 is not published to the host at all; workers
reach it over the Compose network) and
[the multi-worker variant](/operate/deploy-docker-compose#the-multi-worker-variant)
does not; it publishes `50061:50061` on all interfaces. Fix that before running
it anywhere routable.

On Kubernetes, keep 50061 off any `GRPCRoute` or `Ingress`, and add a
`NetworkPolicy` that admits only the worker pods, since nothing else stops a
pod in another namespace from dialling the Service directly.

On [bare metal](/operate/deploy-bare-metal), bind it to the private interface or
firewall it. It should never appear in a load balancer.

## The admin endpoint has no authentication [#the-admin-endpoint-has-no-authentication]

The admin REST API registers under `/admin` by default. Every shipped example
that enables it puts it on the worker API listener; one
(`nativelink-config/examples/worker_with_redis_scheduler.json5`) also enables it
on the public listener, which is the thing not to copy. Its only route is:

```text
POST /admin/scheduler/{scheduler_name}/set_drain_worker/{worker_id}/{is_draining}
```

where the first segment is the scheduler's `name` from the config (the route
parameter is called `instance_name` in the source, but it is looked up against
scheduler names). Anyone who can issue that request can drain every worker in
your fleet, one `worker_id` at a time, and your builds stop. No credential protects it. It
is also the endpoint you will legitimately want during an incident (see
[Runbooks](/operate/runbooks#a-worker-is-oom-killed)), so the answer is to make
it reachable from your operator network and nowhere else, not to disable it.

<SourceLink file="src/bin/nativelink.rs" symbol="set_drain_worker" />

The `health` service is on the same footing: path `/status` by default, five
second timeout, no authentication. It leaks less, but it is another reason the
50061 listener belongs on a private network rather than behind an
authentication proxy you have not written.

<Callout type="warn" title="Do not move admin or health onto the public listener">
  It is tempting to put `/status` on 50051 so a load balancer can probe it. That
  moves the admin API there too if they share a `services` block, and it means
  the health path is the same listener your developers reach. Configure a
  separate listener if a probe needs a reachable endpoint, and put nothing else
  on it.
</Callout>

## The committed certificates are not usable [#the-committed-certificates-are-not-usable]

Two directories in the repository contain a certificate and a key:

- `deployment-examples/docker-compose/example-do-not-use-in-prod-{rootca.crt,key.pem}`
- `kubernetes/resources/insecure-certs/example-do-not-use-in-prod-{rootca.crt,key.pem}`

They are **the same file**, byte for byte. And they are unusable for three
independent reasons:

| Property | Value |
| --- | --- |
| Subject and issuer | `CN = localhost`, self-signed |
| Valid from | October 22, 2023 |
| Valid until | **October 21, 2024** |
| Private key | Committed to a public repository |

The name is not a warning about best practice: the certificate expired, and
its private key is public. Every client that talks to a listener using it must
pass `--insecure` or equivalent, which is what the repository's own TLS
integration test does. That is the only thing these files are for.

No certificate-generation script exists anywhere in the tree. You supply real
material yourself, from whatever your organisation already uses: an internal CA,
cert-manager in a cluster, or a public CA if the endpoint is genuinely public.
[TLS and auth](/how-to/tls-and-auth) walks through what the client side needs
once you have it.

## Configure TLS [#configure-tls]

TLS is per-listener. Each server in the `servers` array carries its own
optional `tls` block. Every field below is documented in
[the configuration reference](/reference/nativelink-config):

```json5
{
  servers: [
    {
      name: "public",
      listener: {
        http: {
          socket_address: "0.0.0.0:50071",
          tls: {
            cert_file: "${NL_TLS_CERT:-/etc/nativelink/tls/tls.crt}",
            key_file: "${NL_TLS_KEY:-/etc/nativelink/tls/tls.key}",
            client_ca_file: "/etc/nativelink/tls/ca.crt",
            client_crl_file: "/etc/nativelink/tls/ca.crl",
          },
        },
      },
      services: {
        // cas, ac, bytestream, capabilities
      },
    },
  ],
}
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="TlsConfig" />

| Field | Effect |
| --- | --- |
| `cert_file` | Server certificate. Required when `tls` is present. |
| `key_file` | Private key. Required when `tls` is present. |
| `client_ca_file` | **Turns on mTLS.** Clients must present a certificate chaining to this CA. Absent means any client is accepted. |
| `client_crl_file` | Revocation list checked against presented client certificates. |

All four paths go through shellexpand, so `${VAR}` and `${VAR:-default}` work,
which is how one configuration file serves several environments. Expansion
happens once, at parse time; rotating a certificate on disk needs a restart.

<Callout type="info" title="client_ca_file is the closest thing to authentication you get">

  mTLS proves that a client holds a certificate your CA issued. It does not map
  that identity to a permission; every authenticated client can do everything.
  No read-only role, no per-instance-name restriction, and no way exists to
  let CI write to the cache while developers only read.

  If you need that distinction, it lives in front of NativeLink: separate
  listeners, separate CAs, and a proxy or network policy that decides which
  clients reach which listener.

</Callout>

### Outbound TLS [#outbound-tls]

Outbound gRPC connections (a `grpc` store's `endpoints`, and the worker's
`worker_api_endpoint`) configure their client side separately with a
`tls_config` block on the endpoint. Object-store backends do not use it; they
carry their own settings (for example `root_certificates` on the `ontap`
provider).

```json5
{
  name: "GRPC_CAS_STORE",
  grpc: {
    instance_name: "",
    store_type: "cas",
    endpoints: [
      {
        address: "grpcs://cas.internal:50071",
        tls_config: {
          ca_file: "/etc/nativelink/tls/upstream-ca.crt",
          cert_file: "/etc/nativelink/tls/client.crt",
          key_file: "/etc/nativelink/tls/client.key",
          use_native_roots: false,
        },
      },
    ],
  },
}
```

<SourceLink file="nativelink-config/src/stores.rs" symbol="ClientTlsConfig" />

`use_native_roots: true` trusts the host's system trust store instead of a
named CA. Convenient against a public endpoint, wrong against an internal one:
it means any publicly-trusted certificate for that hostname is accepted.

## The sandbox is not a security boundary [#the-sandbox-is-not-a-security-boundary]

Workers can be configured with `use_namespaces: true` and
`use_mount_namespace: true`. Both are real, and neither makes an action safe to
run on your behalf.

What the worker actually does before executing an action:

```text
unshare(CLONE_NEWPID | CLONE_NEWUSER | CLONE_NEWIPC | CLONE_NEWUTS)
write /proc/self/setgroups  "deny"
write /proc/self/uid_map    "<uid> <uid> 1"
write /proc/self/gid_map    "<gid> <gid> 1"
```

<SourceLink file="nativelink-worker/src/namespace_utils.rs" symbol="unshare" />

Read the uid map carefully. It is an **identity** map: uid *N* inside the
namespace is uid *N* outside it. The action runs as the worker's own user, with
the worker's own privileges on the host filesystem. This is not a
uid-remapping sandbox; it is a process-reaping and hermeticity mechanism. PID
namespacing means orphaned children die with the action instead of leaking;
IPC and UTS namespacing means actions cannot see each other's semaphores or
hostname (the hostname is set to `nativelink`). With `use_mount_namespace`
the flag list also gains `CLONE_NEWNS`, and the worker's own root action
directory is masked so one action cannot see another's tree; the rest of the
host filesystem stays visible.

Note what is absent from that flag list: **`CLONE_NEWNET`**. No network
namespace is created. An action has the worker's full network access, including to port
50061 on your scheduler, and to anything else on the worker's network.

An action can, by design:

- Read and write every file the worker's user can, outside its work directory
- Open a connection to the worker API and register as a worker
- Reach any internal service the worker host can reach
- Consume the whole host's memory and CPU, since there is no per-action limit
  of any kind in the configuration; the only bound is
  [`max_inflight_tasks` and the container's own limits](/operate/runbooks#a-worker-is-oom-killed)

The practical stance: **treat the right to submit an action as equivalent to
shell access on a worker host.** Then the controls follow: workers on their own
network segment with egress rules, no ambient cloud credentials on worker
instance profiles, no shared secrets in the worker's environment, and a worker
host you are willing to rebuild.

Containerising the worker helps only as far as your container runtime is a
boundary. The shipped Compose executor runs `privileged: true` precisely so
`unshare` works, which removes most of what the container was giving you.

## What to do, in order [#what-to-do-in-order]

| Step | Why |
| --- | --- |
| Put 50061 on a network only workers and operators reach | Everything else on this page is a smaller problem than this one |
| Replace the committed certificates | They expired in 2024 and the key is public |
| Set `client_ca_file` on every reachable listener | mTLS is the only client identity NativeLink can express |
| Give workers their own network segment with egress rules | An action has the worker's network |
| Remove ambient credentials from worker hosts | An action can read them |
| Put no secrets in `environment_variables` on the worker entrypoint | Every action sees them |
| Decide you can rebuild a worker host at any time | Assume one will need it |

<VerifyBlock>

  From a developer workstation, `nc -vz <scheduler-host> 50061` fails to
  connect. If it succeeds, stop and fix the network before anything else.

  `curl -s -o /dev/null -w '%{http_code}' -X POST "http://<scheduler-host>:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/x/0"`
  from outside your operator network fails to connect (a `500` means it
  reached the admin API and only failed to find worker `x`).

  `openssl s_client -connect <cas-host>:50071 -showcerts` presents your
  certificate, not `CN = localhost`, and reports a `notAfter` in the future.

  A client with no certificate is rejected by that listener once `client_ca_file`
  is set. If it still connects, the listener is not the one you configured.

  `grep -r example-do-not-use-in-prod` across your deployed configuration
  returns nothing.

</VerifyBlock>

## Troubleshooting [#troubleshooting]

| Symptom | Cause | Fix |
| --- | --- | --- |
| Clients fail with certificate-expired errors | You are still serving the committed example certificate | Supply real material; it expired October 21, 2024 |
| TLS handshake fails only for some clients | `client_ca_file` is set and those clients have no certificate | Issue them one, or move them to a listener without mTLS |
| Workers cannot register after locking down the network | 50061 is now unreachable from worker hosts too | Admit worker sources explicitly rather than blocking the port outright |
| A certificate rotated on disk but the process serves the old one | Expansion and file reads happen at parse time | Restart the process |
| An action reached an internal service it should not have | No network namespace | Segment the worker network; this is not fixable in configuration |
| Actions fail with permission errors after adding `NoNewPrivileges` | It blocks the `unshare` the worker performs | See [bare metal](/operate/deploy-bare-metal), which omits it deliberately |
| The health probe works from the internet | `health` shares a listener with something public | Give it a dedicated listener |
| Something else is broken and it is not on this list | Not everything is a security problem | [Troubleshooting](/operate/troubleshooting) indexes the rest |

## What's next [#whats-next]

<NextStep href="/how-to/tls-and-auth" title="TLS end to end">
  Certificates on both sides, including what Bazel needs on the client.
</NextStep>

<NextStep href="/operate/runbooks" title="Runbooks" kind="aside">
  What to do when one of these goes wrong at three in the morning.
</NextStep>

<NextStep href="/operate/production-config" title="The rest of the production shape" kind="aside">
  Three processes, their stores, and the pre-launch checklist.
</NextStep>

# https://docs.nativelink.com/operate/scaling-workers

**Scaling workers**: How much work one worker should take, how many workers you need, which signal tells you, and how to let something else make the decision, including the parts NativeLink does not enforce for you.

**Who this is for:** you have a worker pool serving real builds and the queue is
either backing up or sitting idle. **What you'll have at the end:** a sized
worker, a scaling signal you trust, and either a manual playbook or an
autoscaler wired to it. **Time:** an afternoon for the manual version, longer if
you're building the autoscaling path.

<Prerequisites>
  A worker pool serving real builds.
</Prerequisites>

[Run multiple workers](/how-to/run-multiple-workers) is how you get from one
worker to several. This page is about the questions that come next: how much
work each one should take, how many you need, and who decides.

## Two axes, and only one of them is a NativeLink knob [#two-axes-and-only-one-of-them-is-a-nativelink-knob]

Capacity has a vertical axis (how many actions one worker runs at once) and a
horizontal axis (how many workers exist). NativeLink has a field for the first.
It has nothing at all for the second, by design: a worker is a process that
dials the scheduler and announces itself, so "more workers" means "start more
processes," and whatever starts processes for you is the thing that scales.

Start with the vertical axis, because it changes what the horizontal one costs.

## Sizing one worker [#sizing-one-worker]

The knob is `max_inflight_tasks` on the worker's config.

```json5
workers: [{
  local: {
    name: "compile_worker",
    max_inflight_tasks: 16,
    // ...
  },
}],
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

Three things about it are not what most people assume, and each has bitten
somebody.

**The default is `0`, and `0` means unlimited.** An unconfigured worker will
accept as many concurrent actions as the scheduler sends it. That is fine on a
small pool and a genuinely bad time on a large one, because the failure mode is
not a queue; it's memory exhaustion or an out-of-file-descriptors error partway
through a build. Set it explicitly.

**It is enforced on the scheduler, not on the worker.** The worker sends the
number when it registers; the scheduler tracks how many actions each worker is
holding and stops assigning past the limit. The worker itself has no admission
check and will run whatever it is handed. In normal operation this distinction
is invisible. It matters if you ever run two schedulers against one worker, or
if you're reasoning about what protects the host: the answer is the scheduler's
bookkeeping, and nothing else.

<SourceLink file="nativelink-scheduler/src/worker.rs" symbol="Worker" />

**`cpu_count` does not limit anything on the worker.** This is the one that
surprises people most. The `cpu_count` platform property, usually populated
with `query_cmd: "nproc"` in the shipped examples, is *advertising*. It tells
the scheduler what to match against. It does not throttle execution, and the
worker derives no concurrency from the host's CPU count at all. Declare
`cpu_count: 64` on a four-core box and the scheduler will happily route
64-core actions to it.

<Callout type="warn" title="`cpu_count` and `max_inflight_tasks` are unrelated">
  One is what the scheduler matches on; the other is what the scheduler counts.
  If your actions declare `cpu_count` as a `minimum` requirement, that number is
  consumable (the scheduler subtracts it on assignment and restores it on
  completion), so it does bound concurrency indirectly. But it does so only for
  actions that declare it. `max_inflight_tasks` is the bound that applies to
  everything.
</Callout>

A reasonable starting point is `max_inflight_tasks` at roughly the host's core
count for compile-shaped work, then adjusted by observation: if per-action
durations climb as the number in flight rises, you're oversubscribed; if the
host sits at half its CPU with a deep queue, you're undersubscribed.

## The ceilings you hit before you expect to [#the-ceilings-you-hit-before-you-expect-to]

A worker sized generously runs into limits that are not on the worker config,
and they're worth knowing before you go hunting.

**File descriptors.** NativeLink holds a process-wide semaphore over open files,
defaulting to **24,576**. Every action materializing inputs and writing outputs
draws from it. Raise it with `max_open_files` under `global` (the shipped
Kubernetes worker config sets 524,288) and raise the OS limit to match, or the
semaphore will let through more than the kernel allows.

<SourceLink file="nativelink-util/src/fs.rs" symbol="DEFAULT_OPEN_FILE_LIMIT" />

**gRPC connections to the CAS.** The `grpc` store's `connections_per_endpoint`
defaults to **1**. One HTTP/2 connection multiplexes streams well up to a point
and then becomes the bottleneck, and on a worker running dozens of actions
against a remote CAS that point arrives early. This is the first knob to reach
for when a worker's CPU is idle and its actions are slow.

**Store-level concurrency caps.** `max_concurrent_writes`,
`max_concurrent_requests`, and Redis's `connection_pool_size` (default 3) all
sit between the worker and its bytes. A worker that scales up without these
scaling with it just queues internally.

<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

**One `work_directory` per worker process, always.** The worker **purges its
work directory on startup**. Two workers pointed at the same path means the
second one to start deletes the first one's in-flight state. On a shared host,
give each worker its own path; in containers this is free.

## The signal to scale on [#the-signal-to-scale-on]

Not CPU. Worker CPU tells you whether a machine is busy; it cannot tell you
whether there is work waiting that nobody is doing, and that's the only question
an autoscaler needs answered.

The signal is **queue depth**: actions in the `queued` stage rather than
`executing`. NativeLink emits `execution.active.count` as an up-down counter
attributed by `execution.stage`, which reaches Prometheus as
`nativelink_execution_active_count` after the OTLP collector's Prometheus
exporter adds its `nativelink` namespace.

<SourceLink file="nativelink-util/src/metrics.rs" symbol="EXECUTION_METRICS" />

The shipped recording rule is:

```yaml
- record: nativelink:queue_depth
  expr: |
    sum by (instance_name, execution_priority) (
      nativelink_execution_active_count{execution_stage="queued"}
    )
```

That's the series to scale on. Three caveats before you copy it into production.

**Queue depth is a derived series, not a metric the binary emits.** No
metric name exists that an HPA can target directly. You scale on the recording rule, read
through an external-metrics adapter.

**The example alert's window is wrong for autoscaling.** The shipped
`NativeLinkQueueBacklog` alert fires on a queue over 100 sustained for 15
minutes. That is a correct alert ("you are chronically undersized") and a
terrible autoscaling trigger. An HPA wants a window in the tens of seconds. Use
the alert for paging and a much shorter window for scaling.

**The repository disagrees with itself about worker utilization.**
`nativelink:worker_utilization` is defined one way in
`deployment-examples/metrics/prometheus-recording-rules.yml` and differently in
the inline rules inside `prometheus-config.yaml`, and the Kubernetes alert uses
a third expression. `nativelink:actions_per_worker` is also misnamed; it
records a per-worker sum, not an average. Scale on queue depth, which is
unambiguous, and treat the utilization series as a dashboard aid rather than a
control input.

<Callout type="warn" title="The Redis scheduler backend emits no queue metric">
  `execution.active.count` comes from the in-memory awaited-action store only.
  If you have moved the scheduler to `experimental_backend` with Redis, the
  signal this whole page depends on is not being emitted. Confirm which backend
  you're on before you design an autoscaler around it.
</Callout>

## Scaling out [#scaling-out]

<Steps>

<li>

**Set `max_inflight_tasks` explicitly on every worker** and confirm it took
effect by watching per-worker action counts during a busy build. Leaving it at
the default means the horizontal axis is being tuned against an unbounded
vertical one, and nothing you measure will be stable.

</li>

<li>

**Get `nativelink:queue_depth` onto a dashboard** and watch it through a normal
day. You are looking for its shape: how deep it goes at peak, how long it stays
there, and how quickly it drains. That shape is what sizing decisions are made
against.

</li>

<li>

**Find your per-worker throughput.** With a stable pool size, divide actions
completed by workers by time. This is the number that converts "the queue is 400
deep" into "I need N more workers," and it is workload-specific enough that
nobody else's figure helps.

</li>

<li>

**Scale manually first, on a written rule.** Something as plain as "if queue
depth exceeds 200 for two minutes, add four workers; if it's under 20 for ten
minutes, remove two" is enough. Run it by hand for a week. You will learn more
about your build's shape than from any amount of upfront design, and you'll have
the thresholds an autoscaler needs.

</li>

<li>

**Then automate that rule.** On Kubernetes, an HPA on the external metric backed
by `nativelink:queue_depth`, plus a cluster autoscaler or Karpenter underneath
so pods have somewhere to land. Elsewhere, whatever your platform's equivalent
is. The rule you validated by hand becomes the policy.
[The reference deployment](/operate/autoscaling-reference) is that assembled end
to end, including the parts the shipped manifests are missing.

</li>

<li>

**Move the fleet to spot capacity once retries are proven.** A worker that
disappears mid-action has its work requeued with its affinity cleared, so
interruption is a cost rather than an incident. Verify that in a staging pool by
killing a worker mid-build before you rely on it in production.

</li>

<li>

**Wire the drain endpoint to your interruption notice.** For shutdowns you get
warning about (spot termination, node upgrades), call the scheduler's admin
endpoint so the worker finishes what it holds and takes nothing new:

```
POST {admin_path}/scheduler/{scheduler_name}/set_drain_worker/{worker_id}/1
```

The route's second segment is the scheduler's `name` from the `schedulers`
list (`MAIN_SCHEDULER` in the shipped configs), not the REAPI instance name,
and the `admin` service has to be enabled on a listener for the route to
exist at all. Nothing in the repository wires this up for you. It is a short
script, and it's the difference between a clean scale-in and a burst of
retried actions.

<SourceLink file="src/bin/nativelink.rs" symbol="set_drain_worker" />

</li>

</Steps>

<VerifyBlock>

- Queue depth rises under load and returns to near zero afterwards, rather than
  sitting at a floor; a floor means you are permanently undersized.
- Adding workers visibly shortens the queue's drain time. If it doesn't, the
  bottleneck is the CAS or the network, not worker count.
- Per-action durations stay flat as the pool grows. Durations that climb with
  pool size mean the workers are contending for something shared.
- Killing a worker mid-build produces retried actions and a successful build,
  not a failed one.
- Scale-in does not coincide with a spike in retried actions. If it does, the
  drain hook isn't being called.

</VerifyBlock>

## What the repository actually ships [#what-the-repository-actually-ships]

Being explicit, because the gap is larger than the docs elsewhere imply.

**`deployment-examples/docker-compose/docker-compose-multi-worker.yml`** defines
three worker services. They are copy-pasted and identical apart from a
`WORKER_NAME` and a private data volume each: same config file, same shared
CAS volume, two CPUs and 2 GB each. It has no `deploy.replicas` and no
heterogeneity. A fourth worker means pasting a fourth block.

**`deployment-examples/docker-compose/test-multi-worker-simple.json5`** is a
genuinely different pattern worth knowing about: **three workers inside one
process**, as three entries in the `workers` array of a single config. One
binary, one config file, three registrations with the scheduler. It's a useful
shape for a machine you want to subdivide, and it is not the same thing as three
containers.

**`kubernetes/components/worker/worker.yaml`** sets `replicas: 3`. The `lre-cc`
and `lre-rs` overlays patch it to 1. The `siso-chromium` overlay does not patch
it, so it inherits 3, which looks unintentional.

**No HorizontalPodAutoscaler, KEDA `ScaledObject`, or VPA exists anywhere in
the repository for NativeLink.** The single HPA in the tree scales the
OpenTelemetry collector. No heterogeneous-pool example exists in
`nativelink-config/examples/` either; every shipped worker example is a single
homogeneous pool. The autoscaling path on this page is one you assemble from
standard Kubernetes parts; the recording rules are the raw material, not a
finished artifact.

## Backpressure, when a worker needs to stop [#backpressure-when-a-worker-needs-to-stop]

One mechanism lets a worker refuse work without being removed:
`experimental_precondition_script`. It runs before every action, and a non-zero
exit produces a `ResourceExhausted` error. The scheduler requeues the action
without counting it as an attempt, and marks the worker `is_paused` **only if
that worker still has other actions in flight**; the pause clears as soon as
one of them completes. An idle worker whose script fails is never paused: it
keeps its registration and is handed the action again on the next
matching pass, and rejects it again.

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="preconditions_met" />
<SourceLink file="nativelink-scheduler/src/api_worker_scheduler.rs" symbol="update_action" />

This is the hook for host-level conditions the scheduler cannot see: disk
filling, a shared license pool exhausted, a GPU in a bad state. It costs a
process spawn per action, so keep the script trivial. And note the
`experimental_` prefix: it is not covered by stability guarantees.

## Heterogeneous pools [#heterogeneous-pools]

One scheduler can route to pools of different shapes; the mechanism is platform
properties, and it is covered in full on
[platform properties](/remote-execution/platform-properties). The scaling point
is narrower: **each pool is a separate scaling unit with its own queue-depth
series**, because `nativelink:queue_depth` groups by instance and priority, not
by which pool could serve the work.

If you scale a mixed fleet on one aggregate number, a backlog of GPU actions
will scale up your compile pool and change nothing. Either split the recording
rule by the property that distinguishes the pools, or run a scheduler instance
per pool. The second is blunter and easier to reason about.

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Queue depth is high but workers are not busy" id="queue-depth-is-high-but-workers-are-not-busy">

Actions are queued that no worker matches. This is nearly always a platform
property mismatch: an action declaring something no worker advertises, or a
scheduler key configured as `priority` (which constrains nothing) when it should
be `exact` or `minimum`. Adding workers will not help. Check the scheduler's
worker-match logging, and see
[platform properties](/remote-execution/platform-properties).
[Troubleshooting](/operate/troubleshooting#actions-queue-forever-and-workers-sit-idle)
has the log lines that distinguish "nothing matches" from "everything is busy",
and why a bad property *value* hangs where a bad property *name* errors.

</Accordion>

<Accordion title="Adding workers doesn't make builds faster" id="adding-workers-doesnt-make-builds-faster">

The bottleneck is downstream. Workers pull inputs from and push outputs to the
CAS, so a saturated CAS or a saturated network means more workers just wait in a
different place. Check CAS latency as you scale, and check
`connections_per_endpoint` on the workers' `grpc` store; the default of 1 makes
a single connection the ceiling.

</Accordion>

<Accordion title="Workers keep being marked dead and their actions requeued" id="workers-keep-being-marked-dead-and-their-actions-requeued">

`worker_timeout_s` defaults to **5 seconds**, checked once a second. That is a
short leash and it is deliberate (fast detection means fast requeue), but a
worker whose host is briefly starved, or whose network hiccups, will be evicted.
If you see churn without actual failures, raise it. The cost of raising it is a
longer stall before a genuinely dead worker's actions are retried.

</Accordion>

<Accordion title="Setting `max_job_retries: 0` didn't stop retries" id="setting-max_job_retries-0-didnt-stop-retries">

A configured `0` is read as unset and falls back to the default of **3**. The
field cannot express "never retry" through that field. If an action must not be
retried, that has to be handled above NativeLink.

</Accordion>

<Accordion title="A long action fails with an invalid-argument error about timeouts" id="a-long-action-fails-with-an-invalid-argument-error-about-timeouts">

`max_action_timeout_s` on the worker defaults to **20 minutes**, and a client
requesting a longer timeout is **rejected outright** rather than clamped down to
the maximum. Long link steps and EDA runs need this raised on the worker before
the client asks for it. (The field also accepts the older spelling
`max_action_timeout`.)

</Accordion>

<Accordion title="Workers exhaust file descriptors under load" id="workers-exhaust-file-descriptors-under-load">

The process-wide open-file semaphore defaults to 24,576, which a worker running
many concurrent actions over a filesystem store will reach. Raise `max_open_files`
under `global`, and raise the OS `nofile` limit to at least match; the
semaphore does not know what the kernel will allow.

</Accordion>

<Accordion title="Scale-in causes a burst of failed or retried actions" id="scale-in-causes-a-burst-of-failed-or-retried-actions">

Workers are being killed with work in flight. The drain endpoint exists for
exactly this and nothing calls it automatically. Wire it to your platform's
pre-shutdown hook, and give the pod a termination grace period long enough for a
typical action to finish.

</Accordion>

<Accordion title="Two workers on one host behave erratically" id="two-workers-on-one-host-behave-erratically">

Check they do not share a `work_directory`. The worker purges that directory at
startup, so a restart of one worker destroys the other's state. Each worker
process needs its own.

</Accordion>

</Accordions>

<NextStep href="/operate/scaling-cas" title="Scaling the CAS">
  The other half of capacity: composing the store graph, sizing eviction, and
  why a second CAS process is harder than a second worker.
</NextStep>

<NextStep href="/operate/troubleshooting" kind="aside" title="Troubleshooting">
  The full symptom-to-cause index, including the failures that produce no error
  string at all.
</NextStep>

<NextStep href="/operate/observability" kind="aside" title="Observability">
  Getting `execution.active.count` out of the binary and into something an
  autoscaler can query, which is the prerequisite for everything above.
</NextStep>

# https://docs.nativelink.com/operate/scaling-cas

**Scaling the CAS**: How to grow the content-addressable store: composing tiers, sizing eviction, sharding, and the specific reasons a second CAS replica is harder than a second worker.

**Who this is for:** you run a CAS that real builds depend on, and it is
running out of disk, running out of throughput, or both. **What you'll have at
the end:** a store composition sized for your load, an eviction policy that
doesn't thrash, and a clear-eyed view of what running more than one CAS process
costs. **Time:** an afternoon to size and tune; longer if you go horizontal.

<Prerequisites>
  A CAS that real builds depend on.
</Prerequisites>

[Scaling workers](/operate/scaling-workers) is the straightforward half of capacity. This
is the other half, and it is a different problem: a worker holds nothing, so
adding one is free. The CAS holds everything, so adding one is a design
decision.

## Start here: a CAS process is not stateless [#start-here-a-cas-process-is-not-stateless]

Most guidance about scaling services assumes replicas are interchangeable. For
NativeLink's CAS that assumption is wrong in five specific places, and every
horizontal-scaling decision on this page follows from them.

**In-flight resumable uploads live in one process.** The ByteStream service
keeps a registry of active uploads keyed by the upload's UUID, in memory, per
process.

<SourceLink file="nativelink-service/src/bytestream_server.rs" symbol="ByteStreamServer" />

When a client asks `QueryWriteStatus` to resume an interrupted upload, the
server looks the UUID up in that map. If the request lands on a replica that
never saw the upload, the lookup misses, the fallback checks whether the blob is
already complete in the store, and on a miss the client is told to start over
from byte zero. The source comment at that branch says so in as many words. A
large blob interrupted at 90% and resumed against a different replica is
re-uploaded in full.

**LRU state is per process.** The eviction index (recency ordering, running
size total, entry count) belongs to the process that built it.

<SourceLink file="nativelink-util/src/evicting_map.rs" symbol="EvictingMap" />

Two replicas over the same logical dataset make independent eviction decisions
and diverge.

**The existence cache produces stale positives.** An `existence_cache` store
remembers "this digest exists" in memory and is invalidated by its backend's
remove callbacks. Those callbacks are in-process for a filesystem or memory
backend, and there is no cross-process path at all for S3, GCS or Azure, so
a blob removed underneath one replica (an object lifecycle rule, another
replica's eviction, an operator) leaves the other replicas answering "present".
The client skips the upload, and a later fetch fails. The one backend with a
cross-process signal is Redis: with `notify-keyspace-events` set to include
`K` and `A`, every process subscribed to the store hears Redis-side
evictions and TTL expiries and drops the entry.

<SourceLink file="nativelink-store/src/existence_cache_store.rs" symbol="ExistenceCacheStore" />
<SourceLink file="nativelink-store/src/redis_store.rs" symbol="notify-keyspace-events" />

<Callout type="warn" title="Do not put `existence_cache` behind a load balancer">
  It is the most dangerous wrapper to replicate. It is safe on a single process
  and safe in front of a backend nothing else removes from; it is unsafe the
  moment anything other than that process deletes from the shared backing
  store, unless the backing store is Redis with keyspace notifications on. The
  failure mode is a build that skips an upload and then can't find its own
  artifact.
</Callout>

**A filesystem store cannot share a directory.** Two processes pointed at the
same `content_path` each build their own index at boot and each unlink files the
other believes it owns. This is not a supported configuration.

**Read deduplication is intra-process only.** `fast_slow` collapses concurrent
reads of the same digest into a single fetch from the slow tier, but only within one
process. Three replicas asking for the same cold blob at once make three
requests to S3.

It has no leader election, no gossip, no Raft or quorum protocol, no etcd and
no ZooKeeper anywhere in NativeLink. Replicas are independent processes, and any
coordination between them has to come from the semantics of the shared backing
store you point them at.

## Scale vertically first, by composing [#scale-vertically-first-by-composing]

The store graph is where most capacity problems are solved, and it is cheaper
than adding processes because it changes nothing about consistency.
[Compose stores](/how-to/stores/compose-stores) is the reference for the
wrappers; what follows is which one relieves which pressure.

| Pressure | Layer | What it buys |
| --- | --- | --- |
| Disk is full | `compression` | LZ4 on the slow tier. It is the only algorithm the `compression` store offers (zstd exists only as REAPI wire compression on the `grpc` store). |
| Disk is full, inputs are similar | `dedup` | Content-defined chunking, 64 KiB to 512 KiB, so near-identical blobs share chunks. |
| Reads are slow | `fast_slow` | A local hot tier in front of object storage or Redis. |
| Small blobs dominate request count | `size_partitioning` | Route small objects to memory, large ones elsewhere. |
| One backend is the ceiling | `shard` | Spread keys across several backends by hash. |
| `FindMissingBlobs` is hot | `existence_cache` | Answer existence without touching the backend; single process only. |
| AC hits point at missing blobs | `completeness_checking` | Re-check the CAS on every AC read. |

<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

Three details of composition matter for capacity specifically and aren't
apparent from the individual recipes.

**`fast_slow` populates the fast tier on a read miss**, streaming from the slow
store to the caller and into the fast store at the same time. That's what makes
the hot tier fill itself. It also means a single large cold blob can evict a lot
of hot working set, which is what `bypass_dedup_threshold_bytes` is for: at or
above that size the read goes straight to the slow store and does not
populate. It defaults to `0`, which disables the bypass.

**`size_partitioning` splits at `size`, exclusive on the low side**: a blob of
exactly that many bytes goes to the *upper* store. It also rejects non-digest
keys outright, so it must not sit anywhere string-keyed traffic flows.

**A memory store silently drops blobs at or above its `max_bytes`.** The write
is skipped with a log line rather than an error, and the blob is served from the
backing store instead. If a memory fast tier appears to be doing nothing, this
is usually why.

<SourceLink file="nativelink-store/src/fast_slow_store.rs" symbol="FastSlowStore" />

## Eviction is your capacity control, and its default thrashes [#eviction-is-your-capacity-control-and-its-default-thrashes]

Four fields, all on `EvictionPolicy`, all defaulting to `0`, and `0` means
*disabled* in every case. A policy with no fields set is an unbounded store.

```json5
eviction_policy: {
  max_bytes: 500000000000,
  evict_bytes: 50000000000,
  max_seconds: 2592000,
  max_count: 0,
}
```

`max_bytes` is the ceiling. `evict_bytes` is the low watermark: when eviction
triggers, keep going until the store is down to `max_bytes - evict_bytes`.
`max_seconds` expires entries by age since last access. `max_count` caps the
number of entries.

<Callout type="warn" title="`evict_bytes: 0` makes eviction run on nearly every write">
  With no low watermark configured, eviction evicts the bare minimum: it pops
  LRU entries only until the store is back under `max_bytes`, then stops. The
  store sits pinned at its high-water mark, and the next write puts it over
  again. Every example config in the repository leaves `evict_bytes` unset.
  Setting it to some fraction of `max_bytes` is the single cheapest tuning
  change available on this page.
</Callout>

Two more behaviours to plan around:

**No background eviction timer exists.** Eviction runs on inserts and
removals (and when a worker releases a lease on an input it was using). Reads
perform a lazy per-key expiry check on the key being read and nothing else. The consequence is that **an idle store never shrinks**; a CAS
that goes over quota and then stops receiving writes stays over quota, and
`max_seconds` only takes effect on entries someone happens to ask for.

**Eviction is whole entries, one at a time.** It does no partial or chunked
eviction; the unit is one blob, or one file for a filesystem store.

### Size the AC and the CAS together [#size-the-ac-and-the-cas-together]

The Action Cache is just another store with its own `EvictionPolicy`; there is
no separate mechanism. The shipped examples give it roughly a twentieth of the
CAS budget, which is a reasonable starting ratio because AC entries are tiny.

That asymmetry creates the hazard. AC entries are small and survive eviction
readily; the CAS blobs they *point at* are large and get evicted first. The
result is an AC hit that resolves to missing content. `completeness_checking`
exists precisely for this: it re-queries the CAS on every AC read and treats an
incomplete result as a miss. It costs a CAS round trip per AC hit, and on a
shared production cache it is worth it.

<SourceLink file="nativelink-store/src/completeness_checking_store.rs" symbol="CompletenessCheckingStore" />

## Sharding, and what changing it costs [#sharding-and-what-changing-it-costs]

`shard` distributes keys across several backends by hash, with an optional
integer `weight` per shard. The weights are turned into a cumulative table over
the `u32` space at startup and the key is binary-searched into it.

<SourceLink file="nativelink-store/src/shard_store.rs" symbol="ShardStore" />

Two properties decide whether it fits.

**Placement includes the blob's size.** The hash folds the digest *and* its
declared size, so the same content at a different declared size lands in a
different shard. This is fine (it's still deterministic), but it means you
cannot reason about placement from the hash alone.

**Nothing rebalances.** No migration, no virtual-node consistent hashing,
no read repair, no re-shard command. Adding, removing, or reweighting a shard
moves the boundaries and remaps a large fraction of keys. The data is still
physically where it was written; lookups just go somewhere else and never find
it.

<Callout type="warn" title="Treat any shard change as a cache flush">
  Plan a shard-list change the way you would plan wiping the cache: during a
  quiet window, with the expectation that the next build is cold. Size the shard
  count for where you expect to be in a year rather than for today.
</Callout>

Sharding is the right tool when one backend is genuinely the ceiling: a single
Redis instance's memory, one filesystem's IOPS. It is the wrong tool for
capacity you could have bought by pointing at object storage instead.

## Going horizontal [#going-horizontal]

Everything above scales one process. If you have exhausted that, here is the
order to add replicas in.

<Steps>

<li>

**Move the backing store off local disk first.** Replicas only make sense over
storage they can share: S3 or a compatible object store, GCS, Azure, or Redis.
A filesystem store cannot be shared between processes at all, so this is a
prerequisite rather than an optimisation. [S3 and
compatible](/how-to/stores/s3-and-compatible) and [Redis](/how-to/stores/redis)
are the recipes.

</li>

<li>

**Keep the per-replica fast tier local and private.** Each replica gets its own
`fast_slow` hot tier on its own disk, pointed at the shared slow store. This is
the topology that works: shared truth underneath, independent caches above.
Accept that the hot tiers diverge; they are caches of a cache.

</li>

<li>

**Remove `existence_cache` from the replicated path**, or accept that it will
serve stale positives. No cross-process invalidation exists except for a
Redis backend with keyspace notifications enabled. If
`FindMissingBlobs` latency is why you wanted it, a `fast_slow` hot tier gets you
most of the same benefit without the correctness hazard.

</li>

<li>

**Make ByteStream uploads sticky.** Configure your load balancer for session
affinity so a client's upload stream and its `QueryWriteStatus` resume reach the
same replica. Without it, every interrupted large upload restarts from zero.
This is the single most common way a horizontally scaled CAS gets slower rather
than faster.

</li>

<li>

**Set readiness to account for the startup scan.** A filesystem store scans its
entire `content_path` at boot and builds its index *before the process binds any
socket*. During that window there is nothing to connect to: not a degraded
service, an absent one. The window grows with the number of files on disk, so
give readiness and liveness probes an initial delay that reflects a full cache,
not an empty one.

</li>

<li>

**Scale in slowly, and prefer fewer larger replicas.** Removing a replica throws
away a warm fast tier that took hours to fill, and the actions that would have
hit it now go to the slow store. The CAS is the opposite of the worker pool
here: generous, patient, and biased toward vertical growth.

</li>

<li>

**Only then consider `shard`**, and only if a single shared backend is provably
the ceiling. Re-read the cache-flush warning above before you commit to a shard
count.

</li>

</Steps>

<VerifyBlock>

- Store size levels off below `max_bytes` and stays there, rather than sawing up
  to the limit on every write; that saw is `evict_bytes` being unset.
- A cold build populates the fast tier and the next identical build reads almost
  entirely from it.
- Interrupting a large upload and resuming it resumes from where it stopped,
  not from zero. If it restarts, affinity isn't working.
- An AC hit never resolves to a missing blob. If it does, add
  `completeness_checking`.
- Restarting a replica does not fail client requests; it fails readiness first,
  and the load balancer takes it out before it stops answering.
- Disk usage on each replica's fast tier is bounded and roughly equal.

</VerifyBlock>

## The ceilings you hit before you expect to [#the-ceilings-you-hit-before-you-expect-to]

**File descriptors, at 80% of what you configured.** `global.max_open_files`
defaults to 24576, and the process reserves a fifth of it for sockets and pipes,
so the effective permit budget is about 19660.

<SourceLink file="nativelink-util/src/fs.rs" symbol="DEFAULT_OPEN_FILE_LIMIT" />

Every filesystem operation that goes through the store's file helpers takes a
permit, including metadata-only ones such as `metadata`, `read_dir` and
`rename`. (The startup scan is the exception: it holds one permit for the
directory handle and runs its 200-way parallel `stat` outside the semaphore.)
Network sockets are *not* counted by this semaphore but *are* counted by the
kernel, so the kernel's limit must exceed `max_open_files` plus your expected
connection count.

**One connection per gRPC endpoint.** A `grpc` store's
`connections_per_endpoint` defaults to `1`, so all traffic to a remote store
multiplexes over a single HTTP/2 connection. On a busy CAS-of-a-CAS topology
this is usually the first thing to raise.

**Redis permits and pool size.** `max_client_permits` defaults to 500 in-flight
operations and `connection_pool_size` to 3. The permit cap exists specifically
to stop timeouts from unbounded in-flight work, so raise it deliberately rather
than reflexively.

**Object-store upload concurrency.** `multipart_max_concurrent_uploads`
defaults to 10 per request, with a 5 MB retry buffer per request.

**Message size, which is not a blob size limit.** The HTTP listener's
`max_decoding_message_size` defaults to 4 MiB and bounds a single decoded gRPC
message, so it caps `BatchUpdateBlobs`, not ByteStream. No
configuration anywhere caps how large a blob a CAS will accept over
ByteStream. Capacity is bounded by eviction and disk, not by an upload check.

[Tuning](/operate/tuning) has the full lever table for the storage path,
including `evict_page_cache` and its caveat.

## Metrics: nothing is on by default [#metrics-nothing-is-on-by-default]

This is the finding most likely to catch you out while you're trying to measure
any of the above.

NativeLink defines a full set of cache instruments: operation duration,
operation counts by result, bytes in and out, item size distribution, and
(declared but [never emitted](/reference/metrics#declared-but-never-emitted))
current size and entry count.

<SourceLink file="nativelink-util/src/metrics.rs" symbol="CACHE_METRICS" />

**No store emits any of them unless you explicitly wrap it in the
`cache_metrics` store.** That wrapper is the only caller of the instruments, and
the design is deliberately opt-in so unwrapped stores carry no timers or
attribute allocation in their hot path. **None of the shipped deployment
configs wrap anything** (the only `cache_metrics` block in the tree is in the
config-parser coverage fixture `nativelink-config/examples/stores-config.json5`),
so out of the box an operator gets zero hit rate and zero latency data.

```json5
{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: {
      // the composition you would otherwise have written here
    },
  },
},
```

Wrap the layer whose behaviour you want to see, and give each one a distinct
`cache_type`; the recording rules group by it.

`deployment-examples/metrics/prometheus-recording-rules.yml` defines a
`nativelink_cache` group that is genuinely useful once metrics are flowing:
`nativelink:cache_hit_rate`, latency at p50/p95/p99,
`nativelink:cache_eviction_rate`, read and write throughput, and
`nativelink:cache_error_rate`. The same group also defines
`nativelink:cache_size_bytes` and `nativelink:cache_entry_count`, which are
built on the two instruments nothing emits and will stay empty.

<Callout type="info" title="Recording rules exist; alerts on them do not">
  The shipped alert rules cover cache miss rate and eviction rate only. Nothing
  alerts on read latency, on throughput collapse, or on
  `nativelink:cache_error_rate`; write those yourself. No emitted
  size metric exists to alert on at all, so size-approaching-`max_bytes` has to come
  from disk usage on the host rather than from NativeLink.
</Callout>

[Observability](/operate/observability) covers getting the telemetry out of the
process at all, which is the prerequisite for any of this.

## What the repository actually ships [#what-the-repository-actually-ships]

Being explicit, because the gap between the examples and a production CAS is
wide.

**Every shipped deployment runs exactly one CAS process.** The docker-compose
single-node example, the multi-worker compose example, and the Kubernetes
manifest all have one. `docker-compose-multi-worker.yml` replicates *workers*
against one `cas-server`; the only thing replicated anywhere in the repository
is the worker pool.

**No multi-replica CAS example exists**, no load balancer or Service fronting
several CAS pods, no HPA for the CAS, and no readiness gate accounting for the
startup scan. Anyone scaling the CAS horizontally is operating outside every
path this repository exercises.

**The Kubernetes example stores the CAS in the container.**
`kubernetes/nativelink/nativelink.yaml` is a single Deployment with
`replicas: 1`, and its filesystem store points at a path under `/tmp` inside the
container: no PersistentVolumeClaim, no StatefulSet, not even an `emptyDir`.
Every pod restart wipes the CAS and the AC. It is a demo topology, and the
composition inside its config is worth reading even though the storage isn't.

**Sharding has no deployment example.** The only `shard` block in the tree is a
single-shard entry in a config-parser coverage fixture, which exercises no
distribution behaviour at all.

**The `cas-data` volume in the multi-worker compose file is vestigial.** It is
mounted into the CAS server and into all three workers at the same path, but the
workers' config points their fast tier somewhere else entirely and reaches the
CAS over gRPC. Don't read that mount as an endorsement of sharing a filesystem
store directory between processes; it isn't one.

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="Disk usage sits exactly at max_bytes and write latency is spiky" id="disk-usage-sits-exactly-at-max_bytes-and-write-latency-is-spiky">

That's the `evict_bytes: 0` thrash. With no low watermark, every write pushes
the store over the limit and re-enters the eviction loop to remove the minimum.
Set `evict_bytes` to a real fraction of `max_bytes`, enough that a burst of
writes fits under the headroom it creates.

</Accordion>

<Accordion title="The store is over quota but nothing is being evicted" id="the-store-is-over-quota-but-nothing-is-being-evicted">

Eviction is triggered by inserts and removals; there is no background timer. A
store that went over quota and then went idle stays over quota. Writes will
bring it back down. If the store is idle because clients moved elsewhere, the
space is only reclaimed by restarting with a lower `max_bytes` or by clearing
the path directly.

</Accordion>

<Accordion title="Large uploads restart from the beginning" id="large-uploads-restart-from-the-beginning">

The upload registry is per process. If a client's ByteStream write and its
`QueryWriteStatus` resume land on different replicas, the second one has never
heard of the upload and tells the client to start over. Enable session affinity
on the load balancer, keyed so a single client's stream stays on one replica.

</Accordion>

<Accordion title="Bazel says a blob exists, then a later fetch can't find it" id="bazel-says-a-blob-exists-then-a-later-fetch-cant-find-it">

An `existence_cache` served a stale positive. It caches "digest present" in
memory, and unless the backend is Redis with keyspace notifications on there
is no cross-process invalidation, so it will keep answering yes after the blob
was removed underneath it. Remove the wrapper from any replicated
deployment, or accept the failure mode. `completeness_checking` on the AC path
catches the related case where an AC hit points at an evicted CAS blob.

</Accordion>

<Accordion title="A restarted CAS refuses connections for minutes" id="a-restarted-cas-refuses-connections-for-minutes">

The filesystem store scans `content_path` and builds its eviction index before
the process binds a socket. The scan is O(files) (200-way parallel `stat`, then
a serialized index build), so a large cache means a long window with nothing
listening. Raise your probes' initial delay to cover a full cache, and expect
the window to grow as the cache does.

</Accordion>

<Accordion title="Two CAS processes point at the same content_path and files keep disappearing" id="two-cas-processes-point-at-the-same-content_path-and-files-keep-disappearing">

Each process builds an independent index at boot and each evicts against its
own view, unlinking files the other still has indexed. No locking and
no coordination exist. Give each process its own directory and share a slow tier
underneath instead.

</Accordion>

<Accordion title="Entries went missing after changing the shard list" id="entries-went-missing-after-changing-the-shard-list">

Shard placement is a hash into a cumulative weight table. Changing the number of
shards or any weight moves the boundaries, so keys resolve to different shards
than the ones they were written to. The data is still on disk; nothing looks for
it there. Treat the change as a cache flush.

</Accordion>

<Accordion title="Cache dashboards are empty even though the CAS is busy" id="cache-dashboards-are-empty-even-though-the-cas-is-busy">

The OpenTelemetry cache instruments only fire from inside the `cache_metrics`
wrapper store, and no shipped config uses it. Wrap the store you want to observe
and give it a `cache_type`. The recording rules will start evaluating against
real series immediately afterwards.

</Accordion>

<Accordion title="A memory fast tier never seems to hold anything" id="a-memory-fast-tier-never-seems-to-hold-anything">

A memory store skips any write at or above its own `max_bytes`, logging rather
than erroring, and the blob is served from the backing store instead. If your
blobs are larger than the tier, raise `max_bytes` or put a `size_partitioning`
store in front so only small objects are routed there.

</Accordion>

<Accordion title="Throughput to a remote gRPC store plateaus well below the network" id="throughput-to-a-remote-grpc-store-plateaus-well-below-the-network">

`connections_per_endpoint` defaults to 1, so everything multiplexes over one
HTTP/2 connection. Raise it, and check `max_concurrent_requests`; a value of
zero is unlimited, but a low non-zero value queues requests silently.

</Accordion>

</Accordions>

<NextStep href="/operate/autoscaling-reference" title="An autoscaling reference deployment">
  Both scaling axes assembled into one deployment: the manifests, the probes,
  the drain path, and the reaction-time budget behind them.
</NextStep>

<NextStep href="/how-to/stores/compose-stores" kind="aside" title="Compose stores">
  The mechanics of the wrappers used here: what each one is, and how the graph
  is written.
</NextStep>

# https://docs.nativelink.com/operate/autoscaling-reference

**An autoscaling reference deployment**: The complete Kubernetes deployment that scales workers on queue depth without a human in the loop: every manifest, the reaction-time budget nobody publishes, and the five gaps in the shipped examples you have to close first.

**Who this is for:** you have run [scaling workers](/operate/scaling-workers) and
[scaling the CAS](/operate/scaling-cas) by hand, you trust the numbers, and you
want the pool to size itself. **What you'll have at the end:** a worker
deployment that grows and shrinks on queue depth, with probes, drain, and
disruption budgets wired up. **Time:** a day to assemble, a week of watching
before you trust it.

<Prerequisites>
  Workers and a CAS you have already sized by hand.
</Prerequisites>

The two scaling pages tell you *what* to scale on and *why*. This page is the
deployment: the manifests, the config changes they depend on, and the latency
between a build backing up and a pod existing.

<Callout type="info" title="Kubernetes deployments ship with Enterprise">
  The Helm charts for running NativeLink on Kubernetes are part of
  [NativeLink Enterprise](https://enterprise.nativelink.com). What follows is the
  reference for assembling worker autoscaling yourself on top of the Kustomize
  building blocks this repository ships.
</Callout>

## What ships, and what this page supplies [#what-ships-and-what-this-page-supplies]

Being blunt about this up front, because it changes how you read everything
below.

**The repository contains no autoscaling artifacts for NativeLink.** It has
no `HorizontalPodAutoscaler`, no KEDA `ScaledObject`, no VPA, and no metrics
adapter configuration for any NativeLink workload. The one HPA in the tree
scales the OpenTelemetry collector, and the one `PodDisruptionBudget` protects
the same collector. Neither has anything to do with your workers.

Everything on this page is written by these docs against the shipped
configuration schema and the shipped recording rules. It is a worked reference,
not a file you can point `kubectl apply` at from a checkout. Treat the YAML as
a starting shape you adapt, and read the [prerequisites](#five-gaps-you-close-first)
section before you assume any of it drops onto the shipped manifests unchanged.

<Callout type="warn" title="Read the reaction-time budget before you tune thresholds">
  The default pipeline puts up to two and a half minutes between a queue spike
  and an HPA that can see it. Most of that is fixable, and none of it is
  documented anywhere else. It is the next section, and it is the single thing
  most likely to make a correct-looking autoscaler behave badly.
</Callout>

## The reaction-time budget [#the-reaction-time-budget]

The signal an autoscaler reads is not emitted by NativeLink. It is derived,
several hops downstream, and each hop adds staleness. Here is the whole chain
with its shipped defaults.

<Mermaid>
{`flowchart LR
  A["NativeLink<br/>OTLP periodic export<br/><b>60s default</b>"] --> B["OTel collector<br/>batch processor<br/><b>up to 10s</b>"]
  B --> C["Prometheus<br/>scrape_interval<br/><b>15s</b>"]
  C --> D["recording rule<br/>nativelink:queue_depth<br/><b>60s group interval</b>"]
  D --> E["HPA sync loop<br/><b>15s default</b>"]`}
</Mermaid>

Worst case, an action that queues right after an export window closes is
invisible to the HPA for about two and a half minutes. Then the HPA's own
stabilization window applies on top.

Three of those five hops are yours to fix and two are not worth fixing.

**The export interval is the big one, and it is an environment variable.** The
binary builds its meter provider with a periodic exporter and never sets an
interval, so it takes the OpenTelemetry SDK default of 60 seconds. Set
`OTEL_METRIC_EXPORT_INTERVAL` on the scheduler pod to bring it down.

<SourceLink file="nativelink-util/src/telemetry.rs" symbol="init_tracing" />

**The recording-rule interval is the second.** `nativelink:queue_depth` lives in
the `nativelink_performance` group in
`deployment-examples/metrics/prometheus-recording-rules.yml`, and that group is
declared `interval: 60s`. The execution group next to it uses 30s. Either move
the rule into a group with a short interval, or skip the recording rule for
scaling and point the adapter at the raw series.

**The collector batch timeout** is 10 seconds in the shipped collector config
and is not worth chasing; it overlaps the scrape interval anyway.

<Callout type="warn" title="Do not simply drop the export interval to a few seconds">
  Every export is a full OTLP push of every instrument. At 5 seconds you are
  multiplying collector ingest and Prometheus sample volume by twelve for a
  signal you are going to smooth with a stabilization window regardless. Fifteen
  seconds is a good default and the point of diminishing returns for most pools.
</Callout>

## Five gaps you close first [#five-gaps-you-close-first]

The shipped Kubernetes manifests cannot support an HPA as they stand. Not
because of anything subtle, but because of five specific absences. Each one is a
short fix and all five are prerequisites.

| Gap | Where | Why it blocks autoscaling |
| --- | --- | --- |
| No `resources:` block on the worker container | `kubernetes/components/worker/worker.yaml` | A `Resource`-type HPA metric cannot compute utilization without requests. Scheduling is also unbounded. |
| `servers: []` in every worker config | `kubernetes/components/worker/worker.json5`, both compose worker configs | A worker process listens on nothing, so there is no port and no path for a readiness or liveness probe. |
| No `admin: {}` in any `kubernetes/` config | `kubernetes/nativelink/nativelink-config.json5` | The drain endpoint is never registered. Scale-in has no graceful path. |
| No probes on any NativeLink pod | all shipped manifests | Traffic reaches a CAS that is still doing its startup scan, and rollouts have nothing to gate on. |
| No PDB, `preStop`, or `terminationGracePeriodSeconds` | all shipped manifests | A scale-in or node drain kills workers mid-action on the default 30-second grace. |

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="AdminConfig" />

## The health endpoint does less than you expect [#the-health-endpoint-does-less-than-you-expect]

Before you write a readiness probe, understand what `/status` actually
measures, because the answer is surprising in both directions.

Health indicators are registered **only by leaf stores**: memory, filesystem,
S3, GCS, Azure, Redis, and Mongo. Every composing store (`existence_cache`,
`verify`, `fast_slow`, `size_partitioning`, `compression`, `dedup`, `shard`,
`completeness_checking`, `ref_store`) inherits the trait's default
`register_health`, which does nothing and does **not** recurse into its
children. The single exception is `cache_metrics`, which forwards registration
to its backend.

<SourceLink file="nativelink-util/src/store_trait.rs" symbol="StoreDriver" />

That has a sharp consequence. In the shipped Kubernetes config, `CAS_MAIN_STORE`
is an `existence_cache` at the top and `AC_MAIN_STORE` is a
`completeness_checking` at the top. Neither registers anything. The health
registry is empty, `/status` serialises an empty list, and the endpoint returns
`200 OK` unconditionally, including when the filesystem underneath is gone.

Now the other direction. When a leaf store *is* at the top of a composition,
what the check costs depends on which leaf it is. The filesystem store (the
compose example's `AC_MAIN_STORE` is a bare `filesystem`) only `stat`s its
`content_path`, with a two-second ceiling. GCS does a single `object_exists`
on a fixed never-present key, Redis sends `PING` under
`health_check_timeout_ms`, and Mongo pings the database. Memory, S3 and Azure
fall back to the trait's default, which is a full round trip: it generates
one megabyte of deterministic data, writes it with `update_oneshot`, calls
`has`, reads it back, and compares.

<SourceLink file="nativelink-util/src/store_trait.rs" symbol="check_health" />
<SourceLink file="nativelink-store/src/filesystem_store.rs" symbol="check_health" />

The digest is seeded from the store name, so it is the same blob every time and
the store does not grow. But against S3 or Azure that is a PUT, a HEAD and a
GET of a megabyte on every probe, per store, per replica. At a ten-second
probe period on three replicas that is eighteen megabytes a minute of pure
probe traffic and a billable request count to match.

`global.default_digest_size_health_check` lowers the payload. The round trip
itself you cannot turn off short of not enabling the health service. Whatever
the probe, the endpoint returns `503` if any registered indicator reports
`failed` or `timeout` (`HealthConfig.timeout_seconds`, default 5 s, bounds
each check) and `200` otherwise.

<Callout title="What to actually do">
  Enable `health: {}` and probe it, but size the probe honestly: a period of 30
  seconds or more on any process whose top-level store does the round trip,
  and `default_digest_size_health_check` reduced to something like 65536
  unless you specifically want the megabyte to exercise multipart paths. And
  do not read a `200` from a composed store as "the store works"; it means
  "nothing objected," which for most compositions is vacuous.
</Callout>

## Build the deployment [#build-the-deployment]

<Steps>

<li>

**Make the signal fresh.** Two changes, one on the scheduler pod and one in
Prometheus.

```yaml
# scheduler Deployment, container env
env:
  - name: OTEL_EXPORTER_OTLP_ENDPOINT
    value: http://otel-collector-collector.default.svc:4317
  - name: OTEL_EXPORTER_OTLP_COMPRESSION
    value: zstd
  - name: OTEL_METRIC_EXPORT_INTERVAL
    value: "15000"   # milliseconds; SDK default is 60000
```

```yaml
# a rule group of your own, evaluated fast enough to scale on
groups:
  - name: nativelink_autoscaling
    interval: 15s
    rules:
      - record: nativelink:queue_depth_fast
        expr: |
          sum by (instance_name) (
            nativelink_execution_active_count{execution_stage="queued"}
          )
```

Keep the shipped `nativelink:queue_depth` for dashboards. Scale on the fast
copy, summed without `execution_priority` so the HPA sees one series per
instance rather than one per priority band.

</li>

<li>

**Give the worker a server so it can be probed.** Every shipped worker config
has `servers: []`, which means the process listens on nothing at all. Add a
minimal server with only the health service on it.

```json5
servers: [{
  name: "worker_health",
  listener: {
    http: {
      socket_address: "0.0.0.0:50070",
    },
  },
  services: {
    health: {},
  },
}],
```

The worker's own stores register health the same way any other process's do.
In the shipped worker configs the named stores are a `fast_slow` and two
`grpc` stores, none of which registers an indicator, so the report is empty
and the endpoint answers `200` whenever the process is up. Read it as a
liveness signal ("the process is up and serving"), not as a statement about the
CAS.

</li>

<li>

**Name the worker after its pod.** `LocalWorkerConfig.name` is the prefix of the
worker id the scheduler assigns, in the form `{name}{uuidv6}`, and it is
shell-expanded, so the downward API can populate it.

```json5
workers: [{
  local: {
    name: "${POD_NAME:-worker}-",
    max_inflight_tasks: 16,
    // ...
  },
}],
```

```yaml
env:
  - name: POD_NAME
    valueFrom:
      fieldRef:
        fieldPath: metadata.name
```

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="LocalWorkerConfig" />

This is what makes a worker id traceable back to a pod, in logs and in the
`execution_worker_id` metric attribute. You need it for step 7.

</li>

<li>

**Add resources and probes to the worker.**

```yaml
containers:
  - name: nativelink-worker
    image: nativelink-worker
    ports:
      - name: health
        containerPort: 50070
    resources:
      requests:
        cpu: "4"
        memory: 8Gi
      limits:
        memory: 8Gi
    readinessProbe:
      httpGet:
        path: /status
        port: health
      periodSeconds: 30
      failureThreshold: 2
    livenessProbe:
      httpGet:
        path: /status
        port: health
      periodSeconds: 30
      failureThreshold: 4
```

Set `requests.cpu` to match the concurrency `max_inflight_tasks` implies, not to
the node size. A CPU limit is deliberately absent: a compile that gets throttled
at the cgroup boundary shows up as a slow action rather than a failure, which is
harder to diagnose than an occasional noisy neighbour.

</li>

<li>

**Enable the admin service on the scheduler.** It is a service entry, and it is
absent from every `kubernetes/` config. Put it on the worker-API server block,
which is already on its own port and its own permission boundary.

```json5
{
  listener: {
    http: {
      socket_address: "0.0.0.0:50061",
    },
  },
  services: {
    worker_api: {
      scheduler: "MAIN_SCHEDULER",
    },
    admin: {},
    health: {},
  },
},
```

<Callout type="warn" title="The admin path has no authentication">
  The drain route is plain HTTP with no auth of any kind. Keep it on the
  internal worker-API port, keep that port off any Ingress, and restrict it with
  a NetworkPolicy to the namespace that runs your drain job.
</Callout>

</li>

<li>

**Wire the external metric to an HPA.** Kubernetes cannot read Prometheus
directly; something has to serve the `external.metrics.k8s.io` API. Nothing in
the repository does this, so this is a Prometheus Adapter configuration written
for the purpose.

```yaml
# prometheus-adapter values
rules:
  external:
    - seriesQuery: 'nativelink:queue_depth_fast'
      resources:
        overrides:
          namespace: {resource: "namespace"}
      name:
        matches: "^nativelink:queue_depth_fast$"
        as: "nativelink_queue_depth"
      metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>})'
```

```yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: nativelink-worker
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: nativelink-worker
  minReplicas: 3
  maxReplicas: 60
  metrics:
    - type: External
      external:
        metric:
          name: nativelink_queue_depth
        target:
          type: AverageValue
          averageValue: "8"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 30
      policies:
        - type: Percent
          value: 100
          periodSeconds: 60
        - type: Pods
          value: 8
          periodSeconds: 60
      selectPolicy: Max
    scaleDown:
      stabilizationWindowSeconds: 600
      policies:
        - type: Pods
          value: 2
          periodSeconds: 120
```

`AverageValue: "8"` means "aim for eight queued actions per worker": the HPA
divides the metric by the replica count, so the target is a per-worker backlog
rather than an absolute queue length. Pick the number from the per-worker
throughput you measured in [scaling workers](/operate/scaling-workers); eight is
a placeholder, not a recommendation.

The asymmetry between the windows is deliberate and matters more than the
thresholds. Scaling up fast costs money. Scaling down fast costs work, because
every worker you remove mid-action produces retries.

</li>

<li>

**Drain on scale-in, as far as the API allows.** This is the part where honesty
is more useful than a clean recipe.

The drain endpoint takes a worker id the pod does not know at startup; the
scheduler generates it at registration and returns it over the wire (the
worker logs it once, in its `Worker registered with scheduler` line).

```
POST {admin_path}/scheduler/{scheduler_name}/set_drain_worker/{worker_id}/1
```

The second path segment is the scheduler's `name` from the `schedulers` list
(`MAIN_SCHEDULER` in the shipped config), not the REAPI instance name. Because
you set `name` to the pod name in step 3, the id is discoverable: it is the
`execution_worker_id` label on the execution series the scheduler emits for
that worker's actions, prefixed with the pod name.
`nativelink_execution_active_count` does not carry that label (it is
attributed by stage only); `nativelink_execution_stage_transitions_total` and
`nativelink_execution_completed_count_total` do. A `preStop` hook can look it
up and call drain.

```yaml
terminationGracePeriodSeconds: 1800
lifecycle:
  preStop:
    exec:
      command:
        - /bin/sh
        - -c
        - |
          WID=$(wget -qO- "http://prometheus:9090/api/v1/query?query=\
          nativelink_execution_stage_transitions_total%7Bexecution_worker_id%3D~%5C%22\
          ${POD_NAME}-.%2A%5C%22%7D" \
            | sed -n 's/.*"execution_worker_id":"\([^"]*\)".*/\1/p' | head -1)
          if [ -n "$WID" ]; then
            wget -qO- --post-data='' \
              "http://nativelink:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/$WID/1"
          fi
          sleep 1500
```

Three caveats, all of which you should weigh before adopting this.

It only finds an id once that worker has reported at least one action to the
scheduler since the scheduler started; the counter it queries is cumulative,
so a worker that ran something an hour ago is still findable. A worker that
has never run anything has no series, the lookup returns nothing, and the
hook falls through, which is harmless, because such a worker has nothing to
drain.

It depends on Prometheus being reachable and current from inside a terminating
pod. If that is unacceptable in your environment, drop the lookup and rely on
`terminationGracePeriodSeconds` alone.

The `sleep` is what actually holds the pod open while in-flight actions finish;
the drain call only stops new work arriving. Size the grace period to your
longest action, and cross-check it against `max_action_timeout_s` on the
worker.

<Callout title="Without any of this, work is still not lost">
  A worker that disappears mid-action has its actions requeued with worker
  affinity cleared, up to `max_job_retries`. Drain turns a retry storm into a
  clean handover; it is not the difference between correct and incorrect.
</Callout>

</li>

<li>

**Protect the pool from everything that isn't the HPA.** Voluntary disruptions
(node upgrades, cluster autoscaler consolidation) will otherwise take workers
out in parallel with your scale-in.

```yaml
apiVersion: policy/v1
kind: PodDisruptionBudget
metadata:
  name: nativelink-worker
spec:
  minAvailable: 50%
  selector:
    matchLabels:
      app: nativelink-worker
---
# in the worker pod spec
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: ScheduleAnyway
    labelSelector:
      matchLabels:
        app: nativelink-worker
```

The shipped worker manifest has neither, so `replicas: 3` can and does land
three workers on one node.

</li>

<li>

**Leave the CAS out of the autoscaler.** The HPA scales workers and nothing
else. A CAS process holds per-process state that makes a second replica a
project rather than a replica count; [scaling the CAS](/operate/scaling-cas)
covers which state and why. What the CAS needs from this deployment is a
readiness probe with enough patience for its startup scan.

```yaml
readinessProbe:
  httpGet:
    path: /status
    port: 50061
  periodSeconds: 10
  failureThreshold: 60      # ten minutes of scan
  initialDelaySeconds: 10
```

The filesystem store walks its entire content directory before the process binds
any socket, so during a restart the probe gets connection-refused rather than a
`503`. A short `failureThreshold` turns a slow restart into a crash loop.

</li>

</Steps>

<VerifyBlock>

- `kubectl get --raw "/apis/external.metrics.k8s.io/v1beta1/namespaces/default/nativelink_queue_depth"`
  returns a value, and it changes within about thirty seconds of a build
  starting. If it is flat or missing, the adapter rule is wrong.
- `kubectl describe hpa nativelink-worker` shows the current metric and a recent
  scaling event, not `unable to fetch metrics`.
- Starting a large build grows the pool, and the queue drains rather than
  plateauing. A queue that stays deep while replicas sit at `maxReplicas` means
  the ceiling, not the policy, is the constraint.
- After the build, replicas fall back to `minReplicas` in one or two steps
  spread over minutes, not immediately, and not in a single jump.
- Scale-in does not coincide with a spike in retried actions. If it does, the
  `preStop` hook isn't finding the worker id.
- `kubectl drain` on a worker node evicts at most half the pool at once.
- Restarting the CAS produces a pod that stays `NotReady` for the duration of
  its scan and then goes `Ready` once, not a `CrashLoopBackOff`.

</VerifyBlock>

## What this deployment does not do [#what-this-deployment-does-not-do]

Worth stating so the gaps are choices rather than surprises.

**It does not scale the CAS or the scheduler.** Both are single-replica in this
reference. The scheduler's default backend is in-memory, so a second scheduler
is a second independent queue, not a bigger one.

**It does not do predictive or scheduled scaling.** Queue depth is reactive by
construction: pods start after work is already waiting. If your builds arrive on
a known schedule, a `CronJob` that raises `minReplicas` before the wave beats any
metric-driven policy.

**It does not give you node capacity.** An HPA creates pods; something has to
create nodes. Cluster Autoscaler or Karpenter underneath, sized so that
`maxReplicas` is actually reachable.

**It does not survive a move to the Redis scheduler backend.**
`execution.active.count`, the series this whole page depends on, is emitted
only by the in-memory awaited-action store.
Under `experimental_backend`, the queue-depth series is not emitted at all and
the HPA has nothing to read.

## When it doesn't work [#when-it-doesnt-work]

<Accordions>

<Accordion title="The HPA reports `unable to fetch metrics` for the external metric" id="the-hpa-reports-unable-to-fetch-metrics-for-the-external-metric">

Work backwards along the chain. Query the recording rule in Prometheus directly;
if it returns nothing, the problem is upstream of Kubernetes entirely and the
adapter is fine. If the rule returns data but the adapter doesn't, the usual
cause is the `resources.overrides` block: external metrics still need a
namespace association, and a series with no `namespace` label cannot be matched
to one. Add a `namespace` label in the recording rule or via collector
attributes.

</Accordion>

<Accordion title="The pool scales up long after the build has started" id="the-pool-scales-up-long-after-the-build-has-started">

Sum the reaction-time budget for your actual configuration. The two defaults
that dominate are the 60-second OTLP export interval and the 60-second interval
on the shipped `nativelink_performance` rule group. If you skipped step 1, those
two alone put you two minutes behind.

</Accordion>

<Accordion title="Replicas oscillate (up, down, up) during one build" id="replicas-oscillate--up-down-up--during-one-build">

Almost always a `scaleDown` stabilization window that is too short relative to
how bursty the graph is. Large builds have natural lulls between phases;
`scaleDown.stabilizationWindowSeconds: 600` exists to ride through them. If it
still oscillates at ten minutes, the target `averageValue` is close to your
steady-state backlog, so small changes cross the threshold in both directions.
Raise the target.

</Accordion>

<Accordion title="Scale-in produces a burst of retried actions every time" id="scale-in-produces-a-burst-of-retried-actions-every-time">

The `preStop` hook is not resolving a worker id. Check that `POD_NAME` reaches
the config: with the `${POD_NAME:-worker}` form above, a missing variable
falls back to the literal default rather than failing loudly, and every worker
ends up with the same prefix. Confirm by looking at the `execution_worker_id` label values in
Prometheus: they should each begin with a distinct pod name.

</Accordion>

<Accordion title="Workers restart repeatedly under load with no obvious error" id="workers-restart-repeatedly-under-load-with-no-obvious-error">

The liveness probe. `/status` on a process whose registered leaf store is
memory, S3 or Azure does a full write, has and read round trip per store, and
under load that queues behind real traffic; each check is also cut off at
`HealthConfig.timeout_seconds` (default 5 s) and reported as `timeout`, which
the endpoint turns into a `503`. With `periodSeconds: 30` and
`failureThreshold: 4` you have two minutes of grace; anything much tighter
kills healthy workers whose only problem is a congested store. Lower
`global.default_digest_size_health_check` from its one-megabyte default as
well. (A worker on the shipped configs registers no indicator at all, so if
its probe fails the process itself is not answering.)

</Accordion>

<Accordion title="The readiness probe passes but builds fail against that pod" id="the-readiness-probe-passes-but-builds-fail-against-that-pod">

Expected, if the store at the top of the composition is a wrapper. An
`existence_cache` or `completeness_checking` at the top registers no health
indicator, the report is empty, and the endpoint returns `200` regardless of
what is underneath. Either accept the probe as a liveness check only, or restructure
so that a leaf store (or a `cache_metrics` wrapper, which forwards registration)
sits where the registry will see it.

</Accordion>

<Accordion title="Pods are `Pending` while the queue keeps growing" id="pods-are-pending-while-the-queue-keeps-growing">

The HPA is doing its job and the cluster has no room. Check the node autoscaler
and whether `resources.requests` on the worker fits the instance types it can
provision. A worker requesting four CPUs and 8 GiB will never schedule on a
two-CPU node pool, and the HPA has no visibility into that.

</Accordion>

<Accordion title="Scale-down never happens even when the pool is idle" id="scale-down-never-happens-even-when-the-pool-is-idle">

Check whether the metric is going to zero or merely to a small non-zero value.
`nativelink_execution_active_count` is an up-down counter; a stuck or abandoned
operation can leave a queued action counted indefinitely, which pins the HPA
above `minReplicas`. Compare the metric against the actual scheduler state before
assuming the policy is wrong.

</Accordion>

<Accordion title="Everything works in staging and the adapter fails in production" id="everything-works-in-staging-and-the-adapter-fails-in-production">

Usually multiple instance names. The shipped recording rule sums by
`instance_name`, so a cluster serving several instances produces several series,
and an adapter rule without an aggregation returns an ambiguous match. The
`max(<<.Series>>{<<.LabelMatchers>>})` in the `metricsQuery` above collapses
them; if you want per-instance pools, give each its own HPA and select on
`instance_name` in the label matchers instead.

</Accordion>

</Accordions>

<NextStep href="/operate/observability" title="Observability">
  The pipeline this deployment reads from: how metrics get from the binary to
  Prometheus, what the catalog contains, and what the shipped dashboards show.
</NextStep>

<NextStep href="/operate/tuning" kind="aside" title="Tuning">
  The lever table. Once the pool sizes itself, tuning is what decides how much
  each of those pods is worth.
</NextStep>

# https://docs.nativelink.com/operate/runbooks

**Runbooks**: What to do when a NativeLink deployment is on fire: Redis failover, a worker OOM-killed, the disk full, and the queue backpressured. Each entry is a symptom, the signal that confirms it, the immediate action, and the change that stops it recurring.

**Who this is for:** you are on call and something is broken right now.
**What you'll have at the end:** for each of the four incidents NativeLink
actually produces, a confirming signal, a stop-the-bleeding action, and the
configuration change that prevents a repeat. **Time:** read once now, so you do
not read it for the first time at 03:00.

<Prerequisites>
  A deployment other people depend on.
</Prerequisites>

These are incident procedures. They deliberately do not explain how NativeLink
works ([Architecture](/explanations/architecture) does that) and they do not
tune anything. When you have stopped the bleeding and want to make the system
faster rather than merely alive, go to [Tuning](/operate/tuning).

If you are not yet sure *which* incident you are having, start at
[Troubleshooting](/operate/troubleshooting), which indexes symptoms to causes.
This page assumes you already know.

<Callout type="info" title="Two things to have open before you start">

  The scheduler's `INFO` log. Most of NativeLink's diagnostic output is at
  `INFO`, and the shipped Compose example defaults to `warn`. If you are at
  `warn` you will see a hung system and nothing else. `RUST_LOG=info` is the first move in
  almost every procedure below.

  A metrics pipeline. Several of these incidents are only visible as a rate
  change. [Observability](/operate/observability) brings one up in about ten
  minutes; doing it during an incident is a bad trade, so do it beforehand.

</Callout>

## A worker is OOM-killed [#a-worker-is-oom-killed]

The most common production incident, and the one with the most misleading
first symptom: your developers report a *build* failure, not an
*infrastructure* failure.

### Confirm it [#confirm-it]

Two distinct failures land here and they need different responses. Work
out which one you have before doing anything.

**The action's child process was killed.** The worker survived. The action
reports **exit code 9**, and the worker logs:

```text
Runner subprocess terminated by signal (no exit code); likely OOMKilled or
externally killed. If this repeats for the same action, raise
`workers.specs[*].resources.limits.memory` or shrink the action's concurrency.
```

That exit code is a fixed sentinel: every signal collapses to 9, so you cannot
tell SIGKILL from SIGTERM from the code alone. The `warn!` line above is the
only thing that tells you it was a signal at all. From Bazel's side this looks
like an ordinary action failure with a non-zero exit status, so **Bazel will
not treat it as an infrastructure error and will not retry it**. On the
scheduler it lands in
`nativelink_execution_completed_count_total{execution_result="failure"}` like
any other non-zero exit; there is no exported metric that singles out
signal deaths, so the log line is what you alert on.

**The worker process itself was killed.** The scheduler notices after
`worker_timeout_s` (default 5 seconds, checked once a second), evicts it, and
re-queues its in-flight actions. In this tree a disconnect *counts as an
attempt*, so after `max_job_retries` the client finally sees:

```text
Worker disconnected repeatedly while executing this action (4 > 3 attempts);
the runner likely OOMKilled or the pod was evicted. for operation_id: ...,
maybe_worker_id: ...
```

That string is the single best diagnostic for a memory-pressured worker fleet.
If you see it, you are in this case, not the first one.

<Callout type="warn" title="Peak memory is a Linux-only measurement">
  NativeLink samples per-action peak RSS only on Linux. On macOS workers
  `resource_usage` is `None`; no memory data is reported at all, so the only
  evidence you will have is the exit code and the log line.
</Callout>

### Stop the bleeding [#stop-the-bleeding]

Drain the affected worker rather than killing it, so its in-flight actions
finish instead of being destroyed:

```sh
curl -X POST \
  "http://127.0.0.1:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/${WORKER_ID}/1"
```

The path segment after `scheduler/` is the scheduler's `name` from the
`schedulers` list, not a REAPI instance name. The endpoint only exists on a
listener with `admin: {}` in its services (the shipped Kubernetes config has
none); put it on the worker-API listener. It has no authentication; reach it
over a private path only. See
[Security hardening](/operate/security-hardening#the-admin-endpoint-has-no-authentication).

Undrain with a trailing `0` when the host is healthy again. A drained worker
appears in the scheduler's matching diagnostics as:

```text
Worker <id> cannot accept work: is_paused=false, is_draining=true, inflight=0/0
```

### Make it stop happening [#make-it-stop-happening]

| Change | Where | Effect |
| --- | --- | --- |
| Lower `max_inflight_tasks` | worker config | Caps concurrent actions per worker. Defaults to `0`, which means **infinite**; one worker will happily run every action you send it. This is the single most effective knob. |
| Raise the container memory limit | Kubernetes `resources.limits.memory`, or the host | Named explicitly by the worker's own warning. |
| Reduce per-action concurrency | Your build's toolchain flags | A single `-j` in a wrapped compiler multiplies with `max_inflight_tasks`. |

NativeLink itself has **no memory configuration**. No `max_memory`, no
per-action memory cap, nothing. Memory is bounded only indirectly, by store
eviction policies and by whatever the container runtime enforces. Do not go
looking for a config field; it does not exist.

## The disk fills up [#the-disk-fills-up]

### Confirm it [#confirm-it-1]

Writes start failing with an OS-level message wrapped in a NativeLink err_tip:
`Failed to write data into filesystem store`, `Failed to flush in filesystem
store`, `Failed to sync in filesystem store`.

The gRPC code will be `Unknown`, **not** anything storage-shaped. `ENOSPC`
arrives as `ErrorKind::StorageFull`, which NativeLink's `io::ErrorKind` to
gRPC-code map does not name, so it falls to the catch-all arm. No
ENOSPC-specific handling exists anywhere in the tree.

<SourceLink file="nativelink-error/src/lib.rs" symbol="into_code" />

That has an unpleasant consequence: `Unknown` is **retryable**. A full disk
gets retried through the entire retry budget before it fails, so the first
thing you notice is usually latency, not errors.

<Callout type="info" title="At least it does not corrupt the cache">
  The filesystem store issues an explicit flush before renaming a file into the
  content path, precisely because tokio defers write errors to the next write
  or flush. Without it, a truncated file would be renamed in and served as a
  valid blob, so a disk-full event produces errors, not silent corruption.
</Callout>

### The signal to look for first [#the-signal-to-look-for-first]

```text
Bytes: 41231872418 of unlimited;
```

If your eviction snapshot says **"of unlimited"**, no size-based eviction is
configured, and this incident was inevitable. `eviction_policy` is an
`Option` on the filesystem store, and every field in it defaults to `0`, which
means *never evict*. The config's own doc comment is blunt about it: failure to
set this value causes items to never be removed.

### Stop the bleeding [#stop-the-bleeding-1]

Set `max_bytes` and restart the process. Eviction is only ever invoked on map
mutation (on insert and remove), so it runs as soon as traffic resumes.

```json5

{
  "filesystem": {
    "content_path": "/var/lib/nativelink/content",
    "temp_path": "/var/lib/nativelink/temp",
    "eviction_policy": {
      "max_bytes": 50000000000,
      "evict_bytes": 5000000000
    }
  }
}
```

`evict_bytes` creates a low watermark: once `max_bytes` is reached, eviction
continues down to `max_bytes - evict_bytes` rather than evicting one entry per
insert forever. Set it to roughly 10% of `max_bytes`.

<Callout type="warn" title="Nothing consults free space">
  Eviction compares the store's *own accounting* against `max_bytes`. It never
  looks at the filesystem's actual free space, and a failed write does not
  trigger eviction. If you set `max_bytes` larger than the volume, or leave it
  unset, the store will fill the disk and then keep erroring indefinitely.
</Callout>

### Size the volume for twice `max_bytes` [#size-the-volume-for-twice-max_bytes]

The filesystem store keeps an executable-variant directory
(`{content_path}.exec`) alongside the content path, and **that directory is
invisible to `max_bytes`**. A variant is deleted when its primary entry is
evicted, which bounds the total at roughly `2 * max_bytes` in the worst case
(every blob also held as an executable). The variants are only ever created on
a worker, for executable inputs hardlinked out of its fast store, so this
applies to worker fast tiers rather than to a CAS server's store.

<SourceLink file="nativelink-store/src/filesystem_store.rs" symbol="ExecutableVariantRemover" />

Provision accordingly: if you want a 50 GB worker cache, give it a 100 GB
volume.

### Two layout footguns [#two-layout-footguns]

`temp_path` must be on the **same block device** as `content_path`. If it is
not, the atomic rename becomes a copy: slower, and it doubles peak space
during every write.

Everything in `temp_path` is deleted at startup, and the content path is
rescanned and re-accounted. A crash mid-write leaves a partial temp file that
is reclaimed on the next boot, but only on the next boot. A process that
crashes repeatedly without a clean restart accumulates.

## Redis fails over [#redis-fails-over]

### What actually happens [#what-actually-happens]

Both Sentinel and Cluster modes are supported; the default is `standard`. On a
Sentinel failover the demoted master answers `-READONLY`, which NativeLink
classifies as retryable alongside dropped connections, refusals and IO errors.

Each retryable failure reconnects and sleeps 100 ms, up to 5 attempts.
Reconnects are serialised behind a dedicated mutex so a failover storm does not
have every in-flight request stampeding the new master.

**Net behaviour: about 500 ms of retrying, then the operation fails with the
underlying Redis error. The process does not exit.**

<Callout type="warn" title="A liveness probe will not catch this">
  NativeLink serves errors during a Redis outage rather than crashing. Process
  liveness is unchanged and the port still accepts connections. The health
  endpoint only notices if the Redis store is itself a registered indicator (a
  leaf at the top of a named store), in which case its `PING` check fails and
  `/status` returns `503`; wrapped under `fast_slow` or `existence_cache` it
  registers nothing. Alert on the error rate, not on restarts.
</Callout>

### What breaks depends on your backend [#what-breaks-depends-on-your-backend]

| Scheduler backend | Impact of a Redis outage |
| --- | --- |
| `memory` (the default when unset) | Nothing in the scheduler. Only Redis-backed *stores* fail, serving `Unavailable` (or `DeadlineExceeded` on a timed-out command) after roughly 500 ms of retries. |
| `redis` | The awaited-action database is unavailable: no new actions can be tracked and existing subscriptions stop delivering. The process stays up and keeps accepting connections. |

### Stop the bleeding [#stop-the-bleeding-2]

If you are on the `memory` backend, this is a cache-availability incident, not
a scheduler incident. Builds degrade to local execution and cache misses. Fix
Redis at your leisure.

If you are on the `redis` backend, builds hang. Failing over Redis is the fix;
there is no NativeLink-side mitigation.

### Configuration worth checking now [#configuration-worth-checking-now]

| Field | Default | Note |
| --- | --- | --- |
| `max_client_permits` | 500 | A semaphore, not a limit that errors. Exceeding it **blocks**. If Redis is slow, requests pile up here silently. |
| `command_timeout_ms` | 10000 | |
| `connection_timeout_ms` | 3000 | |
| `health_check_timeout_ms` | 4000 | |
| `connection_pool_size` | 3 | |
| `retry.max_retries` |  | A configured `0` is coerced to `1`. |

`response_timeout_s` and `connection_timeout_s` are **deprecated**;
`response_timeout_s` is ignored entirely. Both log a warning at startup. Setting
both `connection_timeout_s` and `connection_timeout_ms` is a startup error.

<Callout type="info" title="The scheduler backend expects a standard-mode store">
  The Redis scheduler backend downcasts to a concrete standard-connection type.
  Pointing `experimental_backend: redis` at a Cluster- or Sentinel-mode store
  can fail at startup with `Could not downcast to redis store in
  RedisAwaitedActionDb::new`. Use a separate standard-mode store for the
  scheduler if you run Redis in cluster mode for your CAS.
</Callout>

## The queue backpressures and never recovers [#the-queue-backpressures-and-never-recovers]

The subtlest incident of the four, because nothing errors and nothing restarts.
Actions re-queue forever.

### The mechanism [#the-mechanism]

`ResourceExhausted` is NativeLink's backpressure code. When a worker returns
it for an action, two things happen: the action goes back to `Queued` and
(this is the important part) **its attempt counter is not incremented**. If
the worker still has other actions in flight it is also marked `is_paused`,
and that flag clears as soon as any of those actions completes.

<SourceLink file="nativelink-scheduler/src/simple_scheduler_state_manager.rs" symbol="due_to_backpressure" />
<SourceLink file="nativelink-scheduler/src/api_worker_scheduler.rs" symbol="update_action" />

That is deliberate. Backpressure is not the action's fault, so it should not
burn a retry. But it means a worker that returns `ResourceExhausted`
*permanently* produces an invisible infinite loop: the scheduler hands it the
action, it rejects it, the action requeues without counting, and the next
matching pass hands it over again. `max_job_retries` does not protect you,
because no attempt is ever counted, and an idle worker is never paused, so
nothing takes it out of the candidate set.

The usual cause is `experimental_precondition_script`, the intended hook for
"do not send me work, my disk is filling up". If the script's condition never
clears, the loop never ends.

### Confirm it [#confirm-it-2]

```sh
RUST_LOG=info
```

then look at the scheduler's periodic pass: the action keeps appearing in
`Oldest actions in state` as `Queued`, its age grows, and it never reaches a
retry limit. The worker itself is quiet; the script's exit status is logged
only at `TRACE` (`Preconditions script returned`), so `RUST_LOG=info` on the
worker shows nothing. If the worker still holds other work you will also see,
in the scheduler's matching diagnostics:

```text
Worker <id> cannot accept work: is_paused=true, is_draining=false, inflight=1/0
```

`is_paused=true` is only ever set while `inflight` is non-zero; a worker with
nothing in flight that is still refusing work does not appear here at all,
because it is, as far as the scheduler can tell, a perfectly available
worker.

### Stop the bleeding [#stop-the-bleeding-3]

Fix or remove the precondition script and restart the worker. Nothing can
clear `is_paused` from the outside; it is set and cleared by the scheduler
in response to what the worker returns.

If the precondition script is doing its job correctly, and the host really is
out of disk, then this is the disk-full incident wearing a different mask. Go
to [The disk fills up](#the-disk-fills-up).

### Related limits that block rather than reject [#related-limits-that-block-rather-than-reject]

Several NativeLink limits queue instead of erroring, which is why "hung" is a
more common symptom than "failed":

| Limit | Default | Behaviour at the limit |
| --- | --- | --- |
| Open file descriptors | 24576 (`max_open_files`) | **Blocks on a semaphore.** Never produces `EMFILE`. Permits are 80% of the *achieved* `nofile` limit (19660 by default), which in a container with a hard cap may be far below what you configured. |
| `max_concurrent_requests` (gRPC scheduler) | unlimited | Queues, does not reject. |
| `max_client_permits` (Redis) | 500 | Blocks. |
| `max_concurrent_writes` (filesystem) | unlimited | Blocks when set. |

The FD limit is the one that catches people. If the kernel's hard limit caps
the raise below what you asked for, the semaphore silently shrinks to 80% of
what it actually got, and the only evidence is a startup warning (if the
raise call fails outright, the semaphore stays at 80% of 24576 regardless of
what the kernel allows):

```text
The new open file limit (1024) is below the recommended value of 24576.
Consider raising max_open_files.
```

## Restarting the scheduler [#restarting-the-scheduler]

Not an incident so much as a procedure that behaves worse than you expect.

Worker reconnect is a **flat 0.5-second retry with no exponential backoff, no
jitter, no cap and no configuration field**. A scheduler outage therefore
produces a 2 Hz reconnect storm from every worker, for the whole outage.

Worse, on reconnect each worker calls `kill_all()` and destroys every action it
had in flight, to avoid resource-locking itself against the new work the
scheduler will send. A scheduler restart is disruptive to in-flight builds by
design.

Before restarting a scheduler with running work, drain each worker (see
[above](#a-worker-is-oom-killed)) and wait for `inflight=0`.

Log lines to grep for during a restart:

```text
Worker registered with scheduler
Worker disconnected from scheduler
Actions in transit did not reach zero before we disconnected from the scheduler
```

The third means the 10-second drain window expired with work still in transit.

<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="CONNECTION_RETRY_DELAY_S" />
<SourceLink file="nativelink-util/src/evicting_map.rs" symbol="should_evict" />
<SourceLink file="nativelink-store/src/redis_store.rs" symbol="is_retryable_redis_error" />

## What's next [#whats-next]

<NextStep href="/operate/troubleshooting" title="Troubleshooting">
  The symptom-to-cause index, for when you do not yet know which incident you
  are in.
</NextStep>

<NextStep href="/operate/tuning" title="Tuning" kind="aside">
  Once it is alive again: the levers that make it fast, and the ones that do
  nothing.
</NextStep>

<NextStep href="/operate/observability" title="Observability" kind="aside">
  Set up the metrics pipeline before the next incident, not during it.
</NextStep>

# https://docs.nativelink.com/operate/troubleshooting

**Troubleshooting**: A symptom-to-cause-to-fix index for NativeLink, anchored on the error strings the system actually emits, including the ones that say nothing useful, and the failures that produce no error at all.

**Who this is for:** something is wrong and you have a symptom but not a cause.
**What you'll have at the end:** the cause, and either a fix or a pointer to
the procedure that fixes it. **Time:** find your row, follow it.

<Prerequisites>
  A deployment that is misbehaving.
</Prerequisites>

Search this page for the error string you have. If your symptom is a *hang*
rather than an error, start at
[Actions queue forever and workers sit idle](#actions-queue-forever-and-workers-sit-idle);
NativeLink's most common failure mode produces no error string at all.

If you already know which incident you are having and need the procedure rather
than the diagnosis, go to [Runbooks](/operate/runbooks).

## The index [#the-index]

| Symptom | Likely cause | Where to go |
| --- | --- | --- |
| Build hangs, no action ever starts, no error | Platform property *value* mismatch | [Actions queue forever](#actions-queue-forever-and-workers-sit-idle) |
| `Unknown platform property '<name>'` immediately | Property *name* missing from the scheduler's `supported_platform_properties` | [The name/value asymmetry](#actions-queue-forever-and-workers-sit-idle) |
| `FAILED_PRECONDITION` mentioning fast or slow store | Workers do not share CAS storage | [Actions fail with a CAS miss](#actions-fail-with-a-cas-miss) |
| `not found in filesystem store ... evicted due to cache pressure` | `max_bytes` too small | [Actions fail with a CAS miss](#actions-fail-with-a-cas-miss) |
| Action fails with exit code 9 | Child process killed by a signal, usually the OOM killer | [Runbooks: OOM](/operate/runbooks#a-worker-is-oom-killed) |
| `Worker disconnected repeatedly while executing this action` | The worker *process* is dying, not the action | [Runbooks: OOM](/operate/runbooks#a-worker-is-oom-killed) |
| `Failed to write data into filesystem store` | Disk full; there is no ENOSPC-specific error | [Runbooks: disk](/operate/runbooks#the-disk-fills-up) |
| Actions re-queue forever, retries never exhaust | `ResourceExhausted` backpressure does not consume a retry | [Runbooks: backpressure](/operate/runbooks#the-queue-backpressures-and-never-recovers) |
| `Operation timed out having no more clients listening` | The client gave up. Says nothing about *why* | [A misleading timeout](#the-timeout-message-that-tells-you-nothing) |
| `Action timeout of N seconds is greater than the maximum allowed` | Client asked for more than `max_action_timeout_s` | [Timeouts](#timeouts) |
| `Command '<cmd>' timed out after N seconds` | The action itself exceeded its timeout | [Timeouts](#timeouts) |
| Worker logs cycle roughly twice a second | Reconnect storm, a flat 0.5 s retry | [Worker connection churn](#worker-connection-churn) |
| Setting `max_job_retries: 0` did not stop retries | Zero is read as unset | [Zero means default](#zero-means-default) |
| Everything slows down, nothing errors | FD semaphore exhaustion, or another blocking limit | [Limits that block](#limits-that-block-instead-of-erroring) |
| Redis errors but the process stays up | By design: NativeLink serves errors during a Redis outage | [Runbooks: Redis](/operate/runbooks#redis-fails-over) |
| `'cas_store': '<name>' does not exist` at startup | Store name typo in the config | [Startup failures](#startup-failures) |

## Actions queue forever and workers sit idle [#actions-queue-forever-and-workers-sit-idle]

The single most important asymmetry in NativeLink:

<Callout type="warn" title="A bad property name errors instantly. A bad property value hangs forever.">

  If your action names a platform property that is **not** in the scheduler's
  `supported_platform_properties`, you get an immediate `InvalidArgument`:

  ```text
  Unknown platform property 'os_famly'
  ```

  If the property name is known but **no worker advertises a satisfying
  value**, the scheduler's matching pass returns without doing anything.
  The action stays `Queued`. Nothing escalates, nothing times out, nothing is
  logged above `INFO`.

</Callout>

That second case is not a bug so much as a consequence of how matching works
("no worker matches yet" is indistinguishable from "no worker matches ever"),
but its operational effect is a build that hangs with no diagnostic.

`max_action_executing_timeout_s` does not help, because it applies only to the
`Executing` state and the action never reaches it. The only thing that ever
ends the wait is `client_action_timeout_s` (default 60 s) noticing the client
has stopped listening, which produces a message about
[clients listening](#the-timeout-message-that-tells-you-nothing) and says
nothing about properties.

### Diagnose it [#diagnose-it]

Raise the log level. Every useful diagnostic here is at `INFO`, and the shipped
examples run at `warn`.

```sh
RUST_LOG=info
```

Then match what you see:

| Log line | Meaning |
| --- | --- |
| `All workers are fully allocated` | No worker can accept work right now (all at `max_inflight_tasks`, paused or draining). Not a config problem; add workers. Checked first, before properties. |
| `No workers in capability index match required properties` | Nothing you have can run this. A property value or a missing worker capability. |
| `Worker <id> cannot accept work: is_paused=…, is_draining=…, inflight=…` | This specific worker matched on properties but is excluded, and the flags say why. |
| `No workers matched!` | The pass finished with no candidate. |
| `Property mismatch on worker property <p>. <worker> < <wanted>` | A `minimum` property the worker does not meet. |
| `Property mismatch on worker property <p>. <worker> != <wanted>` | An `exact` property that differs. |
| `Property missing on worker property <p>` | The worker never advertised it at all. |

All of those lines, and the property mismatch lines below, are only written
during the periodic full-logging pass, which is controlled by
`worker_match_logging_interval_s`, default 10 seconds. `-1` or `0` disables
it; anything invalid logs an error and disables it. The same pass emits
`Oldest actions in state`, capped at five per stage, the intended "why is my
queue stuck" tool.

### Property types that constrain nothing [#property-types-that-constrain-nothing]

| Type | Matching rule |
| --- | --- |
| `minimum` | worker value ≥ requested |
| `exact` | equality |
| `priority` | **always true; constrains nothing** |
| `ignore` | **always true; constrains nothing** |
| unknown | equality |

A scheduler key configured as `priority` when it should be `exact` or `minimum`
is the classic version of this bug, and it fails in the *opposite* direction:
actions get scheduled onto workers that cannot run them. See
[Scaling workers](/operate/scaling-workers) for the fleet-shaped version of
this problem and [Platform properties](/remote-execution/platform-properties)
for the full matching semantics.

The converse hazard is documented in the config itself: if a worker fails to
advertise a property the scheduler requires (`cpu_arch`, say), the scheduler
will never send it any work at all.

## Actions fail with a CAS miss [#actions-fail-with-a-cas-miss]

### The string you will actually see [#the-string-you-will-actually-see]

Bazel reports `FAILED_PRECONDITION`. Not `NOT_FOUND`. That translation happens
on the worker, which checks for a `NotFound` whose message contains
`not found in either fast or slow store` and converts it before returning:

```text
Missing CAS inputs during prepare_action, returning FAILED_PRECONDITION
```

The error your developers report will not contain the word "missing" and
will not look like a cache problem. The underlying message is:

```text
Object <digest> not found in either fast or slow store. If using multiple
workers, ensure all workers share the same CAS storage path.
```

<Callout type="info" title="That wording is load-bearing">
  The worker matches on the substring `not found in either fast or slow store`.
  It is a real coupling between two crates, not a cosmetic message; do not
  "improve" it without changing both sides.
</Callout>

The same warning is pre-announced at worker startup, which is worth grepping
for when you first bring a fleet up:

```text
Starting worker '<id>'. IMPORTANT: If running multiple workers, all workers
must share the same CAS storage path to avoid 'Object not found' errors.
```

### The two causes [#the-two-causes]

**Workers do not share CAS storage.** The fix is exactly what the message says:
every worker's `fast_slow` slow tier must resolve to the same CAS the clients
upload to. In Compose, `docker-compose-multi-worker.yml` does that by pointing
each worker's slow store at the `cas-server` over gRPC (the `cas-data` volume
it also mounts into the workers is not what shares the data); in Kubernetes,
a shared backing store rather than per-pod local storage. See
[Deploy with Docker Compose](/operate/deploy-docker-compose#the-multi-worker-variant).

**Cache pressure evicted the blob.** A different string, and a much more
helpful one:

```text
<digest> not found in filesystem store. This may indicate the file was evicted
due to cache pressure. Consider increasing 'max_bytes' in your filesystem
store's eviction_policy configuration.
```

Take that advice literally, and read
[the disk runbook](/operate/runbooks#the-disk-fills-up) before you pick a
number; the store can reach roughly twice `max_bytes` on disk.

### A near-miss worth knowing [#a-near-miss-worth-knowing]

```text
<digest> is a zero-digest; FilesystemStore does not persist zero-byte files.
```

Zero-byte outputs are not stored. Callers materialise them directly. If you see
this in a stack trace it is a caller bug, not a cache problem.

## The timeout message that tells you nothing [#the-timeout-message-that-tells-you-nothing]

```text
Operation timed out having no more clients listening
```

This means the client stopped waiting, and `client_action_timeout_s` (default
60 s) cleaned the operation up. It is a statement about the *client*, not about
why the action never ran.

It is the message operators find when chasing a hung build, and it is
consistently misleading: the action was almost certainly stuck in
[the silent queue](#actions-queue-forever-and-workers-sit-idle), and this
string is just the tombstone.

<Callout type="warn" title="Do not set client_action_timeout_s below the keepalive interval">
  The scheduler logs an error at startup if you do, because a client that is
  still actively waiting will be reaped between keepalives.
</Callout>

## Timeouts [#timeouts]

| String | Code | Meaning |
| --- | --- | --- |
| `Action timeout of N seconds is greater than the maximum allowed timeout of M seconds` | `InvalidArgument` | The client asked for longer than `max_action_timeout_s` (default 20 minutes). **Rejected outright, not clamped.** |
| `Command '<cmd>' timed out after N seconds` | `DeadlineExceeded` | The action ran past its timeout and was killed. |
| `BatchUpdateBlobs per-blob timeout (30 s) elapsed for digest <d>` | `DeadlineExceeded` | Fixed 30-second per-blob budget. Not configurable. |
| `BatchReadBlobs per-blob timeout (30 s) elapsed for digest <d>` | `DeadlineExceeded` | Same. |
| `Timeout waiting for previous operation cleanup` | `DeadlineExceeded` | The previous action's cleanup exceeded `max_cleanup_wait_s` (default 30 s). |

The timeout defaults that matter: `max_action_timeout_s` 20 minutes,
`max_upload_timeout_s` 10 minutes, `max_cleanup_wait_s` 30 seconds,
`worker_timeout_s` 5 seconds, `client_action_timeout_s` 60 seconds,
`retain_completed_for_s` 60 seconds.

`retain_completed_for_s` is not an execution timeout; it is how long a
completed action stays findable. Set it too low and a client reconnecting after
a network blip cannot find its own finished action.

## Zero means default [#zero-means-default]

Setting a field to `0` to disable it usually does the opposite.

| Field | `0` means |
| --- | --- |
| `max_job_retries` | **3.** Not "no retries". |
| `worker_timeout_s` | 5 seconds. |
| `client_action_timeout_s` | 60 seconds. |
| `retain_completed_for_s` | 60 seconds. |
| `max_action_executing_timeout_s` | Genuinely disabled. The one exception. |
| `max_inflight_tasks` | Genuinely infinite. |
| `max_open_files` | The default of 24576. |
| `eviction_policy.max_bytes` | Genuinely "never evict by size". |
| `max_bytes_per_stream` | The default of 64 KiB. |
| `max_concurrent_requests` | Genuinely unlimited. |
| `max_concurrent_writes` | Genuinely unlimited. |

No single rule applies here, which is exactly why the table exists. When in
doubt, check [the config reference](/reference/nativelink-config) for the field
rather than assuming.

## Worker connection churn [#worker-connection-churn]

A worker that logs at a steady ~2 Hz is reconnecting:

```text
Worker disconnected from scheduler
```

Reconnect is a **flat 0.5-second retry: no backoff, no jitter, no cap, and no
configuration field**. That cadence is the signature, and a scheduler outage
produces it from every worker simultaneously for the duration.

Every reconnect calls `kill_all()`, destroying every in-flight action on that
worker. If your builds are failing intermittently and your workers are
churning, those are the same incident.

| String | Meaning |
| --- | --- |
| `Worker registered with scheduler` | The handshake succeeded. |
| `Worker disconnected from scheduler` | The stream broke; `kill_all()` is about to run. |
| `Actions in transit did not reach zero before we disconnected from the scheduler` | The 10-second drain window expired with work still in flight. |
| `Worker <id> timed out, removing from pool` | Scheduler side. A worker is only evicted when both its local timestamp and the worker registry agree it is dead. |
| `Worker command failed, removing worker <id>` | The scheduler could not deliver work to it. |

The worker sends a keepalive every half of its `worker_api_endpoint.timeout`
(default 5 s, so every 2.5 s), and the scheduler evicts a worker it has not
heard from for `worker_timeout_s` (default 5 s), checking once a second. The
scheduler counts what it receives in `nativelink_worker_keepalives_total` and
departures by reason in `nativelink_worker_disconnections_total`.

## Limits that block instead of erroring [#limits-that-block-instead-of-erroring]

If the symptom is "everything got slow and nothing errored", you are probably
sitting on one of these:

| Limit | Default | Behaviour |
| --- | --- | --- |
| Open file descriptors | 24576 | Blocks on a semaphore. **Never produces `EMFILE`.** |
| `max_concurrent_requests` (gRPC) | unlimited | Queues rather than rejecting. |
| `max_client_permits` (Redis) | 500 | Blocks. |
| `max_concurrent_writes` (filesystem) | unlimited | Blocks when set. |

The FD limit deserves special attention. NativeLink uses only **80% of the
limit it actually achieved**, not 80% of what you configured. In a container
with a hard `nofile` cap, the semaphore silently shrinks and the only evidence
is a startup warning:

```text
The new open file limit (1024) is below the recommended value of 24576.
Consider raising max_open_files.
```

Or, if the raise failed entirely:

```text
set_open_file_limit() failed to assign open file limit. Maybe system does not
have ulimits, continuing anyway.
```

Every filesystem operation takes a permit: opens, creates, hard links, chmod,
directory creation, directory reads, symlink reads. When permits run out,
everything using the filesystem store queues.

## Startup failures [#startup-failures]

These all abort the process before it serves anything. All of them are config
wiring, and all of them name the offending key.

| String | Cause |
| --- | --- |
| `'cas_store': '<name>' does not exist` | The store name referenced by a service is not defined in `stores`. |
| `'experimental_chunking.index_store' of instance '<i>' is required` | Chunking enabled without an index store. |
| `'experimental_chunking.index_store': '<name>' does not exist` | Index store name typo. |
| `'redis_store': '<name>' does not exist` | Scheduler backend points at an undefined store. |
| `Could not downcast to redis store in RedisAwaitedActionDb::new` | The referenced store is not a standard-mode Redis store. |
| `No addresses were specified in redis store configuration.` | Empty `addresses`. |
| `Both connection_timeout_s and connection_timeout_ms were set, can only have one!` | Pick the `_ms` form; the `_s` form is deprecated. |
| `upload_ac_results_strategy is set, but no ac_store is configured` | The worker will upload action results but has nowhere to put them. |
| `Could not remove work_directory '<path>' in LocalWorker` | The worker purges its work directory at startup and could not. Two workers sharing one work directory is the usual cause. |

Config parsing uses `deny_unknown_fields`, so a misspelled key is a startup
error naming the key, not a silently ignored setting. That is a feature; trust
the message.

One string that looks like a startup failure and is not:
`Could not parse the value of precondition_script: '<v>'`. The
`experimental_precondition_script` value is split with `shlex` before every
action, not at startup, so an unparseable value fails each action as it
arrives and the worker stays up.

## Retries [#retries]

Retry exhaustion is **not** a distinct error. It is the last underlying error
with a suffix appended:

```text
... On attempt 5
```

The code and message you see are the *cause*, and the suffix tells you it
had been happening repeatedly.

Non-retryable codes: `InvalidArgument`, `FailedPrecondition`, `OutOfRange`,
`Unimplemented`, `NotFound`, `AlreadyExists`, `PermissionDenied`,
`Unauthenticated`. Everything else retries, which is why a full disk, arriving
as `Unknown` or `Internal`, burns the whole retry budget before failing.

Two other retry strings: `Retry stream ended abruptly on attempt N`, and
`Not retrying permanent error`.

## Where else to look [#where-else-to-look]

[Scaling workers](/operate/scaling-workers) has a "when it doesn't work"
section covering the fleet-shaped versions of several problems above: queue
depth high while workers idle, adding workers not helping, scale-in causing
failed actions, two workers on one host behaving erratically. It complements
this page rather than duplicating it.

[Observability](/operate/observability#troubleshooting) covers the metrics
pipeline itself: no metrics appearing, missing cache metrics, out-of-order
samples. Note that those are problems with the *collector*, not with
NativeLink.

[Security hardening](/operate/security-hardening) covers what is deliberately
absent rather than broken: there is no inbound authentication anywhere, and
the committed certificates expired in 2024.

<SourceLink file="nativelink-scheduler/src/simple_scheduler.rs" symbol="do_try_match" />
<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="is_cas_blob_missing" />
<SourceLink file="nativelink-util/src/platform_properties.rs" symbol="is_satisfied_by" />

## What's next [#whats-next]

<NextStep href="/operate/runbooks" title="Runbooks">
  Once you know which incident you are in, the procedure that ends it.
</NextStep>

<NextStep href="/reference/nativelink-config" title="Configuration reference" kind="aside">
  Every field, every default, generated from the source.
</NextStep>

<NextStep href="/operate/tuning" title="Tuning" kind="aside">
  The levers that make a healthy system faster.
</NextStep>

# https://docs.nativelink.com/explanations/architecture

**Architecture**: The four roles, who actually writes to the action cache, and which parts of the system hold state you can lose.

**Who this is for:** anyone deciding how to deploy NativeLink, or trying to
predict which component's failure costs them what. **What you'll have at the
end:** the four roles, the path an action takes on a miss and on a hit, and
an accurate answer to "what do I have to back up". **Time:** fifteen minutes.

NativeLink implements the
[Remote Execution API](https://github.com/bazelbuild/remote-apis), and that
API describes four roles: a content-addressable store, an action cache, a
scheduler, and workers. It is worth being precise about what "four roles"
means here, because it is the source of the most common deployment mistake.

**Four roles is the shape of the protocol, not the shape of the program.**
One `nativelink` binary exists. Which roles it plays is decided by the
config file you hand it; there are no per-role images and no build features.
A single-developer setup runs all four in one process; a large cluster runs
several deployments of the same image with different configs. See
[the architecture deep dive](/explanations/architecture-deep-dive) for what
that binary actually does at startup.

## The four roles [#the-four-roles]

<Mermaid>{`sequenceDiagram
    participant client as Client (Bazel / Buck2 / Siso)
    participant sched as Scheduler
    participant worker as Worker
    participant cas as CAS + AC
    client->>cas: GetActionResult(action digest)
    cas-->>client: cache miss
    client->>cas: upload missing inputs
    client->>sched: Execute(action)
    sched->>worker: assign action
    worker->>cas: fetch inputs
    worker->>worker: run command
    worker->>cas: upload outputs
    worker->>cas: UpdateActionResult
    worker->>sched: operation finished
    sched->>client: ActionResult
    client->>cas: fetch output blobs
`}</Mermaid>

### CAS: content-addressable storage [#cas]

Every byte NativeLink touches lives in the CAS, keyed by a digest of its
content. Source files, object files, binaries, stdout, all of it. Two
identical files anywhere in your organisation collapse into one stored blob,
because they hash to the same key.

The digest function is **SHA-256 or BLAKE3**, negotiated per request rather
than fixed at build time. Blobs stored under different digest functions do
not share a namespace: switching the function is equivalent to starting with
an empty cache.

The CAS is the only role holding data you would miss. Lose it and every
cached artifact is gone. It is also not one thing: a production CAS is a
*stack* of stores composed in config. [The store model](/explanations/store-model)
is the page for that.

### AC: action cache [#ac]

The action cache maps `action digest → ActionResult`. The action digest
covers the command line, every input file's digest, the environment
variables, and the platform properties, so an AC hit means *this exact
computation has been performed before, and here is where its outputs are*.

The AC does not hold the outputs. It holds digests pointing into the CAS,
which is why an AC entry whose CAS blobs have been evicted is a real failure
mode rather than a theoretical one; see
[correctness and hermeticity](/explanations/correctness-hermeticity).

Losing the AC is survivable: clients re-execute, workers refill it.

### Scheduler [#scheduler]

The scheduler receives `Execute` RPCs, decides which queued action goes next,
finds a worker whose platform properties satisfy the action's, and assigns
the work. It also merges duplicate requests, so two developers building the
same commit wait on one execution rather than two.

In configuration "the scheduler" can be a stack rather than a single
component: a `cache_lookup` scheduler can be placed in front of the real
scheduler to answer `Execute` from the action cache, so an action that hits
the cache never appears in queue depth. None of the shipped example configs
do this, because Bazel already calls `GetActionResult` before it calls
`Execute`. [Scheduler internals](/explanations/scheduler-internals) covers
the merge rule, the ordering rule and the retry rule.

### Workers [#workers]

A worker fetches the action's inputs from the CAS, materialises them into a
directory, runs the command, uploads the outputs, and reports back.

Workers hold no durable state; a worker that dies mid-action costs you that
action, not your cache. What they do hold is a local filesystem that is
deliberately not clean between actions;
[worker execution](/explanations/worker-execution) explains why, and what it
means for hermeticity.

<Callout type="warn" title="The worker writes the action cache, not the scheduler">
  It is natural to assume the scheduler records the result, since the
  scheduler is what answered the client. It does not. The worker writes the
  `ActionResult` into its configured `upload_action_result.ac_store` itself
  (as an `UpdateActionResult` call when that store is a gRPC proxy), from
  <SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="upload_ac_results" />.

  The operational consequence: **the worker's `ac_store` is what decides
  whether anything gets cached.** A worker with no `ac_store` refuses to
  start unless `upload_ac_results_strategy` is `never`; a worker with
  `never` runs every action and caches nothing, with builds still
  succeeding and no errors anywhere in the logs. The symptom is a cache hit
  rate pinned near zero.
</Callout>

## One build, twice [#one-build-twice]

The concrete version. A developer runs `bazel build //app:server`, which
expands to roughly two thousand actions. Take one of them: compiling
`main.cc`.

**Cold, nothing in the cache:**

<Steps>
  <li>
    **Bazel builds the action digest.** The compiler's own digest, the
    command line, `main.cc` and every header it includes, the platform
    properties. All of it hashes into one digest.
  </li>
  <li>
    **`GetActionResult(digest)` → `NOT_FOUND`.** A single round trip, a few
    milliseconds.
  </li>
  <li>
    **Bazel uploads what's missing.** `FindMissingBlobs` first, then
    `BatchUpdateBlobs` for the ones the CAS does not have. On a cold cache
    this is most of the wall time.
  </li>
  <li>
    **`Execute(action)` to the scheduler.** The scheduler either merges this
    into an identical in-flight action or queues it.
  </li>
  <li>
    **A worker is assigned.** The action's platform properties
    (`OSFamily: Linux`, `container-image: ...`) have to be satisfiable by a
    worker's declared properties, or the action queues indefinitely rather
    than failing.
  </li>
  <li>
    **The worker runs it.** Fetch inputs, materialise the input root, execute
    the command under a timeout, capture stdout, stderr and the declared
    output paths.
  </li>
  <li>
    **The worker uploads.** Output blobs to the CAS, then the `ActionResult`
    to the AC.
  </li>
  <li>
    **Bazel downloads what it needs.** With `--remote_download_minimal` it may
    download nothing at all and keep only the digests.
  </li>
</Steps>

**Warm, the same build on a colleague's machine:**

Step 2 returns an `ActionResult` instead of `NOT_FOUND`. Steps 3 through 7
do not happen. The scheduler is never contacted; no worker is involved; no
compiler runs. The cost of that action is one gRPC round trip plus whatever
output bytes the client actually wants.

That gap, one round trip against a compile, is the whole value
proposition, and it is why the cache hit rate is the number worth watching
rather than worker count.

<Callout type="info" title="Why the second build can still be slow">
  A high hit rate with disappointing wall time usually means the client is
  downloading outputs it does not need. `--remote_download_minimal` (Bazel)
  keeps intermediate artifacts remote. The other common cause is a
  mismatched `instance_name` between client and server, which produces a
  permanently empty cache with no error.
</Callout>

## What holds state [#what-holds-state]

Deployment planning comes down to this table, and one widespread claim about
it is wrong.

| Role | Durable state | If you lose it |
| --- | --- | --- |
| CAS | Yes, the blobs | Every cached artifact. Rebuild from scratch. |
| AC | Yes, but cheap | Clients re-execute and refill it. |
| Scheduler | **Yes, by default** | In-flight actions. See below. |
| Worker | No | The action it was running, which is retried. |

The scheduler is frequently described as stateless. It is not, by default:
the awaited-action state (what is queued, what is executing, who is waiting
on it) lives **in memory in that process**. Two scheduler replicas behind a
load balancer with the default configuration are two independent schedulers
that cannot merge each other's actions or answer each other's clients.

Making the scheduler horizontally scalable means moving that state out, which
is what the Redis-backed state manager is for.
[Scheduler internals](/explanations/scheduler-internals) covers what that
changes and what it costs.

## Why Rust, why this shape [#why-rust-why-this-shape]

Three properties drove the design, and all three are operational rather than
aesthetic.

**No garbage collector.** A build's tail latency is very visible: a p99 stall
in the scheduler shows up as developers watching a progress bar. Removing the
GC removes an entire category of latency spike that is difficult to tune away
in a managed runtime.

**Memory safety without a runtime.** A cache serving a wrong artifact is a
worse incident than a cache serving no artifact, because it is silent and it
propagates. Rust removes the memory-corruption class of that bug at compile
time while keeping predictable resource use.

**Content addressing all the way down.** No cache invalidation exists
logic, because there is nothing to invalidate; a blob's key is its content.
Eviction is a size-and-age policy rather than a correctness mechanism. Most
of the hard bugs in a build cache come from invalidation, and this design
does not have the concept.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Which piece do I have to back up?" id="which-piece-do-i-have-to-back-up">
    The CAS, and only the CAS. The AC is a lookup table that refills itself.
    Workers hold nothing durable. The scheduler holds in-flight state that is
    worth nothing after a restart anyway; the clients will retry.
  </Accordion>
  <Accordion title="Can one binary really serve all four roles?" id="can-one-binary-really-serve-all-four-roles">
    Yes. The same executable serves any subset, selected by config. Start
    with one process running everything; split the services out when one of
    them becomes the bottleneck, not before.
  </Accordion>
  <Accordion title="What is an instance name?" id="what-is-an-instance-name">
    A namespace within one cluster. Digests from different instance names
    never collide, so one deployment can serve isolated teams. Every client
    and server must agree on it; a mismatch looks exactly like a
    permanently empty cache, with no error on either side. More vocabulary
    in the [glossary](/reference/glossary).
  </Accordion>
  <Accordion title="Do I need remote execution to get value from this?" id="do-i-need-remote-execution">
    No, and most deployments start without it. A shared cache alone
    (CAS and AC, no scheduler, no workers) is the majority of the benefit
    for the minority of the operational cost. The
    [Remote execution](/remote-execution) section is the path from one to
    the other when you want it.
  </Accordion>
  <Accordion title="Why did my action queue forever instead of failing?" id="why-did-my-action-queue-forever">
    Its platform properties could not be satisfied by any connected worker.
    A key the scheduler does not know is not rejected at submission either:
    the matching pass logs an `Unknown platform property` error and the
    action stays queued. Scheduler internals covers the matching rule in
    detail.
  </Accordion>
</Accordions>

<NextStep href="/explanations/architecture-deep-dive" title="Architecture deep dive">
  The shape of the program rather than the protocol: the crate graph, the
  startup ordering, and where an action's path forks inside the binary.
</NextStep>

# https://docs.nativelink.com/explanations/architecture-deep-dive

**Architecture deep dive**: The crate graph, the startup sequence, and the path an action actually takes through the binary.

**Who this is for:** anyone about to read, extend or debug the NativeLink
source, and anyone who needs to predict what the binary does rather than
what the protocol says it should. **What you'll have at the end:** the crate
graph, the order things come up in at startup, and where an action's path
forks. **Time:** fifteen minutes.

[Architecture](/explanations/architecture) describes four roles (CAS,
action cache, scheduler, worker) because that is the shape of the *protocol*.
This page describes the shape of the *program*. They are not the same shape,
and the difference is where most surprises live: there is one binary, the
roles are configuration, and two of the four never talk to each other in
process at all.

## One binary, five decisions [#one-binary-five-decisions]

`nativelink` is a single executable. Which roles it plays is decided entirely
by the config file it is handed; there are no build features to toggle and
no separate images per role. Everything below happens inside
<SourceLink file="src/bin/nativelink.rs" symbol="src/bin/nativelink.rs" />.

<Mermaid>{`graph TD
  cfg["Parse JSON5 config"] --> stores["Build every store<br/>store_factory + post_init"]
  stores --> sched["Build every scheduler<br/>scheduler_factory"]
  sched --> svc["Register gRPC services<br/>in fixed order"]
  svc --> listen["Bind listeners"]
  listen --> workers["Start local workers"]
  workers --> run["Run forever<br/>(any root future failing = panic)"]
`}</Mermaid>

The ordering is not incidental. Stores are constructed before schedulers
because schedulers hold store handles; schedulers are constructed before
services because the execution service holds a scheduler handle; and workers
are started *after* the listeners are bound, deliberately, so that a worker
in the same process cannot begin announcing itself before the server it
would announce to can answer.

The last step is the one worth internalising: the listener accept loops, the
local workers and the origin-event publisher are joined as root futures with
`try_join_all`, and if any of them returns an error the process panics rather
than continuing in a degraded state. A NativeLink whose worker task has died
does not limp; it dies and lets your supervisor restart it. That is a
deliberate choice about which failure mode is easier to operate.

<Callout type="warn" title="Only HTTP listeners exist">
  `ListenerConfig` has exactly one variant, `http`, and the startup path
  destructures it irrefutably. No other listener kind can be configured;
  a config naming one fails to parse.
</Callout>

## The crate graph [#the-crate-graph]

Twelve library crates plus the binary, in strict layers. The diagram shows
the ten that give the program its shape; the other two are
`nativelink-metric-macro-derive` (the proc macro behind `nativelink-metric`)
and `nativelink-redis-tester` (a fake Redis used by the store crate and by
tests). `nativelink-macro` is a test-only proc macro that no library crate
depends on. An arrow points from a crate to a crate that depends on it.

<Mermaid>{`graph BT
  proto["nativelink-proto"] --> error["nativelink-error"]
  metric["nativelink-metric"] --> error
  error --> config["nativelink-config"]
  config --> util["nativelink-util"]
  util --> store["nativelink-store"]
  store --> scheduler["nativelink-scheduler"]
  store --> worker["nativelink-worker"]
  scheduler --> service["nativelink-service"]
  service --> bin["src/bin/nativelink.rs"]
  worker --> bin
`}</Mermaid>

Two absences carry more information than any of the edges.

**`nativelink-worker` does not depend on `nativelink-scheduler`.** A worker
reaches its scheduler over gRPC and nothing else, whether that scheduler is
in the same process or in another datacentre. No in-process
fast path exists to accidentally rely on, which is why "run everything in one
process" and "run a worker fleet" are the same code with different config.

**`nativelink-service` does not depend on `nativelink-worker`.** The gRPC
surface has no notion of a local worker. The root binary is the only place
where a service and a worker are ever mentioned together.

<Callout type="info" title="nativelink-proto's library root is not under src/">
  Its `src/` holds vendored Bazel `.proto` files; the crate's library root is
  checked-in generated code at
  <SourceLink file="nativelink-proto/genproto/lib.rs" symbol="genproto/lib.rs" />.
  Editing a `.proto` file does not change the build until that generated tree
  is regenerated with `bazel run nativelink-proto:update_protos`, which is a
  manual step, not a build script.
</Callout>

## Configuration is JSON5, and it is validated early [#configuration-is-json5-and-it-is-validated-early]

Config parsing is JSON5, not JSON: comments and trailing commas are legal,
which is why every example in these docs uses them. The entry point is
`CasConfig::try_from_json5_file` in
<SourceLink file="nativelink-config/src/cas_server.rs" symbol="CasConfig" />.

One cross-field check runs before anything is constructed: `check_store_conflict`
rejects a config in which the CAS service and the action cache service for
the same instance name point at the same store. The action cache is keyed by
the `Action` digest, and the CAS holds the `Action` message itself under that
same digest, so sharing a store lets an `ActionResult` overwrite the blob it
describes. Catching it at parse time rather than at first collision is the
whole reason the check exists.

Everything else about the config file (every field, every default) lives in
the [configuration reference](/reference/nativelink-config), which is
generated from these same Rust types rather than transcribed from them.

## Stores are built in two phases [#stores-are-built-in-two-phases]

`store_factory` constructs each configured store, and then a separate
`run_post_init()` pass runs across all of them. The second pass exists because
a `ref_store` names another top-level store (possibly one declared later in
the file), and a single-pass construction would make declaration order
load-bearing. Wrappers such as `fast_slow` and `verify` do not need it: their
inner stores are inline specs, built recursively by the same factory call. By
the time `post_init` runs, every store in the config exists and can be
resolved.

The consequence for anyone adding a store: work that needs *other* stores
belongs in `post_init`, not in the constructor. [The store model](/explanations/store-model)
covers what a store actually is and how composition behaves.

## Services come up in a fixed order [#services-come-up-in-a-fixed-order]

The registration order is: action cache, CAS, execution, `Operations`, Remote
Asset fetch, Remote Asset push, ByteStream, capabilities, worker API, and the
experimental Build Event Protocol service. Each is registered only if the
config asks for it.

Two of those are not gRPC at all. The health endpoint at `/status` and the
admin endpoint at `/admin` are plain HTTP handlers on the same listener, so
a health check does not need a gRPC client, and an admin call is not
discoverable through reflection. [Protocol and API surface](/reference/protocol-api)
enumerates every RPC, including the ones that are registered but return
`unimplemented`.

## Where an action's path forks [#where-an-actions-path-forks]

The single most important structural fact about execution is that **a cache
hit never reaches the scheduler**. Bazel calls `GetActionResult` before it
calls `Execute`, so on a hit the execution service is never contacted at all.

For clients that skip that step, `CacheLookupScheduler` (the `cache_lookup`
scheduler type) is a decorator, not a stage. It wraps the real scheduler and
sits in front of it. On a hit it synthesises an
`ActionStage::CompletedFromCache` and returns; the action is never queued,
never assigned a worker, and never appears in any queue-depth metric. On a
miss it delegates to the wrapped `SimpleScheduler` and the action enters the
world described in [scheduler internals](/explanations/scheduler-internals).
None of the shipped example configs use it; the diagram below shows the
stack with it present.

<Mermaid>{`graph LR
  ex["Execute RPC"] --> pre["Shallow input check"]
  pre --> cl["CacheLookupScheduler"]
  cl -->|hit| done["CompletedFromCache<br/>never queued"]
  cl -->|miss| ss["SimpleScheduler"]
  ss --> match["Matching engine"]
  match --> w["Worker"]
`}</Mermaid>

This explains a metric shape that otherwise looks broken: on a
well-cached build, the scheduler's queue metrics stay near zero while the
execution service is extremely busy. Nothing is wrong. The hits are being
served in front of the thing that counts.

### The pre-flight check is deliberately racy [#the-pre-flight-check-is-deliberately-racy]

Before an `Execute` is accepted, the execution service does a shallow check
that the action's inputs are present: the `Action` proto itself, its
`command_digest` and its `input_root_digest`. It is documented in-source as
deliberately shallow and deliberately racy: it does not walk the full input
tree, and a blob can be evicted between the check passing and the worker
asking for it.

That is the right trade. A complete check would mean walking an entire
Merkle tree on every submission, and it still would not be a guarantee,
because eviction can happen at any point afterwards. The check exists to turn
the common client mistake (submitting an action whose inputs were never
uploaded) into a fast `FAILED_PRECONDITION` carrying a `PreconditionFailure`
that lists the missing digests, so Bazel re-uploads and retries, instead of
a worker-side failure minutes later. It is a courtesy, not an invariant. See
<SourceLink file="nativelink-service/src/execution_server.rs" symbol="ExecutionServer" />.

## Two details that surprise people reading the wire [#two-details-that-surprise-people-reading-the-wire]

**`GetTree` page tokens are `"{hash}-{size}"`.** They are constructed and
parsed by splitting on `-`. They are not opaque, and they are not signed,
though the protocol says clients should treat them as opaque, and clients
that do will keep working if this ever changes.

**ByteStream resource names are parsed backwards.** A resource name looks
like `{instance}/blobs/{hash}/{size}`, and the instance name is allowed to
contain slashes. Parsing left to right therefore cannot tell where the
instance name ends; parsing from the right can, because the trailing
components are fixed. See
<SourceLink file="nativelink-util/src/resource_info.rs" symbol="ResourceInfo" />.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="If it's one binary, why does the deployment guide use three?" id="why-three-processes">
    Because the roles have different scaling and failure characteristics, not
    because the code requires it. A CAS wants durable storage and steady
    memory; a worker wants CPU and is disposable; a scheduler wants to be
    restarted freely. Splitting them lets you size and autoscale each one
    separately. See [production configuration](/operate/production-config).
  </Accordion>
  <Accordion title="Can I add a store without touching the scheduler or worker?" id="add-a-store-in-isolation">
    Yes. `nativelink-store` sits below both, and neither knows about concrete
    store types, with one exception worth knowing: the worker's fast-CAS
    path downcasts to a concrete `FilesystemStore` and refuses to start if it
    is given anything else. That is covered in
    [worker execution](/explanations/worker-execution).
  </Accordion>
  <Accordion title="Why does the process panic instead of retrying?" id="why-panic">
    Because the alternative is a process that is running, passing its health
    check, and quietly not doing one of its jobs. Every root future is
    something the process exists to do; if one ends, the process has nothing
    useful left to be. Dying is the honest signal, and it is the one your
    orchestrator already knows how to act on.
  </Accordion>
</Accordions>

<NextStep href="/explanations/store-model" title="The store model">
  Stores are the layer everything else is built on, and the one whose
  composition rules are hardest to see from the config file.
</NextStep>

# https://docs.nativelink.com/explanations/store-model

**The store model**: What a store actually is, how composition behaves, and where a digest stops being a claim and becomes a fact.

**Who this is for:** anyone composing stores in a config file and wanting to
predict the result, and anyone writing a new store backend. **What you'll
have at the end:** the trait, the key space, the composition rules, and the
one place verification actually happens. **Time:** twenty minutes.

A NativeLink store is not a database binding. It is one small trait
(eight required methods), and everything you configure as a "store" is either
an implementation of it or a wrapper around other implementations of it. The
`stores` block in your config is a graph, and its edges have semantics that
the JSON5 does not show you.

## Three names, one file [#three-names-one-file]

`Store`, `StoreLike` and `StoreDriver` all live in
<SourceLink file="nativelink-util/src/store_trait.rs" symbol="store_trait.rs" />,
and the split between them matters when you read the source.

`StoreDriver` is the trait a backend implements. `Store` is a
`#[repr(transparent)]` newtype around `Arc<dyn StoreDriver>`, the handle
everything else holds. `StoreLike` is the ergonomic layer callers use: its
methods (`get`, `has`, `update_oneshot` and the rest) accept anything that
converts into a `StoreKey` and forward to the pinned driver, so callers never
deal with `Pin<&dyn StoreDriver>` directly.

A driver must supply eight methods: `post_init`, `has_with_results`,
`update`, `get_part`, `inner_store`, `as_any`, `as_any_arc`, and
`register_remove_callback`. Only the middle three are the ones you would
guess. The other five are what makes composition work, and `inner_store` in
particular is where composition either succeeds or silently stops.

## The key space is not just digests [#the-key-space-is-not-just-digests]

`StoreKey` has two variants: `Digest(DigestInfo)` and `Str(Cow<str>)`. Most
of NativeLink deals in the first. The second exists because two subsystems
need to store things that are not content-addressed at all: the Build Event
Protocol service, and the scheduler's awaited-action database.

This has a consequence people hit and misread. `StoreKey::into_digest()`
converts a `Str` key to a digest by BLAKE3-hashing the string, with the hash
function hardcoded. That conversion is lossy and irreversible: a store that
only understands digests will accept a `Str` key and file it somewhere
deterministic, but the original string is gone. The filesystem store handles
this more honestly by keeping the two in separate namespaces on disk (`s/`
for string keys, `d/` for digests) so the two can never collide.

<Callout type="info" title="Why a hardcoded hash here is fine">
  Elsewhere, a hardcoded hash function would be a correctness bug, because
  the client picks the digest function per request. Here it is not: these
  keys never cross the REAPI wire and are never compared against a
  client-supplied digest. The hash is an internal placement function, not an
  identity.
</Callout>

## The digest on the wire is a claim, not a measurement [#the-digest-on-the-wire-is-a-claim-not-a-measurement]

This is the single most important thing on this page.

**NativeLink does not derive a blob's digest from its bytes on upload.** The
client declares the digest in the ByteStream resource name, and the server
stores the bytes under the declared key. See
<SourceLink file="nativelink-service/src/bytestream_server.rs" symbol="ByteStreamServer" />.
(The one exception is a zstd-compressed ByteStream upload, where the decoder
already has to read every byte and rejects a decoded stream whose size or
hash does not match the digest.)

For plain uploads, the only thing that turns a claimed digest into a verified
one is a `verify` store somewhere in the graph, and both of its checks
default to **off**:

| Field | Default | What it costs when on |
| --- | --- | --- |
| `verify_size` | `false` | Nearly nothing; a running byte count |
| `verify_hash` | `false` | One hash pass over every uploaded byte |

A cache with no `verify` store in the write path is a cache that will store,
and later serve, whatever a buggy or hostile client claims. That is not a
NativeLink-specific hazard (it is how the REAPI is specified), but the
default being `false` means it is a decision you are making whether or not
you know it. [Correctness and hermeticity](/explanations/correctness-hermeticity)
covers what to do about it.

### How the verify store actually works [#how-the-verify-store-actually-works]

The implementation in
<SourceLink file="nativelink-store/src/verify_store.rs" symbol="VerifyStore" />
is worth understanding, because three of its properties are not apparent from the config.

**Size is checked eagerly, before a byte is read.** If the declared size is
already wrong at the point the request arrives, the upload is rejected
without transferring anything.

**The hash function is read per-request from the ambient context**, not from
config. This is what lets one store serve both SHA-256 and BLAKE3 clients
without duplicating the graph.

**Verification runs concurrently with the write, not before it.** The inner
store's `update` and the checker are driven together, so verification adds no
serialised pass. The cost of that design is that a failing upload has already
been partially written to the inner store when the failure is detected.

Size is enforced along three separate paths (overshoot, an exact hit that
turns out not to be EOF, and undershoot at EOF), and the hash is compared at
EOF. Every failure is `InvalidArgument`, which is the correct code: the
client sent something that did not match what the client said it was sending.

<Callout type="warn" title="Nothing is re-verified on read">
  `VerifyStore::get_part` is a bare pass-through. Verification is a
  **write-side** property only. A blob that entered the store before you added
  the verify wrapper, or through a path that bypassed it, will be served
  without complaint forever.
</Callout>

## Composition, and the barriers in it [#composition-and-the-barriers-in-it]

`inner_store` is how the system looks *through* a wrapper to find an
optimisation opportunity, for example, to discover that the real backing
store is a filesystem store and a hardlink is possible.

Most wrappers return `self` from `inner_store`. That is not laziness; it is a
statement that looking through them would be wrong, because they transform
the bytes or the key. A `compression` store cannot let a caller hardlink its
inner file: the inner file is not the blob, so most wrappers are
**optimisation barriers**, and only three forward: `size_partitioning`,
`shard`, and `ref_store`, the three that route without transforming.

The practical rule when composing: **put transforming wrappers as close to the
backend as you can**. A `verify` in front of a `fast_slow` is a barrier in
front of the fast path.

<Mermaid>{`graph LR
  caller["Caller<br/>(CAS service, worker)"]
  subgraph forward["Forward inner_store: routing only"]
    sp["size_partitioning"]
    sh["shard"]
    rs["ref_store"]
  end
  subgraph barrier["Return self: optimisation barriers"]
    vf["verify"]
    cp["compression"]
    dd["dedup"]
  end
  fs["filesystem<br/>hardlink possible here"]
  caller --> forward
  forward --> barrier
  barrier --> fs
  forward -. "caller can see through" .-> fs
`}</Mermaid>

A caller reaching through the routing wrappers can discover the filesystem
store and hardlink from it. A caller reaching through a transforming wrapper
cannot, because what is on disk underneath it is not the blob.

`StoreOptimizations` is the vocabulary for the other half of this:
`FileUpdates`, `NoopUpdates`, `NoopDownloads`, `LazyExistenceOnSync`,
`SubscribesToUpdateOneshot`, capabilities a store advertises so callers can
skip work rather than discovering at runtime that it was unnecessary.

### `fast_slow` writes through, and deliberately re-uploads [#fast-slow-writes-through-and-deliberately-re-uploads]

With the default `fast_direction` and `slow_direction`, writes go to both
tiers concurrently and **both must succeed**. It has no write-behind and no
eventual consistency window: an upload that returns OK is durable in the slow
tier.

The surprising part is existence. `fast_slow`'s `has_with_results`
**intentionally never consults the fast store** (unless the slow tier is a
`noop` store, in which case only the fast tier is checked). It asks the slow
tier, then counts any slow-tier writes still in flight. A blob that is
present only in the fast tier is reported absent, which causes the client to
upload it again, and that upload is what gets it into the slow tier. The
apparent inefficiency is the mechanism that stops the fast tier from becoming
a place where data can be lost.

Reads use a leader/follower scheme so that a hundred simultaneous misses for
the same blob produce one fetch from the slow tier rather than a hundred.
Followers wait up to one minute for the leader before giving up and fetching
for themselves; that timeout is what stops a stuck leader from wedging every
reader behind it.

### `dedup` is a chunking index, and it has opinions [#dedup-is-a-chunking-index-and-it-has-opinions]

A `dedup` store splits blobs with content-defined chunking (FastCDC), stores
the chunks, and writes an index under the **original** key. Two large blobs
that share most of their content share most of their chunks.

Two things to know before enabling it. First, chunk hashing is **hardcoded to
BLAKE3** regardless of the digest function the request used, which is
correct for the same reason as `StoreKey::into_digest`, since chunk hashes
are placement, not identity, but it does mean a `dedup` store's internal
layout is unrelated to your clients' digest function.

Second, a corrupted index reports itself inconsistently: `has()` returns
`NotFound` while `get_part` returns `Internal`. If you are debugging a store
where existence checks and reads disagree, a damaged dedup index is the
first thing to suspect.

## Two things that are not what their filenames say [#two-things-that-are-not-what-their-filenames-say]

<SourceLink file="nativelink-store/src/r2_store.rs" symbol="r2_store.rs" /> and
<SourceLink file="nativelink-store/src/oci_store.rs" symbol="oci_store.rs" />
are not store implementations. Both are thin constructors that return an
`S3Store`. Cloudflare R2 and OCI object storage are S3-compatible, and
NativeLink treats them as exactly that. If you are chasing a bug in R2
behaviour, the code you want is in `s3_store.rs`.

`RefStore` is the other oddity: it resolves its target by name at
`post_init`, holding a weak reference to the store manager, using
`UnsafeCell` with a hand-written `Sync` impl. It is the mechanism that lets
one store be named by several others without duplicating it, and it is why
[store construction happens in two phases](/explanations/architecture-deep-dive#stores-are-built-in-two-phases).

<Callout type="warn" title="A filesystem existence check can have a side effect">
  When `FilesystemStore::has_with_results` is asked about the empty blob (the
  digest of zero bytes) and that entry is not on disk yet, it creates the
  zero-byte file as a side effect of what reads, from its name, like a pure
  query. Anything counting inodes or watching the content directory should
  expect an entry that no upload produced.
</Callout>

## Where the exact truth lives [#where-the-exact-truth-lives]

Every store type, every field and every default is in the generated
[store configuration reference](/reference/nativelink-config/store-overview). This
page deliberately does not restate field names, because the reference is
generated from the Rust types and this page is not.

For choosing a backend rather than understanding the model, start at
[Store backends](/how-to/stores).

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Should I turn on verify_hash?" id="should-i-turn-on-verify-hash">
    On the write path of anything shared between people, yes. The cost is one
    hash pass over uploaded bytes, concurrent with the write rather than
    serialised before it. The thing it buys is that a digest in your cache
    means what it says. On a purely local single-developer cache the
    calculation is different, because you are the only client and you are not
    defending against yourself.
  </Accordion>
  <Accordion title="Why is my fast tier getting the same blob uploaded twice?" id="fast-tier-duplicate-uploads">
    That is `fast_slow` working as designed: existence checks skip the fast
    tier, so a blob present only there is reported missing and re-uploaded,
    which is what lands it in the slow tier. The alternative would be a cache
    that reports a blob present when the only durable copy of it does not
    exist.
  </Accordion>
  <Accordion title="Does wrapping a store in compression break hardlinking?" id="compression-and-hardlinks">
    Yes, and it must. Hardlinking works by handing the worker the store's own
    file; a compression store's file is not the blob. That is why it returns
    itself from `inner_store`; the barrier is the correct answer, not a
    missing optimisation.
  </Accordion>
  <Accordion title="Can I point the CAS and the action cache at one store?" id="cas-and-ac-same-store">
    No, and the config parser rejects it before anything starts. They hold
    different value types under keys from the same space, so sharing a store
    corrupts both.
  </Accordion>
</Accordions>

<NextStep href="/explanations/scheduler-internals" title="Scheduler internals">
  How an action that missed the cache becomes an action running on a worker.
</NextStep>

# https://docs.nativelink.com/explanations/scheduler-internals

**Scheduler internals**: Action merging, the awaited-action database, how a worker is chosen, and what happens when an action fails.

**Who this is for:** anyone tuning a scheduler, debugging an action that
queued forever, or wondering why two identical builds did not share work.
**What you'll have at the end:** the merge rule, the ordering rule, the
matching rule, and the retry rule, each of which has at least one
counter-intuitive edge. **Time:** twenty-five minutes.

The scheduler's job sounds small: take actions from clients, give them to
workers. Almost all of its complexity comes from three questions it has to
answer that the protocol does not specify: when are two actions the same
one, which of the queued actions goes next, and what does "this failed" mean.

## No `ActionScheduler` trait exists [#there-is-no-actionscheduler-trait]

If you go looking for one (and older material about NativeLink will send you
looking) you will not find it. The abstraction is split by *who is asking*,
into three traits in
<SourceLink file="nativelink-util/src/operation_state_manager.rs" symbol="operation_state_manager.rs" />:

**`ClientStateManager`** is the client's view: submit an action, subscribe to
its progress. **`WorkerStateManager`** is the worker's view: report that an
operation changed state. **`MatchingEngineStateManager`** is the matcher's
view: list actions that could be dispatched, assign one to a worker.

Two more sit beside them: `WorkerScheduler`, which owns worker lifecycle, and
`KnownPlatformPropertyProvider`, which is how the capabilities service learns
which platform property keys an instance's scheduler knows about.

The split is what allows the whole state layer to be swapped:
`SimpleSchedulerStateManager` implements all three traits and is generic over
an `AwaitedActionDb`, which is either `MemoryAwaitedActionDb` or the
store-backed `StoreAwaitedActionDb` used for Redis. It is also why "the
scheduler" in configuration can be a stack (the `cache_lookup` decorator is
optional, and none of the shipped example configs use it):

<Mermaid>{`graph TD
  client["Client<br/>Execute"]
  cl["CacheLookupScheduler<br/>(decorator)"]
  ac["Action cache"]
  ss["SimpleScheduler"]
  me["Matching engine"]
  sm["State manager<br/>in-memory or Redis"]
  wk["Worker"]
  client --> cl
  cl -->|"lookup"| ac
  ac -->|"hit: never enters the queue"| client
  cl -->|"miss"| ss
  ss --> sm
  me --> sm
  me -->|"assign"| wk
  wk -->|"state updates"| sm
`}</Mermaid>

The decorator in front is why a cache hit never appears in queue depth, and
why `ActionStage::CacheCheck` is never emitted by the scheduler crate.

## When are two actions the same action? [#when-are-two-actions-the-same-action]

Two clients asking for the same work should wait on one execution. The key
that decides this is `ActionUniqueKey`, and it has exactly three fields:
instance name, digest function, and the action digest.

That key is wrapped in `ActionUniqueQualifier`, which is either `Cacheable`
or `Uncacheable`. Only the `Cacheable` variant merges.

Three things people expect to be in that key and are not:

**No salt exists.** A comment in the source refers to one; the struct has no
such field. If you need two textually identical actions to be distinct, the
difference has to be in the action itself.

**`do_not_cache` is never read.** The REAPI flag exists on the message and
NativeLink parses it, but no code in the repository branches on it. Setting
it does not stop a result being written.

**The flag that actually matters is `skip_cache_lookup`,** and its effect is
asymmetric in a way worth stating plainly: it disables the cache **read**,
and it disables **merging**, but it does **not** disable the cache
**write**. An action submitted with `skip_cache_lookup` runs on its own,
ignoring any existing result, and then overwrites that result.

<Callout type="info" title="Why that asymmetry is the useful one">
  It is exactly what you want for "re-run this and fix the cache": force a
  real execution, and let everyone else benefit from the answer. If it also
  suppressed the write, the poisoned entry you were trying to replace would
  survive.
</Callout>

## Ordering is one 64-bit integer [#ordering-is-one-64-bit-integer]

`AwaitedActionSortKey` packs the entire queue ordering into a single `u64`:
the upper 32 bits are the priority (shifted from `i32` into unsigned range so
byte order matches numeric order), and the lower 32 bits are the insertion
timestamp in whole seconds XORed with `u32::MAX`.

The XOR is what makes an *earlier* timestamp sort *higher*, so the rule falls
out of the encoding: **priority dominates absolutely, and ties break in
favour of the action that arrived first**, at one-second resolution; actions
inserted within the same second are ordered by operation ID, which is
effectively arbitrary. It has no aging, no starvation
guard, and no fairness across clients. A client that submits everything at
priority 1000 will starve a client at priority 0 indefinitely, and nothing in
the scheduler will intervene.

Priority is a contract between the people sharing a cluster, enforced
socially. If you need it enforced mechanically, that has to happen in front
of the scheduler.

## Two state enums, and one that never appears [#two-state-enums-and-one-that-never-appears]

`ActionStage` is the wire-facing state a client sees.
`SortedAwaitedActionState` is the database index state the scheduler sorts
on. They overlap but are not the same, and conflating them while reading the
source is a reliable way to get lost.

Notably: **`ActionStage::CacheCheck` is never produced by the scheduler
crate.** It exists in the protocol, but by the time an action reaches the
scheduler the cache lookup has already happened in the decorator in front of
it; see [where an action's path forks](/explanations/architecture-deep-dive#where-an-actions-path-forks).
A client waiting to observe `CacheCheck` will wait forever.

## Matching is first-fit over an LRU [#matching-is-first-fit-over-an-lru]

The matching engine walks the queue in sort order, and for each action looks
for the first worker that satisfies the action's platform properties. The
source still carries an `O(n*m)` TODO in
<SourceLink file="nativelink-scheduler/src/simple_scheduler.rs" symbol="SimpleScheduler" />,
but the walk is narrowed first: a `WorkerCapabilityIndex` (an inverted index
over `Exact` and `Priority` properties) produces the candidate set, and only
that set is scanned in LRU order, checking availability and `Minimum`
values per worker.
<SourceLink file="nativelink-scheduler/src/worker_capability_index.rs" symbol="WorkerCapabilityIndex" />

For clusters of tens or low hundreds of workers this is not the bottleneck.
It is worth knowing about before you plan for thousands.

`allocation_strategy` is a smaller knob than its name suggests: it chooses
**which end of the LRU the scan starts from**, and nothing else. Starting
from the least-recently-used end spreads load; starting from the
most-recently-used end packs work onto hot workers and leaves the rest idle
long enough to be scaled down. Neither is a bin-packing algorithm.

### Platform properties are typed, and `Minimum` is a resource pool [#platform-properties-are-typed-and-minimum-is-a-resource-pool]

Every property key a worker advertises has a declared type:

| Type | Meaning |
| --- | --- |
| `Exact` | The action's value must equal the worker's |
| `Minimum` | The worker's value must be at least the action's, **and is consumed** |
| `Priority` | The worker must have the key; its value is not compared (the source marks value-based preference as a TODO) |
| `Ignore` | Accepted and disregarded |

A *worker* announcing a property key the scheduler does not know is rejected
at connect time. An *action* naming an unknown key is not rejected at
submission: the key is only translated during the matching pass, where it
produces an `Unknown platform property` error that is logged, and the action
stays queued. That is the harder of the two to diagnose, so if an action
sits in the queue with workers idle, check the scheduler log for that
message before anything else.
<SourceLink file="nativelink-scheduler/src/platform_property_manager.rs" symbol="make_prop_value" />

`Minimum` is the one that changes how you think about a worker. Its value is
**decremented when an action is dispatched and restored when the action
completes**; see
<SourceLink file="nativelink-scheduler/src/worker.rs" symbol="Worker" />. A
worker advertising `cores: 16` is not saying "I am a 16-core machine, send me
anything"; it is a pool of 16 units that actions draw down and return. That
is the mechanism behind running several actions concurrently on one worker,
and it is why getting the advertised number wrong oversubscribes the machine
rather than queueing.

[Platform properties](/remote-execution/platform-properties) covers this from
the configuration side.

## Retries, and what "failed" means [#retries-and-what-failed-means]

The attempt counter is incremented by two different events, with different
rules.

**`UpdateWithError` increments the attempt count, unless the error is
`ResourceExhausted`.** Backpressure is free. A worker saying "I am too busy"
does not spend one of the action's lives, which is what makes it safe for a
worker to shed load.

**`UpdateWithDisconnect` always increments.** A worker that vanishes mid-action
always costs an attempt. Without this, an action that reliably OOM-kills its
worker would retry forever, taking a worker down each time.

The limit is checked as `attempts > max_job_retries`, strictly, so the
default of `3` permits **four** total executions. That off-by-one is in the
name: three *retries* after one initial attempt.

<Callout type="warn" title="Final failure is not an RPC error">
  When an action exhausts its retries, the client receives
  `ActionStage::Completed` with `ActionResult.error` set: a *successful* RPC
  describing a failed action. A client that only inspects the gRPC status will
  read this as a clean completion. This is what the REAPI specifies, and it
  is the single most common source of "the build passed but nothing was
  built" reports.
</Callout>

### Two timeouts with very different defaults [#two-timeouts-with-very-different-defaults]

`max_action_executing_timeout_s` **defaults to `0`, which means disabled.**
It is not the only guard against a hung action: the worker kills the process
when the action's own timeout (or the worker's `max_action_timeout_s`, 20
minutes by default) elapses, and the scheduler re-queues an executing action
whose worker stops sending updates for `worker_timeout_s`. What this knob
adds is a ceiling for the case where the worker is alive and heartbeating
but sends no update for a specific action; with it disabled, that case has no
ceiling. On a shared cluster it is worth setting.

`client_action_timeout_s` defaults to 60. If no client is listening for that
long, the action is killed with `DeadlineExceeded`. This is a deliberate
anti-orphan measure (a client that pressed Ctrl-C should not leave work
running), but it also means a client that disconnects and reconnects slowly
can lose work it expected to still be running.

## Who writes the action cache entry? [#who-writes-the-action-cache-entry]

Not the scheduler. **The worker writes the action cache entry, and it does so
before telling the scheduler that the action finished.** The ordering is what
guarantees that a client which observes completion and then queries the cache
finds the result there.

What gets written depends on the exit code, via `should_cache_result` in
<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="RunningActionsManager" />.
With the defaults, a **successful** action's result goes to the action cache
(`SuccessOnly`), and a **failed** action's result goes to the historical
results store (`FailuresOnly`) instead. A non-zero exit code is not an error
in the REAPI sense (it is a legitimate, cacheable outcome), but by default
NativeLink keeps it out of the action cache so that a failing compile is
re-attempted rather than replayed.

## Where the exact truth lives [#where-the-exact-truth-lives]

Every scheduler field and default is in the generated
[scheduler configuration reference](/reference/nativelink-config#namedschedulerconfig).
For the operational levers, see [tuning](/operate/tuning).

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Why did two identical builds not share work?" id="why-no-merge">
    Check the three fields in the merge key first: instance name, digest
    function, action digest. A different instance name is the usual answer,
    and the second most common is one client sending `skip_cache_lookup`,
    which opts that action out of merging entirely.
  </Accordion>
  <Accordion title="My queue depth is zero but clients are waiting. Why?" id="queue-depth-zero">
    Cache hits never enter the queue; on a Bazel client they are answered by
    `GetActionResult` before `Execute` is ever sent, and with a `cache_lookup`
    scheduler they are answered by the decorator in front of the queue. If
    clients are genuinely waiting with an empty queue, look at whether the
    `Execute` calls are being rejected up front (missing inputs return
    `FAILED_PRECONDITION`) rather than queued.
  </Accordion>
  <Accordion title="What does the Redis backend change?" id="redis-backend-differences">
    It swaps the state layer for one backed by RediSearch indexes with
    optimistic concurrency. Two consequences: `Code::Aborted` from an internal
    operation means "reload and retry", not "failed"; it is the normal
    outcome of a version conflict. And action merging becomes best-effort,
    because there is a small window where a just-written action is not yet
    visible to the index; a short retry narrows it but does not close it. Two
    identical actions submitted within milliseconds of each other may both
    run.
  </Accordion>
  <Accordion title="Is fallback_match_interval_s just a safety net?" id="fallback-match-interval">
    Not with the Redis backend. Matching there is driven by pub-sub with this
    interval as a poll fallback, and pub-sub delivery is not guaranteed, so
    the interval is load-bearing, and raising it raises worst-case dispatch
    latency directly. It defaults to five seconds.
  </Accordion>
  <Accordion title="Can I stop one team starving another?" id="priority-starvation">
    Not inside the scheduler. Ordering is priority-dominant with no aging and
    no per-client fairness, so a client submitting everything at high priority
    will starve everyone below it. Enforcement has to sit in front of the
    scheduler: separate instance names with separate schedulers, or an
    admission layer that normalises priority.
  </Accordion>
</Accordions>

<NextStep href="/explanations/worker-execution" title="Worker execution">
  What the worker does with an action once the scheduler hands it over,
  including exactly how much isolation you are and are not getting.
</NextStep>

# https://docs.nativelink.com/explanations/worker-execution

**Worker execution**: How an action becomes a running process: input materialization, what isolation you actually get, and what the worker throws away.

**Who this is for:** anyone running workers, debugging an action that behaves
differently on a worker than locally, or deciding how much to trust the
actions they run. **What you'll have at the end:** the lifecycle, the
filesystem layout, the exact isolation boundary, and the list of things the
worker deliberately discards. **Time:** twenty-five minutes.

The worker is where the abstractions stop. Everything upstream is digests and
state machines; here there is a directory, a process, and an exit code. Most
of the surprising behaviour in a NativeLink cluster originates in this crate,
because this is the only place where the operating system gets a vote.

## The connection is one stream [#the-connection-is-one-stream]

A worker opens exactly one bidirectional gRPC stream to the scheduler
(`ConnectWorker`, the sole RPC on NativeLink's own `WorkerApi` service), and
everything flows over it. It has no polling loop and no second connection.

The worker supplies only an ID *prefix*; the scheduler mints the actual
worker ID by appending a UUIDv6, so IDs are unique and time-ordered. Which
worker gets an action is decided separately, by the LRU-ordered
[matching engine](/explanations/scheduler-internals#matching-is-first-fit-over-an-lru).

<Callout type="warn" title="The heartbeat and the eviction timeout are two separate knobs">
  The worker's heartbeat period is derived as **half** its configured
  `worker_api_endpoint.timeout`, 2.5 seconds by default. Eviction is
  governed by the *scheduler's* `worker_timeout_s`, 5 seconds by default.
  Nothing reconciles them. Raise the scheduler's timeout without raising the
  worker's, or vice versa, and you get workers evicted mid-heartbeat and
  reconnecting in a loop.
</Callout>

## The filesystem, and the hardlink that everything depends on [#the-filesystem-and-the-hardlink-that-everything-depends-on]

On startup the worker **purges and recreates its `work_directory`**. Anything
left there by a previous run is gone. This is intentional (a worker that
inherits state is a worker whose actions are not hermetic), but it means the
work directory must not be somewhere you also keep things.

Each action gets a tree:

```
<work_directory>/
  <operation_id>/
    work/                 <- the action's cwd root
      <Command.working_directory>/
    <side channel file>   <- sibling of work/, not inside it
```

Input materialization's hot path is a **hardlink**: no copy, no `chmod`, no
byte movement at all. The worker links the file straight out of the
filesystem CAS into the action's input tree.

This one decision produces four consequences that account for a large share
of worker bug reports.

**Inputs are read-only.** A hardlink shares the inode, and CAS blobs are
stored `0o444` (or `0o555` when executable). An action that modifies an input
file in place gets `EACCES`. Actions that work locally and fail remotely are
very often doing exactly this.

**Executables cost a second inode.** You cannot `chmod` a hardlink without
`chmod`-ing the shared CAS blob, so an executable input needs its own copy,
one per digest, not one per action. This also sidesteps `ETXTBSY`, where
writing to a file that is currently executing fails.

**`work_directory` must be on the same filesystem as the CAS content path.**
Hardlinks cannot cross filesystem boundaries. This is the single most common
worker misconfiguration, and its symptom (everything works but is
inexplicably slow, or fails outright) does not visibly point at a mount
table.

**The fast store must be a `FilesystemStore`.** The worker downcasts to the
concrete type to find the file to link, and **refuses to start** if it is
given anything else. It does not degrade to copying.

<Callout type="info" title="Output parent directories are created inconsistently">
  Parents of `output_files` and `output_paths` are pre-created. Parents of
  `output_directories` are **not**. An action that expects its output
  directory's parent to exist has to create it.
</Callout>

## What isolation you actually get [#what-isolation-you-actually-get]

This section is deliberately blunt, because overclaiming here is dangerous.

**With the default configuration, a NativeLink worker provides no
OS-level isolation whatsoever.** `use_namespaces` defaults to `false`. The
action runs as an ordinary child process of the worker, with the worker's
privileges, on the worker's network, with access to whatever the worker can
reach.

Turning namespaces on improves this, but far less than the word "sandbox"
suggests. Here is the complete list, in both directions:

| Provided when enabled | **Not** provided, even when enabled |
| --- | --- |
| PID namespace | Network namespace; the action has the host's network |
| IPC namespace | seccomp filtering; every syscall is available |
| UTS namespace | cgroups; no CPU or memory limit is applied |
| User namespace (identity map only) | `chroot` / `pivot_root`; the host filesystem is visible |
| An init-stub reaper, via double fork | Any privilege drop; the uid map is identity, range 1 |
| Optional mount namespace that tmpfs-masks sibling action directories | Anything at all on macOS or Windows |

The user namespace maps one uid to itself. It does not sandbox the action's
identity; it exists so the other namespaces can be created without
privileges. Namespaces are **Linux-only** and hard-error elsewhere; this is
not a silent no-op you can leave enabled in a mixed fleet.

<Callout type="warn" title="Real isolation is delegated, not provided">
  If you run untrusted actions, the isolation boundary has to come from
  somewhere else: the `entrypoint` wrapper, which can invoke whatever sandbox
  you choose, or the container the worker itself runs in. NativeLink's
  namespace support is a hygiene measure (it stops actions tripping over
  each other's processes and mounts), not a security boundary. Note also that
  inside Docker, creating namespaces requires a privileged container, which
  trades away more than it buys back.
</Callout>

## The environment is built from nothing [#the-environment-is-built-from-nothing]

The worker calls `env_clear()` first. Nothing is inherited. Variables are
then applied in this order:

1. `additional_environment` from the worker's config
2. On Windows only, `SystemRoot` and `PATH`
3. The action's own `Command.environment_variables`, **last**

Two things follow. **Action-declared variables override worker config**, not
the other way round; worker config is a default, not a policy. And on Linux
and macOS **no `PATH`, `HOME` or `TMPDIR` is injected at all**. An action that
does not declare them does not have them, which is correct for hermeticity
and is why toolchains that assume a working `PATH` fail here. See
[toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity).

## Timeouts, signals, and the output you lose [#timeouts-signals-and-the-output-you-lose]

A requested timeout above `max_action_timeout_s` (20 minutes if unset) is
**rejected with `InvalidArgument`, not clamped**. The client is told its
request was unacceptable rather than silently receiving a shorter deadline
than it asked for. That is the right choice, and one worth knowing before you
set a low ceiling on a cluster whose clients ask for long ones. An action
that requests no timeout at all runs under `max_action_timeout_s`.

Three things about failure output are worth internalising:

**A timed-out or killed action returns empty stdout and stderr.** The
captured bytes are explicitly discarded, not truncated. The single most
useful diagnostic for "why did this hang" is the one you do not get.

**Signal death is reported as exit code 9**, indistinguishable from a genuine
`exit(9)`. An OOM kill and a program that chose to exit 9 look identical.

**stdout and stderr are buffered entirely in worker memory**, with no cap and
no streaming. An action that writes a gigabyte to stdout costs a gigabyte of
worker RAM, and no part of it is visible until the action ends.

## Uploading outputs, and what is dropped [#uploading-outputs-and-what-is-dropped]

The ordering here is subtle and matters for anyone reasoning about
consistency. When the command exits, the worker sends `ExecuteComplete` to
the scheduler (the process is done, so a new action can be assigned) and
*then* uploads outputs, deletes the action directory, and writes the action
cache entry. The final result update, which is what a client is waiting on,
comes after all of that, so a client that observes completion and
immediately queries the cache finds the result; but the scheduler knows the
command finished slightly before the bytes exist.
<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="execution_complete" />

<Mermaid>{`sequenceDiagram
    participant sched as Scheduler
    participant w as Worker
    participant cas as CAS + AC
    sched->>w: assign action (over ConnectWorker)
    w->>w: create work directory for the operation
    w->>cas: fetch missing blobs
    w->>w: hardlink inputs from filesystem CAS
    w->>w: env_clear(), then run the command
    w->>sched: ExecuteComplete (the process exited)
    w->>cas: upload outputs
    w->>w: delete the action directory
    w->>cas: write ActionResult to AC
    w->>sched: final result update
`}</Mermaid>

Four behaviours in the upload path:

**A declared-but-missing output is not an error.** An action that promises
`output_files` and does not produce them completes successfully with those
outputs absent. Bazel will notice; a hand-rolled client may not.

**Absolute symlinks are dereferenced and uploaded as content.** Only
*relative* output symlinks are preserved as symlinks. An action that emits an
absolute symlink gets a copy of the target instead, which is usually what
you want across machines, and occasionally very much not.

**`node_properties` is `None`.** Output file modes and mtimes are dropped. A
file that was executable in the action's output tree is not necessarily
executable when a client materialises it.

**`server_logs` is not implemented.** The field exists in the protocol; the
worker never populates it.

## Cleanup, and three things called eviction [#cleanup-and-three-things-called-eviction]

The entire action directory is deleted after every action, with a `Drop`
backstop for the panicking case and a path-canonicalization check so that a
malicious or malformed operation ID cannot direct the delete outside the work
directory.

Three separate mechanisms get called "eviction" in worker discussions, and
conflating them makes capacity planning impossible:

| Domain | Governed by | Behaviour |
| --- | --- | --- |
| The action directory | nothing | Deleted unconditionally after every action |
| The filesystem CAS | `eviction_policy.max_bytes` | LRU over stored blobs |
| The directory cache (only if `directory_cache` is configured) | `max_entries` (1000), `max_size_bytes` (10 GB) | LRU over materialized input trees |

Only the second is your cache size. The third is a distinct, much smaller
cache of materialized trees, off by default, and its defaults are not derived
from the second.

<Callout type="info" title="Persistent workers keep more than you might expect">
  A persistent worker process is spawned once (with the worker's own
  environment, and **without** namespaces or `env_clear()`) and reused across
  actions. Each action still gets its own materialized input directory, which
  is handed to the process as the request's `sandbox_dir`, but the process
  itself carries whatever state it accumulates from one request to the next.
  That is the point (it is what makes them fast), but it also means a
  persistent worker is a shared mutable environment across the actions it
  serves. See [persistent workers](/remote-execution/persistent-workers).
</Callout>

## Where the exact truth lives [#where-the-exact-truth-lives]

Every worker field and default is in the generated
[worker configuration reference](/reference/nativelink-config#workerconfig). The
execution code itself is
<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="RunningActionsManager" />
and
<SourceLink file="nativelink-worker/src/local_worker.rs" symbol="LocalWorker" />.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="My action works locally and fails remotely with EACCES. Why?" id="eacces-on-inputs">
    It is almost certainly modifying an input file in place. Inputs are
    hardlinks to read-only CAS blobs. Copy the file into the output tree
    first, or declare it as an output rather than an input.
  </Accordion>
  <Accordion title="Is it safe to run untrusted actions on a NativeLink worker?" id="untrusted-actions">
    Not on the strength of NativeLink's own isolation. With defaults there is
    none; with namespaces enabled there is still no network isolation, no
    seccomp, no cgroups, no filesystem root change and no privilege drop. Put
    the boundary in the `entrypoint` wrapper or in the container the worker
    runs in, and size that boundary for genuinely hostile input.
  </Accordion>
  <Accordion title="Why did my action time out with no logs at all?" id="timeout-no-logs">
    Because output from a timed-out or killed action is discarded rather than
    returned. To get diagnostics out of a hanging action, have it write to a
    declared output file rather than to stdout, or lower the action's own
    internal timeout so it exits normally before the worker kills it.
  </Accordion>
  <Accordion title="Can I put work_directory on a different disk from the CAS?" id="work-directory-separate-disk">
    No. Materialization is hardlink-based and hardlinks cannot cross
    filesystems. Both must live on the same filesystem, which is also the
    thing to check first when a worker is unexpectedly slow.
  </Accordion>
  <Accordion title="Why is my worker using so much memory on one action?" id="worker-memory-one-action">
    stdout and stderr are buffered entirely in memory with no cap. An action
    that is chatty on stdout costs RAM proportional to everything it has
    printed, held until it exits.
  </Accordion>
</Accordions>

<NextStep href="/explanations/correctness-hermeticity" title="Correctness and hermeticity">
  Which of these guarantees you get by default, which you have to configure,
  and which the protocol never offered in the first place.
</NextStep>

# https://docs.nativelink.com/explanations/correctness-hermeticity

**Correctness and hermeticity**: What content addressing guarantees, what it doesn't, and which of the gaps are yours to close.

**Who this is for:** anyone who needs to know how much to trust a cache hit,
and anyone chasing a build that is correct on one machine and wrong on
another. **What you'll have at the end:** an honest inventory of the
guarantees: the ones you get for free, the ones you have to configure, and
the ones nobody is offering. **Time:** twenty minutes.

"Content-addressed" is often heard as "therefore correct". It is not. Content
addressing gives you one very strong property (*if two things have the same
hash, they are the same bytes*), and that property is only as useful as the
process that assigned the hash. Most real correctness incidents in a remote
build system happen in the space between "the hash matches" and "the hash
meant anything".

This page is the inventory. It leans on
[the store model](/explanations/store-model),
[scheduler internals](/explanations/scheduler-internals) and
[worker execution](/explanations/worker-execution) for mechanism, and does not
repeat them.

## What you get for free [#what-you-get-for-free]

**Collision resistance.** SHA-256 and BLAKE3 are the supported digest
functions. Two different blobs will not share a key.

**Immutability by construction.** A blob is stored under a key derived from
its content, so an overwrite is a no-op rather than a mutation. Concurrent
uploads of the same blob cannot interleave into a corrupt result.

**Deduplication across everything.** One instance name, one digest function,
one copy, regardless of which repository, which developer, or which CI job
produced it.

**Action identity.** Two actions with the same instance name, digest function
and action digest are the same action, and the system will merge them rather
than run both.

## What you do not get by default [#what-you-do-not-get-by-default]

Five gaps, in rough order of how often they bite.

### The digest is a claim until you verify it [#the-digest-is-a-claim-until-you-verify-it]

The server does not hash plain uploaded bytes. The client declares the digest
and the server files the bytes under it. (Zstd-compressed ByteStream uploads
are the exception: the decoder checks size and hash as it decodes.) For
everything else the only thing that checks is a `verify` store, and both
`verify_hash` and `verify_size` default to `false`.

A cache without one will faithfully store, and forever serve, a blob under a
digest that does not describe it. Every downstream consumer that trusts the
digest (which is all of them) then gets the wrong bytes with no error
anywhere. This is the highest-severity, lowest-visibility failure mode in the
system.

**What to do:** put a `verify` store with `verify_hash: true` in the write
path of any cache shared between more than one person. The cost is one hash
pass, run concurrently with the write.

### Verification is write-side only [#verification-is-write-side-only]

Even with a `verify` store, nothing is re-checked on read. Blobs that entered
before you added the wrapper, or through a path that bypassed it, are served
without complaint.

**What to do:** treat adding verification as a change that only protects
blobs uploaded from that point forward. If you have reason to distrust
existing content, the answer is a new instance name, not a config change.

### Hermeticity is the client's job [#hermeticity-is-the-clients-job]

The action digest covers exactly what the client put into the `Action`
message: the command, the input tree, the platform properties, the
environment variables it declared. Anything the action reads that is not in
there is invisible to the cache, and a cache hit will confidently return a
result computed with a different value of it.

NativeLink helps in one significant way: the worker calls `env_clear()` and
injects no `PATH`, `HOME` or `TMPDIR` on Linux or macOS, so an action that
depends on an undeclared environment variable tends to fail loudly rather
than succeed differently.

It does not help in the way that matters most: **there is no network
isolation, with or without namespaces enabled.** An action that fetches from
the network will do so, will succeed, and will produce a result cached under
a digest that says nothing about what it downloaded. That is the classic
poisoned-cache-entry mechanism, and no server-side setting prevents it.

**What to do:** pin toolchains and inputs at the client. `--incompatible_strict_action_env`
and equivalent flags exist for this. [Local Remote Execution](/explanations/lre)
is the toolchain half of the answer.

### Actions can run more than once [#actions-can-run-more-than-once]

The default `max_job_retries` of 3 is checked strictly, so an action may
execute **four** times. A worker disconnect always costs an attempt; a
`ResourceExhausted` never does.

For a pure function this is invisible. For an action with side effects (one
that publishes, deploys, increments, or writes anywhere outside its declared
outputs) it means the side effect happens up to four times, and NativeLink
has no way to know that it should not.

**What to do:** actions with side effects do not belong in a remote execution
system. If one must exist, make it idempotent, and set `max_job_retries: 0`
for the instance that runs it.

### Some output metadata is discarded [#some-output-metadata-is-discarded]

`node_properties` is never populated, so output file modes and mtimes do not
survive a round trip. Absolute output symlinks are dereferenced and uploaded
as content; only relative ones stay symlinks. A declared output that the
action did not produce is not an error.

**What to do:** do not depend on an output's mode. If an action produces
something that must be executable, the consumer has to set that itself.

## Caching semantics that surprise people [#caching-semantics-that-surprise-people]

Three, all of which have bitten someone:

**`do_not_cache` does nothing.** NativeLink parses the REAPI flag and no code
reads it. Setting it does not prevent a result being written.

**`skip_cache_lookup` still writes.** It disables the cache read and disables
merging, but the result of the re-execution is written to the cache. This is
the correct behaviour for "re-run this and fix the cache", and the wrong
mental model if you thought it meant "run this without touching the cache".

**Failures are not cached, by default.** A non-zero exit code goes to the
historical results store, not the action cache, so a failing compile is
re-attempted rather than replayed. That is usually what you want and
occasionally not; a deterministic failure will be re-run by every client
that encounters it.

## A checklist for a cache you can trust [#a-checklist-for-a-cache-you-can-trust]

<Steps>
  <li>
    **Put `verify_hash: true` in the write path.** Everything below is
    downstream of this being true.
  </li>
  <li>
    **Pin the toolchain at the client**, so the compiler's identity is inside
    the action digest rather than beside it.
  </li>
  <li>
    **Make undeclared inputs fail rather than succeed.** Strict action
    environment, strict include checking, no reliance on ambient `PATH`.
  </li>
  <li>
    **Assume the network is reachable from actions**, and design so that no
    action wants it.
  </li>
  <li>
    **Set `max_action_executing_timeout_s`.** It defaults to disabled. The
    worker's own `max_action_timeout_s` (20 minutes by default) still kills a
    hung process, but an action whose worker is alive and heartbeating yet
    never reports on it has no ceiling without this.
  </li>
  <li>
    **Separate trust domains with instance names**, not with configuration
    inside one. Instance names are the only isolation boundary the protocol
    actually has.
  </li>
  <li>
    **Verify by rebuilding.** Build a target twice on different machines and
    compare. It is the only test that covers the leaks a checklist misses.
  </li>
</Steps>

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="How do I find a hermeticity leak?" id="find-a-hermeticity-leak">
    Build the same target on two machines that differ, and compare outputs
    byte for byte; anything that differs is reading something undeclared.
    At cluster scale the symptom is subtler: cache misses where you expected
    hits, because a leaked input changed and moved the action digest with it.
    A rising miss rate on unchanged code is the signal.
  </Accordion>
  <Accordion title="Someone poisoned a cache entry. What now?" id="poisoned-cache-entry">
    You cannot selectively invalidate by content, because the key is the
    content. The practical answers are to move to a new instance name, which
    is a clean namespace with no migration, or to delete the affected
    action-cache entries if you can identify them. Then add the `verify` store
    that would have prevented it, and find the client that was wrong.
  </Accordion>
  <Accordion title="Is a cache hit as trustworthy as a real execution?" id="is-a-cache-hit-trustworthy">
    It is exactly as trustworthy as the execution that produced it, plus the
    integrity of the path in between. Content addressing secures the second
    part, once verification is on. The first part is the client's hermeticity
    and nothing else.
  </Accordion>
  <Accordion title="Does turning on verification slow uploads down?" id="does-verification-slow-uploads">
    Marginally. The check runs concurrently with the write rather than as a
    separate pass, so the cost is CPU for one hash over uploaded bytes, not an
    extra round trip. On any realistic network the transfer dominates.
  </Accordion>
  <Accordion title="Do I need namespaces enabled to get correct results?" id="namespaces-and-correctness">
    No. Namespaces are about isolation between concurrent actions, not about
    result correctness, and they provide no network, seccomp or cgroup
    isolation in any case. Correct results come from the action declaring
    everything it reads. Enable namespaces to stop actions interfering with
    each other; do not enable them expecting hermeticity.
  </Accordion>
</Accordions>

<NextStep href="/explanations/lre" title="Local Remote Execution">
  The toolchain-pinning half of hermeticity, and the one piece of it that
  NativeLink ships rather than delegates.
</NextStep>

# https://docs.nativelink.com/explanations/lre

**Local Remote Execution**: What LRE actually is: a way to make your local toolchain and your remote workers the same toolchain, so both sides share one cache.

**Who this is for:** anyone deciding whether LRE is worth adopting, and
anyone who has read about it and formed the wrong model of what it does.
**What you'll have at the end:** the actual mechanism, the guarantee it makes
and its exact qualifiers, and what it does not cover. **Time:** fifteen
minutes.

Local Remote Execution is a way of building your toolchain with Nix such that
**the exact same toolchain binaries are used by your local build and by your
remote workers**. Because the toolchain is part of every action's digest, the
two sides then produce identical action digests, and a cache entry written
by one is a cache hit for the other.

That is the whole idea. It is worth stating what it is *not*, because the
name invites a different reading.

<Callout type="warn" title="LRE does not mean running a worker on your laptop">
  No local NativeLink worker exists in this design, and no container runs
  on your machine during a build. Your local build is an ordinary local
  Bazel build. What LRE changes is *which compiler that build invokes*. The
  toolchain container images are, in the source's own words, "never really
  instantiated"; they exist to have their environment extracted.
</Callout>

## The mechanism, precisely [#the-mechanism-precisely]

The trick is absolute paths.

A Nix-built toolchain lives at a path like
`/nix/store/<hash>-clang-…/bin/clang`, where the hash is derived from every
input that produced it. LRE generates Bazel toolchain configurations that
hardcode those absolute store paths: not `/usr/bin/cc`, not a `PATH` lookup,
the literal store path.

Then it arranges for that exact path to exist on both sides:

<Mermaid>{`graph TD
  nix["Nix derivation<br/>/nix/store/&lt;hash&gt;-clang"]
  nix --> gen["Generated Bazel toolchain<br/>hardcodes the store path"]
  nix --> img["Worker image<br/>Env/PATH = the same store paths"]
  nix --> shell["nix develop<br/>realizes the closure locally"]
  gen --> local["Local build<br/>invokes /nix/store/&lt;hash&gt;-clang"]
  gen --> remote["Remote action<br/>invokes /nix/store/&lt;hash&gt;-clang"]
  shell --> local
  img --> remote
  local --> same["Same digest → one shared cache"]
  remote --> same
`}</Mermaid>

**On the remote side**, the worker image is built with `nix2container` and
its environment is composed entirely of `/nix/store/…` paths. The image is a
carrier for that closure rather than a conventional root filesystem.

**On the local side**, entering the Nix development shell realizes the same
derivations into your own `/nix/store`. This is what the commented-out `PATH`
lines in the generated `lre.bazelrc` are for: Bazel ignores them, but Nix
needs a reason to fetch the closure, and those lines are it. The
<SourceLink file="local-remote-execution/README.md" symbol="local-remote-execution/README.md" />
says so directly: "all paths printed in `lre.bazelrc` will be available on
your local system."

What is shared between local and remote is **the store paths**, not the
image. Understanding that explains almost every question people have about
LRE's edges.

## The guarantee, with its qualifiers intact [#the-guarantee-with-its-qualifiers-intact]

The source states the promise carefully, and the qualifiers are load-bearing:

> a framework to build, distribute, and rapidly iterate on custom toolchain
> setups that are transparent, fully hermetic, and reproducible **across
> machines of the same system architecture** […] This lets you reuse build
> artifacts with **virtually** perfect cache hit rate.

Both emphases are the source's meaning, not an editorial softening.
Reproducibility is claimed within an architecture, not across them; an
`aarch64-darwin` laptop and an `x86_64-linux` worker are not going to share
C++ artifacts, because they are not running the same binaries. And "virtually
perfect" acknowledges that the toolchain is one input among several, and LRE
only pins the toolchain.

<Callout type="warn" title="LRE is explicitly experimental">
  The repository's own documentation opens with a warning that LRE is "highly
  experimental and can be challenging to set up in downstream projects", and
  a note that at the time of writing it "only works on `x86_64-linux`". The
  Rust toolchains have since grown to four systems; the C++ toolchain has
  not. Treat the setup cost as real.
</Callout>

## What is pinned, and what is not [#what-is-pinned-and-what-is-not]

This is where careful reading pays off.

**Not the image digest.** Every `container-image` value in the generated
platforms is `name:tag`, where the tag is a Nix derivation hash, never a
`sha256:` content digest. The upstream config generator's digest-resolution
step was deliberately patched out, with the in-source justification that
"skipping this hash is not a generally safe practice and only works for us
because our images are reproducible."

**Not a URL that gets pulled.** `container-image` is a *raw string matched by
the scheduler* against what a worker advertises. NativeLink does not fetch it.
Two different conventions coexist in-tree (`docker://lre-cc:<tag>` for C++
and a `ghcr.io/...` reference for Rust), and a downstream user has to match
whatever string their own workers advertise. See
[platform properties](/remote-execution/platform-properties).

**Not the image you actually deploy.** Three distinct images are in play: a
generator image, the image the platform config nominally names, and the
worker image that actually runs. The repository warns about this in its own
deployment manifests. The relationship is maintained by convention and by
tag propagation, not by any mechanism that would catch a mismatch.

**Not a single shared closure.** The local closure is whatever the dev shell
realizes; the worker image's closure is that plus extras the worker needs.
They are compatible by construction, not identical.

## Coverage [#coverage]

| Language | Status |
| --- | --- |
| **Rust** | Full toolchains across four Nix systems, stable and nightly |
| **C/C++** | Full toolchain, **`x86_64-linux` only** |
| **Java** | An image and a generated runtime exist, but no Bazel toolchain is registered |
| **Python, Go** | Not provided |

Windows is not supported. On macOS the C++ path has no local toolchain at
all (the flake module only emits the C++ platform and toolchain flags on
Linux), so an `lre-cc` action can only run on a Linux worker carrying the
image; you get hermeticity there, not locality.

<Callout type="info" title="LRE is not an offline mode">
  It is tempting to read "local remote execution" as "remote execution without
  the network". It is not. LRE pins the toolchain; whether a given action
  runs locally or on a remote worker is a separate question decided by
  platform matching. On the platforms where the toolchain exists locally,
  builds are local, but that is a consequence, not the guarantee.
</Callout>

## Why Nix, specifically [#why-nix-specifically]

Because the design needs every tool the build invokes to be identified by a
hash of everything that produced it, and to live at a path that encodes that
hash. That is precisely what a Nix store path is.

`/usr/bin/clang` cannot be pinned. Its identity is whatever your package
manager last installed, it differs across machines that both call themselves
"Ubuntu 22.04", and no amount of hashing the binary recovers the headers and
libraries it will reach for. `/nix/store/<hash>-clang-…` has already answered
all of those questions by existing.

Everything else about LRE follows from that one property. The container
images, the generated Bazel toolchains, the `lre.bazelrc`, all of it is
plumbing to get those store paths onto both sides of the build.

For Nix itself, see [Develop with Nix](/contribute/nix).

## When it is worth it [#when-it-is-worth-it]

| Situation | LRE? |
| --- | --- |
| Cache hits are poor between developers and CI on the same architecture | Yes; this is the case it was built for |
| You already build with Bazel and already use Nix | Yes; the marginal cost is small |
| You want C++ hermeticity on `x86_64-linux` | Yes |
| Your fleet is mixed-architecture and you expected cross-arch sharing | No; the guarantee does not extend there |
| You need Python or Go toolchains pinned | Not yet |
| You are looking for sandboxing or isolation | No; different problem, see [correctness and hermeticity](/explanations/correctness-hermeticity) |
| You want a quick win with a low setup cost | No; the source calls the setup challenging, and means it |

## Where the exact truth lives [#where-the-exact-truth-lives]

The flake-side and Bazel-side wiring, the regeneration commands, and the
downstream-project setup are in
<SourceLink file="local-remote-execution" dir symbol="local-remote-execution/" />.
For actually running it (the commands, and what is local versus remote once
you leave `x86_64-linux`) see
[toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity),
which is in the Remote execution section.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Does LRE replace a remote worker fleet?" id="lre-vs-fleet">
    No, and it does not try to. LRE is about toolchain identity, not about
    where work runs. Fan-out across hundreds of cores still wants remote
    workers; the point is that the workers and your laptop now agree on what
    a compiler is, so both populate one cache.
  </Accordion>
  <Accordion title="Do I need Docker installed?" id="do-i-need-docker">
    The README lists "a functional local container setup" alongside Nix as a
    prerequisite. Where it is actually exercised in-tree is regenerating the
    toolchain configurations, which loads a generator image into the local
    container daemon; consuming the checked-in configs is driven by Nix.
  </Accordion>
  <Accordion title="I'm getting cache misses between local and remote. Where do I look?" id="cache-misses-local-vs-remote">
    Compare the `/nix/store/...` paths in your generated `lre.bazelrc`
    against the ones in the toolchain configs of the `local-remote-execution`
    module at the commit your project pins. A mismatch there is the usual
    cause, and it usually means the flake input and the Bazel module override
    have drifted apart. They are pinned in two separate files and nothing
    checks that they agree.
  </Accordion>
  <Accordion title="Does one command regenerate all the toolchains?" id="regenerate-all-toolchains">
    No. `generate-toolchains` regenerates the Java configuration only; C++ and
    Rust have their own separate commands. No single in-tree target runs all
    three, so a flake update needs all of them.
  </Accordion>
  <Accordion title="Is the C++ toolchain fully hermetic?" id="is-the-cc-toolchain-fully-hermetic">
    Not quite, and the source says so. In-tree TODOs in
    `local-remote-execution/overlays/stdenv.nix` record that the result is
    not a pure Clang/LLVM toolchain, that an unwanted `libgcc_s` dependency
    remains, and that the stdenv used for the Cargo build and the "mostly
    hermetic" LLVM toolchain used by the Bazel build are still two separate
    toolchains. LRE substantially improves hermeticity; it does not currently
    complete it.
  </Accordion>
</Accordions>

<NextStep href="/remote-execution/toolchains-and-hermeticity" title="Toolchains and hermeticity">
  The same ideas as commands you can run, in the Remote execution section.
</NextStep>

# https://docs.nativelink.com/explanations/history

**History and licensing**: What NativeLink was built to replace, the three properties that drove the rewrite, and exactly how the source is licensed.

**Who this is for:** anyone evaluating NativeLink against an existing build
farm, and anyone who needs to answer a licensing question precisely. **What
you'll have at the end:** the design pressures that produced the current
system, and a file-by-file account of what license applies. **Time:** ten
minutes.

Remote build execution is not a new idea. Google shipped an internal version
of it long before the
[Remote Execution API](https://github.com/bazelbuild/remote-apis) was
published, and by the early 2020s there were several open implementations of
the published protocol. NativeLink started in December 2020 as a Rust
implementation of the CAS half of that protocol (first called Rust CAS, then
Turbo Cache from April 2022), was renamed to NativeLink in December 2023
(the `cas` executable became `nativelink` at the same time), and cut its
first tagged release, 0.1.0, on 2023-12-20. The
argument for writing a new implementation rather than running an existing
one is the one below: the protocol is narrow enough that a rewrite is
tractable, and what was expensive in the existing options was the substrate.

## What the existing options traded away [#what-the-existing-options-traded-away]

| Family | Strength | The recurring problem |
| --- | --- | --- |
| JVM-based servers | Mature, complete, well understood | Tail latency under GC pressure, and a memory footprint that makes horizontal scaling the only answer |
| Go services | Fast on the happy path, straightforward to operate | Contention on hot artifacts and on control-plane locks, which shows up as a cluster-wide stall rather than a slow request |
| Proprietary clouds | Fast, someone else operates it | No source access, and pricing that scales with the thing you are trying to do more of |

Two of those three problems are runtime problems rather than design problems.
That observation is the whole argument for the rewrite: the protocol was
fine, the storage model was fine, and what was expensive was the substrate.

## The three properties that drove the design [#the-three-properties]

**Predictable latency.** In a build system, the p99 is the number developers
experience: a single stalled action blocks everything downstream of it in
the graph. Removing the garbage collector removes a class of latency spike
that is very hard to tune away rather than eliminate.

**Memory safety without a runtime.** A build cache that serves a *wrong*
artifact is a far worse incident than one that serves *no* artifact: it is
silent, and it propagates to everyone who pulls from the cache. Rust removes
the memory-corruption path to that outcome at compile time, without adding a
managed runtime back in.

**One binary, many roles.** CAS, action cache, scheduler and worker are the
same executable with different configuration. It makes the small deployment
trivial and the large deployment a scaling exercise rather than a
re-architecture. [Architecture](/explanations/architecture) covers what that
means in practice, and
[the deep dive](/explanations/architecture-deep-dive) covers what the binary
does with the config it is handed.

A fourth property arrived with the project rather than driving it: because
the toolchain definitions could be shared between the local machine and the
remote workers, the same hermetic build could run in both places. That became
[Local Remote Execution](/explanations/lre), and it is the part of NativeLink
with no real equivalent elsewhere.

## Licensing, precisely [#licensing]

This question gets answered vaguely often enough to be worth stating exactly.

The repository's root
<SourceLink file="LICENSE" symbol="LICENSE" />
is the **Functional Source License 1.1 with an Apache 2.0 future grant**
(`FSL-1.1-Apache-2.0`). In practice that means two things. You may use, read,
modify and deploy the software for any purpose *except* offering it to others
as a competing commercial product or service. And each release converts to
plain Apache 2.0 on the second anniversary of its publication; the grant is
irrevocable and written into the license itself, not a promise made
elsewhere.

Five files are **Business Source License** instead, and they are worth
knowing by name because they are the ones an internal license review will
flag:

| File or module | What it is |
| --- | --- |
| `nativelink-util/src/metrics.rs` | The metrics collection layer |
| `nativelink-worker/src/persistent_worker/`: `mod.rs`, `pool.rs`, `live_worker.rs`, `protocol.rs` | Remote persistent workers |

<Callout type="info" title="Copy the header, don't recall it">
  When you add a file, copy the license header from the file next to it
  rather than from memory. A new file under `persistent_worker/` needs the
  BUSL header; a new file in the same crate's `src/` does not.
</Callout>

Nothing in the repository requires a contributor license agreement or a
Developer Certificate of Origin sign-off. What it does require is
cryptographically signed commits; see
[the contribution guidelines](/contribute/guidelines).

## Versioning [#versioning]

NativeLink versions look like SemVer and are not.
<SourceLink file="MODULE.bazel" symbol="module()" />
declares `compatibility_level = 0`, which is Bazel's way of saying no
compatibility promise is being made across versions. The config schema and
the metrics surface have both changed in minor releases; the changelog marks
such entries as breaking.

The wire protocol is the exception, and it is the one that matters for most
deployments: NativeLink implements the Remote Execution API, which is
versioned upstream and changes slowly. A client that worked against one
NativeLink release will keep working against the next.
[Release and versioning](/contribute/release-versioning) is the page for
the details.

## Why these docs were rewritten [#why-these-docs-were-rewritten]

The previous documentation accumulated during the period when the project
was changing shape weekly. It described commands that no longer existed,
components that had been renamed, and performance numbers with no source.
This rewrite holds every page to the same contract: every command is run or
schema-checked before it ships, every claim about the code links to the code,
and pages that cannot meet that bar say what they do not know.

The source lives in
<SourceLink file="web/apps/docs/content/docs" dir symbol="content/docs/" />
and corrections are the most welcome kind of pull request.
[Working on documentation](/contribute/docs) is the how.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Is NativeLink open source?" id="is-nativelink-open-source">
    It is source-available today and open source on a timer. The FSL 1.1
    grant permits everything except offering NativeLink itself as a
    competing commercial service, and each release converts to Apache 2.0
    two years after publication. If your concern is "can I read and modify
    the code that runs my builds", the answer is yes, without qualification.
  </Accordion>
  <Accordion title="Can I use it commercially?" id="can-i-use-it-commercially">
    Yes. Running it to build your own software is a permitted purpose,
    commercial or not. The restriction is on reselling NativeLink as a
    competing product or service. The two BUSL modules are the ones to
    check with counsel if you plan to redistribute.
  </Accordion>
  <Accordion title="Why Rust and not Go?" id="why-rust-and-not-go">
    Predictable tail latency without a garbage collector, and memory safety
    without a managed runtime. The cost is compile time and a smaller pool
    of contributors, a real trade, made deliberately, for a component whose
    worst failure mode is silently serving the wrong bytes.
  </Accordion>
  <Accordion title="Does the version number mean anything?" id="does-the-version-number-mean-anything">
    Less than it looks like. `compatibility_level = 0` in `MODULE.bazel`
    means no cross-version compatibility guarantee. Read the changelog
    before upgrading, and pin the version in your deployment.
  </Accordion>
</Accordions>

<NextStep href="/explanations/lre" title="Local Remote Execution">
  The idea that came out of the rewrite rather than driving it: the same
  hermetic toolchain on your laptop and on the remote workers.
</NextStep>

# https://docs.nativelink.com/reference/nativelink-config

**Configuration reference**: Every knob in the NativeLink JSON5 configuration — types, defaults, and links to source, autogenerated from the Rust config crate.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/nativelink-config._

# https://docs.nativelink.com/reference/nativelink-config/store-overview

**Store overview**: The mental model behind every store in the configuration reference: what holds data, what wraps another store, and how they compose.

The [full reference](/reference/nativelink-config) lists every field on
every store. This page is the map you need before that list is useful:
what a store actually is, the two families every store falls into, and
the compositions people actually run in production.

## What a store is [#what-a-store-is]

A store is a named, pluggable backend for CAS or AC data. You declare
each one once in the top-level `stores` array, then reference it by
name from `servers`, `workers`, or from inside another store:

```json5
{
  stores: [
    { name: "CAS_MAIN_STORE", filesystem: { /* ... */ } },
  ],
  servers: [{
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
    },
  }],
}
```

The store's *type* is whichever key you put inside its object:
`filesystem`, `memory`, `compression`, `fast_slow`, and so on. NativeLink
picks the implementation from that key; everything else in the object is
that implementation's config. The set of legal keys is the variant list of a
single enum, so the reference below and the parser cannot disagree.
<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

## Two families [#two-families]

Every store type is either **terminal** (it actually holds or fetches
bytes) or a **wrapper** (it adds behavior in front of another store,
which it embeds as a nested `StoreSpec`). Wrappers nest arbitrarily
deep: a `verify` can wrap a `compression`, which wraps a `fast_slow`,
which wraps two more terminal stores:

<Mermaid>
{`flowchart LR
    V["verify\\n(hash + size check)"] --> C["compression\\n(lz4)"]
    C --> F["fast_slow"]
    F -->|fast| M[("memory")]
    F -->|slow| D[("filesystem")]`}
</Mermaid>

Reads and writes flow through every wrapper in order. Nothing about the
client changes; it still speaks plain CAS/AC RPCs against whatever
store name the server exposes.

### Terminal stores: where bytes actually live [#terminal-stores--where-bytes-actually-live]

| Store key | Holds data in | Notes |
| --- | --- | --- |
| `memory` | An in-process hash map | Fastest, gone on restart. Good for AC tiers and dev. |
| `filesystem` | Local disk | Survives restarts; scans and rebuilds its index on startup. |
| `experimental_cloud_object_store` | A cloud object bucket | One store type, six `provider`s: `aws` (S3), `gcs`, `azure`, `ontap` (NetApp ONTAP S3), `r2` (Cloudflare), `oci` (Oracle Cloud Infrastructure). All speak an S3-compatible API except `gcs` and `azure`. |
| `redis_store` | Any Redis-API-compatible service | Pairs well with `size_partitioning`; most Redis services cap uploads around 256 to 512 MB. |
| `experimental_mongo` | MongoDB | Supports CAS and scheduler data, with optional change streams for scheduler subscriptions. |
| `grpc` | Another NativeLink-compatible gRPC endpoint | Proxies calls upstream. Useful for a regional cache in front of a central cluster. |
| `ref_store` | Nothing itself; points at another store by name | Lets two composition trees share one underlying store instance (see below). |
| `noop` | Nothing | Reads 404. Writes vanish. Used to explicitly discard a partition of data. |

Every implementation in this table lives in `nativelink-store/src`.
<SourceLink file="nativelink-store/src" dir symbol="nativelink-store/src/" />

### Wrapper stores: behavior layered on a backend [#wrapper-stores--behavior-layered-on-a-backend]

| Store key | Wraps | What it adds |
| --- | --- | --- |
| `compression` | one store | LZ4-compresses on write, decompresses on read. |
| `dedup` | two stores (`index_store` + `content_store`) | Rolling-hash chunking so only changed slices upload. |
| `fast_slow` | two stores (`fast` + `slow`) | Reads try `fast` first, fall back to `slow`, and backfill `fast`. Writes mirror to both. |
| `shard` | N stores | Routes by digest hash. The standard shape for scaling CAS past one backend. |
| `size_partitioning` | two stores (`lower_store` + `upper_store`) | Routes by blob size instead of hash. CAS-only; see the gotcha below. |
| `verify` | one store | Rejects uploads that fail a hash and/or size check before they ever reach the backend. |
| `existence_cache` | one store | Caches `has()` results. CAS-only. |
| `completeness_checking` | one store + a CAS store | Confirms an `ActionResult`'s output digests exist in CAS before returning it. AC-only. |
| `cache_metrics` | one store | Emits OpenTelemetry cache-hit/miss metrics for the wrapped store. Opt-in; stores you don't wrap pay nothing extra. |
| `ontap_s3_existence_cache` | (built-in ONTAP S3 backend) | Purpose-built existence cache for ONTAP S3 specifically, with disk-persisted sync instead of a generic wrapped backend. |

The nested-`StoreSpec` shape each wrapper takes (one backend, two named
backends, or N) is declared alongside the terminal specs.
<SourceLink file="nativelink-config/src/stores.rs" symbol="StoreSpec" />

<Callout type="warn" title="A few wrappers only make sense one way round">
  - Put `compression` *inside* `dedup` (as its `content_store`, so each
    chunk is compressed after chunking), never the other way round;
    `dedup` as `compression`'s backend negates dedup's gains, because
    chunking then runs over an already-compressed stream whose bytes change
    completely on any edit.
    <SourceLink file="nativelink-config/src/stores.rs" symbol="DedupSpec" />
  - `fast_slow` never checks whether an object in `fast` also exists in
    `slow`. If you need a durability guarantee (e.g. remote execution
    artifacts that must survive a `fast`-tier wipe), write-through both
    tiers deliberately rather than assuming it.
    <SourceLink file="nativelink-store/src/fast_slow_store.rs" symbol="FastSlowStore" />
  - `existence_cache` and `size_partitioning` are CAS-only; `completeness_checking`
    is AC-only. Using them on the other store type produces confusing
    correctness bugs, not a config error.
</Callout>

## Choosing a terminal backend [#choosing-a-terminal-backend]

| Backend | Durability | Shared across instances | Typical role |
| --- | --- | --- | --- |
| `memory` | None (restart wipes it) | No | Fast tier in a `fast_slow`, or a whole AC store for a short-lived CI runner. |
| `filesystem` | Survives restarts | No (single node) | Single-node dev/CI cache, or the fast/local tier in front of shared storage. |
| `experimental_cloud_object_store` | Durable, provider-managed | Yes | The shared, multi-node backing store in most production clusters. |
| `redis_store` | As durable as your Redis deployment | Yes | Low-latency shared tier, usually fronted by `size_partitioning` to keep large blobs out of it. |
| `experimental_mongo` | Durable | Yes | Alternative shared backend when you already operate MongoDB and want scheduler-state change streams. |

## Common compositions [#common-compositions]

**Single-node dev cache**: no wrapping at all. See
[Configuration → Stores](/configuration/stores).

**Validated, compressed, tiered CAS**: the shape most self-hosted
production clusters converge on:

```json5
{
  name: "CAS_MAIN_STORE",
  verify: {
    verify_size: true,
    verify_hash: true,
    backend: {
      compression: {
        compression_algorithm: { lz4: {} },
        backend: {
          fast_slow: {
            fast: { memory: { eviction_policy: { max_bytes: "2gb" } } },
            slow: {
              experimental_cloud_object_store: {
                provider: "aws",
                region: "us-east-1",
                bucket: "nativelink-cas",
              },
            },
          },
        },
      },
    },
  },
}
```

**Sharing one store instance across two trees**: use `ref_store`
instead of declaring the same backend twice:

```json5
{
  stores: [
    {
      name: "FS_CONTENT_STORE",
      filesystem: { content_path: "/var/lib/nativelink/cas", temp_path: "/var/lib/nativelink/tmp" },
    },
    {
      // The worker's fast/local tier and the AC's fast tier now share
      // one filesystem store instead of each scanning their own copy.
      name: "AC_MAIN_STORE",
      fast_slow: {
        fast: { ref_store: { name: "FS_CONTENT_STORE" } },
        slow: { noop: {} },
      },
    },
  ],
}
```

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Does fast_slow guarantee the slow tier has everything?" id="does-fast-slow-guarantee-the-slow-tier-has-everything">
    No; it never checks that an object in `fast` also made it to `slow`.
    If an artifact must survive a fast-tier wipe, make sure the write
    path covers both tiers deliberately.
    <SourceLink file="nativelink-store/src/fast_slow_store.rs" symbol="FastSlowStore" />
  </Accordion>
  <Accordion title="Why pair redis_store with size_partitioning?" id="why-pair-redis-store-with-size-partitioning">
    Redis caps uploads at roughly 256-512 MB. Route large blobs to a
    disk- or object-store backend with `size_partitioning` and keep Redis
    serving the small, hot objects it's good at.
  </Accordion>
  <Accordion title="Which wrappers only apply to the CAS or the AC?" id="which-wrappers-only-apply-to-the-cas-or-the-ac">
    `existence_cache` and `size_partitioning` are CAS-only;
    `completeness_checking` is AC-only. Misplacing one isn't a config
    error; it surfaces as a confusing correctness bug.
    <SourceLink file="nativelink-store/src/completeness_checking_store.rs" symbol="CompletenessCheckingStore" />
  </Accordion>
</Accordions>

## What's next [#whats-next]

- [Configuration reference](/reference/nativelink-config): every
  field, default, and JSON5 example for every store type above.
- [Configuration](/configuration): how stores fit alongside servers,
  schedulers, and workers.
- [Production configuration](/operate/production-config): the store
  composition a real cluster uses end to end.

# https://docs.nativelink.com/reference/protocol-api

**Protocol and API surface**: Every gRPC service NativeLink serves, which RPCs are implemented, and the values it advertises.

**Who this is for:** anyone writing or debugging a client, or checking
whether a given REAPI feature is available. **What you'll have at the end:**
the complete service list with implementation status. **Time:** use as a
lookup.

NativeLink implements the [Remote Execution API](https://github.com/bazelbuild/remote-apis)
plus the adjacent protocols it depends on (ByteStream, long-running
Operations, Remote Asset and the Build Event Protocol), and defines exactly
one service of its own. Every vendored `.proto` under `nativelink-proto/` is
copied from upstream with only its licence header replaced; there are no
NativeLink-specific fields or extensions on the standard messages.

## Services [#services]

| Service | Origin | Status |
| --- | --- | --- |
| `Execution` | REAPI | Implemented |
| `ActionCache` | REAPI | Implemented |
| `ContentAddressableStorage` | REAPI | Implemented |
| `Capabilities` | REAPI | Implemented |
| `google.bytestream.ByteStream` | Google API | Implemented |
| `google.longrunning.Operations` | Google API | Partially implemented, see below |
| Remote Asset `Fetch` | Remote Asset v1 | Implemented |
| Remote Asset `Push` | Remote Asset v1 | Implemented |
| `PublishBuildEvent` | Build Event Protocol | Implemented, marked experimental |
| `WorkerApi` | **NativeLink** | Implemented |

Each is registered only if the configuration asks for it. Registration order
at startup is fixed: action cache, CAS, execution, `Operations`, fetch, push,
ByteStream, capabilities, worker API, BEP.

<Callout type="info" title="Two endpoints are not gRPC">
  The health endpoint (`/status` by default) and the admin endpoint (`/admin`
  by default; both paths are configurable) are plain HTTP handlers on the same
  listener. They will not appear in gRPC reflection, and a health check does
  not need a gRPC client.
</Callout>

## `WorkerApi`: the only NativeLink-defined service [#worker-api]

Defined in
<SourceLink file="nativelink-proto/com/github/trace_machina/nativelink/remote_execution/worker_api.proto" />.
It has **exactly one RPC**:

```proto
rpc ConnectWorker(stream UpdateForScheduler) returns (stream UpdateForWorker);
```

One bidirectional stream carries a worker's entire relationship with its
scheduler: registration, heartbeats, action assignment, progress and
completion. No polling RPC and no second connection exist. See
[worker execution](/explanations/worker-execution#the-connection-is-one-stream).

`events.proto` in the same tree declares message types only, **no services**.

## Partially implemented RPCs [#partially-implemented-rpcs]

Three `Operations` RPCs are registered and return `unimplemented`:

| RPC | Status |
| --- | --- |
| `GetOperation` | Implemented |
| `WaitOperation` | Implemented |
| `ListOperations` | `unimplemented` |
| `DeleteOperation` | `unimplemented` |
| `CancelOperation` | `unimplemented` |

A client that relies on cancellation to release cluster resources will not
get it here. Actions are instead reaped by timeout; see
[`client_action_timeout_s`](/explanations/scheduler-internals#two-timeouts-with-very-different-defaults),
which kills an action nobody is listening to.

<Callout type="info" title="SplitBlob and SpliceBlob are upstream RPCs">
  Both appear in NativeLink's vendored CAS proto and are sometimes mistaken
  for NativeLink inventions. They are part of the upstream REAPI definition
  (content-defined chunking), vendored along with everything else. NativeLink
  serves them only for CAS instances that configure `experimental_chunking`
  (or proxy to a `grpc` store that does); elsewhere the capabilities response
  reports them unsupported and the RPCs return `Unimplemented`.
  <SourceLink file="nativelink-service/src/cas_server.rs" symbol="split_blob" />
</Callout>

## What `Capabilities` advertises [#what-capabilities-advertises]

| Field | Value |
| --- | --- |
| Supported API versions | 2.0.0 to 2.3.0 |
| Digest functions | `SHA256`, `BLAKE3` |
| `max_batch_total_size_bytes` | 65536 (64 KiB) |
| `max_cas_blob_size_bytes` | 0 (no limit advertised) |
| `symlink_absolute_path_strategy` | `DISALLOWED` |
| `update_enabled` (action cache) | `true` |
| `supported_compressors` | `ZSTD` when the instance's `capabilities` entry sets `remote_cache_compression: true`; empty otherwise |
| `split_blob_support`, `splice_blob_support`, `fast_cdc_2020_params` | set when the instance's `cas` entry configures `experimental_chunking`; otherwise `false`/unset |
| `execution_capabilities` | present only when the instance's `capabilities` entry has `remote_execution`; then `exec_enabled` is `true` and the priority range is 0 to `i32::MAX` |

<SourceLink file="nativelink-service/src/capabilities_server.rs" symbol="get_capabilities" />

Two of these need a caveat.

**`exec_enabled` is hardcoded to `true`**, with a TODO in the source
acknowledging it. Whenever a `capabilities` entry names a `remote_execution`
scheduler, execution is advertised as enabled; there is no way to advertise
`exec_enabled: false`. A cache-only `capabilities` entry (no
`remote_execution`) omits `execution_capabilities` altogether, which is the
correct signal for a cache-only deployment.

**The 64 KiB batch limit is advisory and small** relative to what some
clients assume. The CAS handlers do not enforce it themselves; the hard stop
is the listener's `max_decoding_message_size` (4 MiB unless configured), and
a batch request above that is rejected at the gRPC layer. Clients that honour
the advertised limit send anything larger through ByteStream.

## Wire-level details worth knowing [#wire-level-details-worth-knowing]

**ByteStream resource names are parsed right to left.** A resource name is
`{instance_name}/blobs/{hash}/{size}`, and the instance name may itself
contain slashes, so left-to-right parsing cannot find the boundary, while
right-to-left can, because the trailing components are fixed. See
<SourceLink file="nativelink-util/src/resource_info.rs" symbol="ResourceInfo" />.

**`GetTree` page tokens are `"{hash}-{size}"`**, constructed and parsed by
splitting on `-`. They are neither opaque nor signed. The protocol asks
clients to treat them as opaque, and clients that do will keep working if the
format changes.

**The `Execute` pre-flight input check is shallow and racy by design.** It
does not walk the full input tree, and a blob can be evicted between the
check and the worker's fetch. It exists to turn the common "inputs were never
uploaded" mistake into a fast error, not to provide a guarantee. See
<SourceLink file="nativelink-service/src/execution_server.rs" symbol="ExecutionServer" />.

**`ActionStage::CacheCheck` is never emitted.** The cache lookup happens in a
decorator in front of the scheduler, so the state is skipped entirely. A
client waiting to observe it will wait forever.

**Final action failure arrives as success.** An action that exhausts its
retries returns `ActionStage::Completed` with `ActionResult.error` set: a
successful RPC describing a failed action. This is what the REAPI specifies.
Clients that only inspect the gRPC status will misread it.

**`server_logs` is never populated.** The field exists; the worker does not
fill it.

## Related [#related]

- [Architecture deep dive](/explanations/architecture-deep-dive): where each
  service sits in the process
- [Scheduler internals](/explanations/scheduler-internals): what happens
  between `Execute` and a worker
- [Worker execution](/explanations/worker-execution): the `WorkerApi` stream
  from the worker's side
- [Configuration reference](/reference/nativelink-config): which services a
  given config enables

# https://docs.nativelink.com/reference/metrics

**Metrics reference**: Every OpenTelemetry instrument NativeLink declares, with its type, unit, attributes, Prometheus series, and whether the binary actually emits it. Autogenerated from the Rust source and its call sites.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/metrics._

# https://docs.nativelink.com/reference/cli-and-env

**CLI and environment**: Every command-line argument the nativelink binary accepts and every environment variable it reads: the telemetry variables, the ones the configuration expands, and the ones it passes through to actions.

**Who this is for:** you are writing a systemd unit, a Dockerfile `CMD`, or a
Kubernetes `env:` block and need to know what the process actually reads.
**What you'll have at the end:** the complete argument and variable surface,
with the sharp edges marked.

NativeLink's command line is deliberately tiny. Almost every decision lives in
the JSON5 configuration file, not in flags, which means the interesting surface
here is the *environment*, split across three unrelated jobs: configuring
telemetry, filling in holes in the configuration file, and reaching the
processes that actions run in.

## The command line [#the-command-line]

<SourceLink file="src/bin/nativelink.rs" symbol="Args" />

```sh
nativelink <config_file>
```

That is the whole thing. `config_file` is a required positional argument, the
path to a JSON5 configuration file, and there are no other arguments. The
binary is a [clap](https://docs.rs/clap) parser with `version`, `about` and
`author` derived, so `--help` and `--version` work and nothing else does.

| Argument | Required | Meaning |
| --- | --- | --- |
| `<config_file>` | Yes | Path to the JSON5 configuration file. Parsed with `serde_json5`; the config structs are `deny_unknown_fields`, so an unrecognised key is a startup failure, not a warning. |
| `--help` | No | Print usage and exit. |
| `--version` | No | Print the crate version and exit. |

No `--port`, no `--log-level`, no `--config-from-env` exist. If you are
looking for a knob, it is in the configuration file; see the
[configuration reference](/reference/nativelink-config).

## Telemetry variables [#telemetry-variables]

These are read once, at startup, by `nativelink-util`'s tracing initialiser.
They are the only variables that change the binary's own behaviour.

<SourceLink file="nativelink-util/src/telemetry.rs" symbol="init_tracing" />

| Variable | Default | Effect |
| --- | --- | --- |
| `NL_LOG` | `pretty` | Format of the stdout log layer. `compact` and `json` are recognised; **any other value silently falls back to `pretty`**, including a typo like `Json`. |
| `RUST_LOG` | `info` | Log filter, parsed by `tracing_subscriber`'s `EnvFilter` in lossy mode; an unparseable directive is dropped rather than fatal. Applies to the stdout layer *and* to the OTLP log, trace and metric layers. `hyper`, `tonic`, `h2`, `reqwest` and `tower` are forced off afterwards and cannot be re-enabled from here. |
| `NL_OTEL_ENDPOINT` | not set | When set, all three OTLP exporters share one client-side load-balanced gRPC channel pointed at this endpoint. When unset, each exporter is built with the SDK's own tonic defaults instead. |
| `OTEL_EXPORTER_OTLP_ENDPOINT` | `http://localhost:4317` | Standard OpenTelemetry SDK variable. Applies only when `NL_OTEL_ENDPOINT` is unset; an explicit channel wins over the SDK's endpoint resolution. The signal-specific `OTEL_EXPORTER_OTLP_{LOGS,TRACES,METRICS}_ENDPOINT` variables behave the same way. |
| `OTEL_METRIC_EXPORT_INTERVAL` | `60000` (ms) | How often the periodic metric reader flushes to the collector. NativeLink sets no interval of its own, so the SDK default of one minute applies unless you override it here. This is the first hop in the [reaction-time budget](/operate/autoscaling-reference). |

<Callout type="warn" title="NL_OTEL_ENDPOINT requires a scheme and an explicit port">

  The endpoint is parsed as a URL and then the host and port are pulled out
  individually, and each step `unwrap`s. A value with no scheme, or with no
  explicit port, yields `None` for the port and **panics the process during
  startup** rather than falling back to a default.

  `http://otel-collector:4317` is correct. `otel-collector:4317` and
  `http://otel-collector` both crash the binary before it binds a socket.

</Callout>

The service name is fixed. `Resource::builder().with_service_name("nativelink")`
is called unconditionally, so `OTEL_SERVICE_NAME` does not change what appears
on your spans. `service.instance.id` is a fresh UUIDv4 generated per process,
which means it changes on every restart: useful for telling replicas apart,
useless as a stable series identity.

## Variables the configuration file expands [#variables-the-configuration-file-expands]

Many configuration fields are deserialized through a shellexpand wrapper rather
than directly, so their string values undergo environment substitution before
the value is used.

<SourceLink file="nativelink-config/src/serde_utils.rs" symbol="convert_string_with_shellexpand" />

Two forms are available:

| Form | Behaviour |
| --- | --- |
| `${VAR}` or `$VAR` | Substitutes the variable. An unset variable is an error, and the error surfaces as a config parse failure. |
| `${VAR:-default}` | Substitutes the variable, or the literal default when it is unset. A variable that is set but empty expands to the empty string, not to the default. |

This is what makes one configuration file work across environments:

```json5
{
  stores: [
    {
      name: "CAS_MAIN_STORE",
      memory: {
        eviction_policy: {
          max_bytes: "${NATIVELINK_CAS_MEMORY_CONTENT_LIMIT:-100mb}",
        },
      },
    },
  ],
}
```

Not every field is expanded; the wrapper is applied per field, so whether a
given key supports substitution depends on how its struct member is annotated.
Numeric, boolean, duration and data-size fields have their own expanding
deserializers, which is why `max_bytes` above accepts the *string* `"100mb"`
rather than a number. When in doubt, the
[configuration reference](/reference/nativelink-config) records the type each
field actually deserializes from.

<Callout type="info" title="Expansion happens at parse time, not at use time">
  The substitution runs once, while the configuration file is being read at
  startup. Changing a variable in the environment of a running process has no
  effect until it restarts.
</Callout>

## Variables the worker passes to actions [#variables-the-worker-passes-to-actions]

A worker does not forward its own environment to the commands it runs: the
child is spawned with `env_clear()`. An action sees exactly two things: the
`environment_variables` its own `Command` message carries, and whatever the
worker's `additional_environment` map (on the `local` worker configuration)
names explicitly.

<SourceLink file="nativelink-config/src/cas_server.rs" symbol="EnvironmentSource" />

| Source | What the action receives |
| --- | --- |
| `Value` | A literal string, itself shellexpanded at config parse time, so `Value` is one way to deliberately pass a worker-side variable through. |
| `FromEnvironment` | The worker process's value for the variable of the same name (empty if unset). This is the only runtime pass-through, and it is opt-in per variable. |
| `Property` | The value of the named platform property from the action, or the empty string if the action did not set it. |
| `TimeoutMillis` | The timeout the worker will enforce, in milliseconds: the client-requested timeout, or `max_action_timeout` when the client asked for none. |
| `SideChannelFile` | A path (a fresh UUID under the action directory) the action may write JSON to in order to influence how its result is interpreted; `{"failure": "timeout"}` marks the action as timed out. |
| `ActionDirectory` | A scratch directory that is purged after the action completes. The usual pattern is an `entrypoint` script that creates a `tmp` subdirectory under it and exports `TMPDIR`. |

<Callout type="warn" title="The precondition script and property-query commands inherit almost nothing">
  `experimental_precondition_script` and any `query_cmd` platform property are
  run with `env_clear()` and split with `shlex` rather than through a shell.
  The only variables they receive are the `Value` and `FromEnvironment`
  entries of `additional_environment` (the precondition script also gets
  `Property` entries). Unless you pass `PATH` that way, use an absolute path,
  and shell syntax (pipes, globs, `&&`) is not interpreted.
</Callout>

## What NativeLink does not read [#what-nativelink-does-not-read]

Stated explicitly, because each of these is a reasonable guess that is wrong:

- **No environment-based configuration.** No `NL_CAS_PORT` or
  equivalent. Ports, stores, schedulers and services come from the
  configuration file only. The escape hatch is `${VAR}` substitution *inside*
  that file.
- **No `NATIVELINK_CONFIG` variable.** The configuration path is the positional
  argument and nothing else.
- **No Prometheus scrape configuration.** The binary exports OTLP and only
  OTLP; the `namespace`, `const_labels` and scrape endpoint you may be looking
  for belong to the OpenTelemetry collector's configuration, not NativeLink's.
  See the [metrics reference](/reference/metrics).
- **No `OTEL_SERVICE_NAME` honouring**, as above; the service name is a
  compile-time constant.

## Reading the source [#reading-the-source]

If anything here disagrees with the binary, the source wins:

- <SourceLink file="src/bin/nativelink.rs" symbol="Args" />: the argument parser
- <SourceLink file="nativelink-util/src/telemetry.rs" />: every telemetry variable
- <SourceLink file="nativelink-config/src/serde_utils.rs" />: the shellexpanding deserializers
- <SourceLink file="nativelink-config/src/cas_server.rs" symbol="EnvironmentSource" />: action environment sources

## Where to go next [#where-to-go-next]

<NextStep href="/reference/nativelink-config" title="Configuration reference">
  Everything the command line does not do, which is nearly everything,
  generated from the config crate itself.
</NextStep>

<NextStep href="/operate/observability" title="Wire up telemetry" kind="aside">
  The variables above in the context of a working collector pipeline.
</NextStep>

<NextStep href="/reference/oss-and-enterprise" title="Open source and Enterprise" kind="aside">
  Why there is no licence flag here, and what is licensed differently anyway.
</NextStep>

# https://docs.nativelink.com/reference/oss-and-enterprise

**Open source and Enterprise**: What the open-source distribution actually contains, which two modules are licensed differently, why nothing in the binary enforces that, and what the paid tiers are for.

**Who this is for:** anyone who has to answer "are we allowed to run this?",
and anyone deciding between self-hosting and NativeLink Enterprise. **What you'll
have at the end:** an accurate picture of what is licensed how, what activates
it, and what the open-source build does and does not restrict. **Time:** ten
minutes.

<Callout type="warn" title="This page describes the repository, not your legal position">
  Everything below is checkable against the source tree at the commit you are
  reading: file paths, licence headers, and the code that reaches them. None
  of it is legal advice, and the operative text is the
  [licence page](https://nativelink.com/license) and the `LICENSE` file in the
  repository root, not this page. If your intended use is near a boundary,
  read those and ask.
</Callout>

## The short version [#the-short-version]

The whole of NativeLink is in the public repository. No separate
"enterprise build", no stripped-down community edition, and no compile-time
feature flag that divides them. What differs between the free and paid product
is the *licence you hold*, not the *code you run*.

Two modules inside that one codebase carry a different licence from the rest.
Both are compiled into every binary, both are active by default, and nothing
checks whether you are entitled to them.

| Question | Answer |
| --- | --- |
| What licence covers most of the code? | `FSL-1.1-Apache-2.0`, the Functional Source License, converting to Apache 2.0 two years after each version ships |
| What covers the rest? | Business Source License 1.1, on two modules: metrics and persistent workers |
| Does the binary enforce the split? | No. No licence key, no check, no gate, no feature flag |
| Is the free build limited in size, workers, or team? | No |
| Does it contact Trace Machina? | No |

## Two licences, one repository [#two-licences-one-repository]

The repository root carries `LICENSE`, the Functional Source License 1.1 with
an Apache 2.0 future grant. Its restriction is narrower than most people
expect on first reading: it permits everything except a **Competing Use**,
which the licence defines as making the software available to others in a
commercial product or service that substitutes for NativeLink, substitutes for
something Trace Machina already offers using it, or offers substantially
similar functionality.

Named as explicitly permitted, in the licence text itself: internal use and
access, non-commercial education, non-commercial research, and use in
connection with professional services you provide to someone else who is
themselves complying.

The practical read for most readers is that running NativeLink as your own
build farm (for a team, for a company, in production, commercially) is
internal use, and internal use is permitted. Reselling a managed NativeLink
service is what the licence is written to prevent.

<Callout type="info" title="The Apache conversion clock is per version, not a single date">
  The future grant is effective "on the second anniversary of the date we make
  the Software available." That clock runs per released version, not once for
  the project. Version 1.6.5 converts to Apache 2.0 two years after 1.6.5 was
  published; a version released next month converts a month later. No
  future date exists on which the whole repository becomes Apache; there is a rolling
  two-year trailing edge.
</Callout>

## The Business Source Licensed modules [#the-business-source-licensed-modules]

Five files in the tree carry a Business Source License 1.1 header instead of
the repository licence. They form two modules:

| Module | Files | What it is |
| --- | --- | --- |
| Metrics | `nativelink-util/src/metrics.rs` | Every OpenTelemetry instrument NativeLink declares: the cache counters and histograms, the execution stage and result metrics, and the attribute sets attached to them |
| Persistent workers | `nativelink-worker/src/persistent_worker/{mod,pool,live_worker,protocol}.rs` | Bazel's persistent-worker protocol: keeping a compiler process warm across actions instead of starting one per action |

Their headers say the same thing, verbatim:

```text
// Use of this module requires an enterprise license agreement, which can be
// attained by emailing contact@nativelink.com or signing up for Nativelink
// Cloud at app.nativelink.com.
```

(The header still names NativeLink Cloud, which is no longer offered; the
managed option today is NativeLink Enterprise, and the email address is the
route to either agreement.)

The wording matters. It is a licence obligation attached to *use of the
module*, which is a different question from whether you may run the binary at
all. You may run the binary; whether you may exercise those two modules in a
shared, production or commercial setting depends on a separate agreement.

## Nothing in the binary enforces any of this [#nothing-in-the-binary-enforces-any-of-this]

<Callout type="warn" title="No licence check exists anywhere in the codebase">
  No licence key field exists in the configuration schema. No startup
  validation reads one. No runtime gate wraps either module. Searching the
  whole tree for enforcement turns up exactly five hits: the five header
  comments listed above, and nothing else. Both modules are compiled
  unconditionally and both run by default. **You can be out of compliance
  without ever seeing an error, a warning, or a log line.** The boundary here
  is legal, not technical, and the software will not help you find it.
</Callout>

This is not an oversight to route around. It is the reason this page exists:
if you rely on the binary to tell you which features you are entitled to, it
will tell you that you are entitled to everything.

### Metrics are always on [#metrics-are-always-on]

No configuration turns instrumentation off. The meter provider
is constructed unconditionally during startup and installed globally, and the
scheduler and store layers call into the licensed instruments on their normal
paths: `simple_scheduler_state_manager.rs` records completed executions,
`memory_awaited_action_db.rs` records stage transitions, and
`cache_metrics_store.rs` records every cache operation.

`NL_OTEL_ENDPOINT` is often mistaken for the switch. It is not. Setting it
chooses a load-balanced gRPC channel for the OTLP exporters; leaving it unset
means the exporter builds its own channel to the OTLP default instead. Either
way the instruments are declared and recorded. What changes is where the
export attempt goes, not whether measurement happens.

The consequence for licensing is that "we don't use metrics" is not a
configuration you can be in. What you can choose is whether you *consume*
them: whether a collector is running, whether dashboards and alerts are built
on them, whether they are part of how your team operates the cluster. That
consumption is what the licence is about. See
[Observability](/operate/observability) for the pipeline itself, and note that
page's own [licence callout](/operate/observability#do-metrics-require-a-commercial-licence).

### Persistent workers activate from the action, not from your config [#persistent-workers-activate-from-the-action-not-from-your-config]

No `persistent_worker` key exists anywhere in the configuration schema;
searching `nativelink-config` for it returns nothing at all. The module is
reached entirely from the action's own platform properties:

| Property | Values | Effect |
| --- | --- | --- |
| `supports-workers` | `1` activates; anything else, or absent, does not | The only trigger. The worker checks this before every action |
| `requires-worker-protocol` | `proto` (default) or `json` | Wire format. Any other value (or a persistent worker that cannot be started) makes the worker log and fall back to ordinary one-shot execution of the action |

Those properties come from your build tool. Bazel sets them on rules that
declare worker support: Java and Scala compilation, TypeScript, and anything
using `ctx.actions.run` with `execution_requirements` asking for workers.
A Bazel build that already uses persistent workers locally will exercise
the licensed module remotely the first time it runs an affected action, with
no NativeLink-side opt-in and nothing in your configuration mentioning it.

Because there is no config surface, the pool's behaviour is fixed at
compile-time defaults:

| Behaviour | Value | Tunable? |
| --- | --- | --- |
| Live workers per key | 4 (matching Bazel's `worker_max_instances` default) | No |
| Idle shutdown | 5 minutes without a request | No |
| Requests before recycling a worker | 200 | No |

Two actions share a worker process when their *worker key* matches: same
resolved executable, same startup-flag prefix (the arguments before the first
`@response-file`), and same wire format. The full picture, including what
isolation you give up, is on
[Persistent workers](/remote-execution/persistent-workers).

## What the open-source build does not limit [#what-the-open-source-build-does-not-limit]

Nothing in the code artificially constrains scale. Specifically, there is no
cap on cache size, stored object count, action throughput, connected worker
count, or number of people using the cluster; no trial period, expiry date, or
time-limited mode; and no registration, activation, or account required to
start the binary.

Nor does the binary report anything home. The only outbound HTTP clients in
the tree belong to storage backends you configure yourself (the S3-compatible,
GCS and Azure object stores), and the only other egress is to the other
backends you configure (Redis, MongoDB, `grpc` stores), gRPC to the peers in
your own cluster, and the OTLP exporters. Those exporters send to whatever endpoint you point
them at, which is your collector; if you point them somewhere else, that is
your configuration doing it, not a default.

The limits you will actually hit are the operational ones (file descriptors,
disk, memory, scheduler backpressure), and those are documented in
[Troubleshooting](/operate/troubleshooting#limits-that-block-instead-of-erroring)
and [Runbooks](/operate/runbooks).

## What Enterprise adds [#what-cloud-and-enterprise-add]

The honest answer, from inside this repository, is that the paid tier is not
a different codebase, so this documentation cannot enumerate its feature list
by reading the source. What can be said accurately:

It carries the licence for the two Business Source modules in shared,
production and commercial settings, which for most teams running metrics-driven
operations is the substantive difference, since as shown above there is no way
to run without touching that module.

It removes the operational work. The whole of the
[Operate](/operate) section (the deployment shape, the metrics
pipeline, the eviction sizing, the incident runbooks) describes work that a
managed tier does instead of you. That is the trade
[Shared cache](/getting-started/shared-cache) frames as an operational-cost
question (control-plane compute, CAS storage, worker compute, and the engineer
hours to run it) rather than a licence question.

For what Enterprise includes, its pricing, and what support attaches to it, the
sources of truth are [nativelink.com/pricing](https://nativelink.com/pricing)
and the [licence page](https://nativelink.com/license). Meaningful contributors
to the project may be eligible for waivers on the Business Source modules;
ask the maintainers before relying on one.

## Telling what you are running [#telling-what-you-are-running]

The binary is a [clap](https://docs.rs/clap) parser with `version` derived from
the crate, so:

```sh
nativelink --version
```

prints the crate version (`1.6.5` at the time of writing) and exits. The
container image is tagged with the same version prefixed by `v`
(`ghcr.io/tracemachina/nativelink:v1.6.5`); pin that tag in production rather
than a floating one. See
[CLI and environment](/reference/cli-and-env) for the complete flag surface,
which is `--help` and `--version` and nothing else, and
[Upgrade versions](/how-to/upgrade-versions) for moving between them.

No command prints licence status, because there is no licence
status to print.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Can I run NativeLink commercially for free?" id="can-i-run-nativelink-commercially-for-free">
    Running it as your own build infrastructure is internal use, which the
    Functional Source License permits regardless of whether your company is
    commercial. The separate question is the two Business Source modules:
    metrics in a shared or production setting needs a licence, and metrics
    cannot be switched off. Read the
    [licence page](https://nativelink.com/license) against your actual
    situation rather than inferring from this summary.
  </Accordion>
  <Accordion title="Can I fork it and remove the Business Source modules?" id="can-i-fork-it-and-remove-the-business-source-modules">
    The Functional Source License permits modification and redistribution for
    non-competing purposes, so a fork for your own use is within its terms.
    Removing the metrics module is more invasive than it looks: the scheduler
    and store layers call into it on their normal paths, so you are removing
    instrumentation from the code that emits it, not deleting a plugin. The
    Redistribution clause also requires you to carry the terms and keep the
    copyright notices.
  </Accordion>
  <Accordion title="Does the free build phone home or collect telemetry?" id="does-the-free-build-phone-home-or-collect-telemetry">
    No. The binary has no analytics client, no usage reporting, and no request to any
    Trace Machina endpoint anywhere in the binary. The OTLP exporters send to
    the endpoint you configure, which in every documented setup is a collector
    you run.
  </Accordion>
  <Accordion title="When does NativeLink become Apache 2.0?" id="when-does-nativelink-become-apache-2-0">
    Each released version does, two years after that version was made
    available, a rolling edge rather than a single cutover date. The Business
    Source modules are governed by their own headers and are not covered by the
    repository licence's future grant.
  </Accordion>
  <Accordion title="Is there a feature I can only get by paying?" id="is-there-a-feature-i-can-only-get-by-paying">
    Not in the binary. Every code path in the open-source distribution runs in
    the open-source distribution; that is the point of the section above about
    enforcement. What you buy is the entitlement to two of those paths in
    settings the Business Source License restricts, plus the managed service
    and support that sit outside the repository entirely.
  </Accordion>
</Accordions>

<SourceLink file="LICENSE" symbol="Grant of Future License" />
<SourceLink file="nativelink-util/src/metrics.rs" symbol="CACHE_METRICS" />
<SourceLink file="nativelink-worker/src/running_actions_manager.rs" symbol="action_supports_persistent_workers" />

<NextStep href="/getting-started/shared-cache" title="Shared cache">
  The self-host-or-Enterprise decision worked through as an operational question
  rather than a licensing one.
</NextStep>

<NextStep href="/remote-execution/persistent-workers" title="Persistent workers" kind="aside">
  What the licensed module actually does, and the isolation it trades away.
</NextStep>

<NextStep href="/operate/observability" title="Observability" kind="aside">
  The metrics pipeline the other licensed module feeds.
</NextStep>

# https://docs.nativelink.com/reference/glossary

**Glossary**: The vocabulary you need to read the rest of the docs.

**Who this is for:** anyone reading the rest of these docs who hit a word that
was used as if it needed no explanation. **What you'll have at the end:** the one
sentence you needed, and a link to the page that goes deeper. **Time:** use as
a lookup.

Terms are alphabetical. Every heading has a stable anchor, so a link to a
definition keeps resolving even if the wording around it changes.

## Action [#action]

A single unit of build work. Encodes a command line, input file
digests, the platform requirements, and the expected outputs. Hashing
an action produces a stable identifier; identical actions have
identical hashes.

## Action Cache (AC) [#action-cache-ac]

A keyed store: `hash(Action) → ActionResult`. Cache hits skip the
work entirely.

## ActionResult [#actionresult]

What an action produces. Output file digests, exit code, captured
stdout/stderr, timing metadata.

## Awaited action [#awaited-action]

The scheduler's record of one action between `Execute` and completion: its
current [execution stage](#execution-stage), its priority, and every client
waiting on it. Two identical actions submitted at the same time join the same
awaited action instead of running twice. By default this record lives in the
scheduler process's memory
(<SourceLink file="nativelink-scheduler/src/memory_awaited_action_db.rs" />),
which is why two scheduler replicas behind a load balancer do not share
in-flight work. See [scheduler internals](/explanations/scheduler-internals).

## ByteStream API [#bytestream-api]

The streaming transport for blobs too large to move in a single
`BatchUpdateBlobs` or `BatchReadBlobs` call, with the [digest](#digest)
encoded in the resource name. Where the batch calls stop and this one starts
is a server limit the [Capabilities API](#capabilities-api) advertises.

## Capabilities API [#capabilities-api]

The RE-API service a client calls first, to learn which
[digest functions](#digest-function) the server accepts and what its size
limits are rather than assuming. See
[protocol and API surface](/reference/protocol-api).

## CAS: Content-Addressable Storage [#cas--content-addressable-storage]

Stores every blob (source file, intermediate output, final binary)
under the digest of its contents. The same byte sequence stored from anywhere
collapses to one entry.

## Completeness-checking store [#completeness-checking-store]

An [Action Cache](#action-cache-ac) wrapper that checks every output
[digest](#digest) still exists in the CAS before serving a hit. Without it a
CAS eviction turns old cache entries into hits that reference blobs nobody can
download, and the build fails while fetching outputs instead of quietly
missing the cache. Valid only on AC stores.

## Dedup store [#dedup-store]

A store that splits blobs into content-defined chunks with FastCDC
(<SourceLink file="nativelink-util/src/fastcdc.rs" />) and stores each chunk
once, so two versions of a large file that differ in the middle share
everything else. The cost is an index lookup per read. See
[the store model](/explanations/store-model).

## Digest [#digest]

A content hash plus a size. Used everywhere instead of file paths.

## Digest function [#digest-function]

The hash a client and server agree on for [digests](#digest): SHA-256 or
BLAKE3, negotiated per request rather than fixed at build time
(<SourceLink file="nativelink-util/src/digest_hasher.rs" symbol="DigestHasherFunc" />).
Two clients using different functions produce different keys for the same
bytes, so they do not share cache entries.

## Eviction policy [#eviction-policy]

The limits a store evicts against: `max_bytes`, `max_seconds`, `max_count`,
and the `evict_bytes` low watermark that stops eviction thrashing right at the
limit. Each defaults to `0`, which means *never evict on that dimension*: a
store configured without a policy grows until the disk does.

## Execution stage [#execution-stage]

Where an action is in its life: `CacheCheck`, `Queued`, `Executing`, then
`Completed` or `CompletedFromCache`
(<SourceLink file="nativelink-util/src/action_messages.rs" symbol="ActionStage" />).
An action stuck in `Queued` has no free [worker](#worker) whose
[platform properties](#platform-properties) match what it asked for.
`CacheCheck` is declared but the scheduler never reports it, because the
cache lookup happens in a wrapper in front of the scheduler.

## Existence cache store [#existence-cache-store]

A store that remembers which digests it has already confirmed exist, so
repeated [`FindMissingBlobs`](#findmissingblobs) calls stop re-querying the
backend. It answers the existence question only; reads pass through.

## `fast_slow` store [#fast-slow-store]

A two-tier store. Reads try the fast backend, fall back to the slow one, and
copy what they find into the fast tier on the way back; writes mirror to both.
Its one sharp edge: it never checks the slow tier for something the fast tier
already has, so a blob that reached the fast tier without reaching the slow one
stays invisible to everything reading the slow tier directly.

## FindMissingBlobs [#findmissingblobs]

The CAS call that makes uploads incremental: the client offers a list of
[digests](#digest), the server returns the ones it does not have, and the
client uploads only those. On a warm cache this and the [AC](#action-cache-ac)
lookup are most of what a client does.

## Hermetic build [#hermetic-build]

A build whose outputs depend only on its declared inputs. Same
inputs → same outputs, on any machine, any time. What NativeLink does
and does not guarantee here is in
[correctness and hermeticity](/explanations/correctness-hermeticity).

## Input root [#input-root]

The complete file tree an action is allowed to see, encoded as a Merkle tree
of `Directory` messages that reference their files and each other by
[digest](#digest). Its root digest is part of the [action](#action) hash, so
changing one input file changes the cache key.

## Instance name [#instance-name]

A namespace inside a NativeLink cluster. Each service entry maps an
instance name to a store or scheduler, so one cluster can serve multiple
isolated environments, and the scheduler keys in-flight actions by instance
name as well as digest. Two instance names that point at the same store share
its contents.
<SourceLink file="nativelink-config/src/cas_server.rs" symbol="InstanceName" />

## Isolation [#isolation]

Keeping an action from seeing state outside its declared inputs.
On Linux, workers can isolate actions with kernel namespaces via
the `use_namespaces` and `use_mount_namespace` worker options,
which also reap zombie processes and improve hermeticity.
NativeLink doesn't currently integrate external sandboxing tools
like `bwrap`, `landlock`, or `sandbox-exec`.
<SourceLink file="nativelink-config/src/cas_server.rs" symbol="use_mount_namespace" />

## JSON5 [#json5]

The dialect NativeLink's configuration file is written in: JSON plus
comments, trailing commas, and unquoted keys. Every example in
[the configuration reference](/reference/nativelink-config) is JSON5.

## LRE: Local Remote Execution [#lre--local-remote-execution]

Building your toolchain with Nix so that your local build and your
remote workers invoke the *same* toolchain binaries, at the same
`/nix/store` paths, which makes their action digests match and lets
both share one cache. No worker runs on your laptop. See
[LRE](/explanations/lre).

## Operation [#operation]

The handle a client holds on a running action, from
`google.longrunning.Operation`. `Execute` returns one immediately and the
client watches it for [stage](#execution-stage) changes; `WaitExecution`
re-attaches to it after a disconnect, which is what stops a dropped connection
from re-running the work.

## Origin events [#origin-events]

An experimental stream of the requests and responses a server handled,
published to a store for an external consumer to read
(<SourceLink file="nativelink-proto/com/github/trace_machina/nativelink/remote_execution/events.proto" />).
Off unless `experimental_origin_events` is configured.

## Platform properties [#platform-properties]

Key/value tags attached to an action ("this action needs Linux,
x86_64, a GPU") and to a worker ("this worker has Linux, x86_64, a
GPU"). The scheduler matches them.
<SourceLink file="nativelink-util/src/platform_properties.rs" symbol="is_satisfied_by" />

## Priority [#priority]

An integer on `ExecuteRequest`'s `execution_policy`; NativeLink dequeues
the highest value first when workers are scarce (the RE-API leaves the
direction to the server, and its suggested default is the reverse). It orders
the queue and nothing more; an action already executing is not interrupted for
a higher-priority one that arrives after it.
<SourceLink file="nativelink-scheduler/src/awaited_action_db/awaited_action.rs" symbol="AwaitedActionSortKey" />

## Remote Execution API (RE-API) [#remote-execution-api-re-api]

The standard gRPC protocol every supported build system speaks.
[Spec](https://github.com/bazelbuild/remote-apis).

## Scheduler [#scheduler]

The dispatcher. Receives `Execute` calls, picks workers, tracks
in-flight actions.

## Shard store [#shard-store]

A store that routes each key to one of several backends by digest hash, with
a `weight` per backend. It spreads capacity; it is not replication, so losing
one backend loses that backend's share of the data rather than none of it.

## Store [#store]

NativeLink's unit of storage configuration: a named thing that gets and puts
bytes under a key. Stores compose, because a store's backend is another store,
so the CAS a server serves is usually a stack of wrappers over one real
backend. See [the store model](/explanations/store-model).

## Toolchain [#toolchain]

The bundle of binaries an action needs to run: compiler, linker,
standard library, etc. See
[how toolchains are provided](/remote-execution/toolchains-and-hermeticity#faq).

## Verify store [#verify-store]

A wrapper that, depending on `verify_size` and `verify_hash`, checks a
blob's size and re-hashes it on write, failing the upload when the bytes do
not match the [digest](#digest) they were offered under. It turns silent corruption into a write error. Both checks
belong on CAS stores and neither belongs on an [AC](#action-cache-ac) store.

## Worker [#worker]

The process that runs an action. Fetches inputs from CAS, runs the
command, uploads outputs to CAS.

## Worker API [#worker-api]

NativeLink's own gRPC service, not part of the RE-API, over which a worker
registers with a scheduler and receives work
(<SourceLink file="nativelink-proto/com/github/trace_machina/nativelink/remote_execution/worker_api.proto" />).
One bidirectional `ConnectWorker` stream carries the whole conversation, which
is what keeps a worker talking to the same scheduler instance even behind a
load balancer.

# https://docs.nativelink.com/reference/changelog

**Changelog**: Notable changes per release. Latest first.

_Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/changelog._

# https://docs.nativelink.com/contribute/guidelines

**Contribution guidelines**: What NativeLink accepts, the git setup it requires, and the commit and review conventions a PR is held to.

**Who this is for:** anyone about to open their first pull request.
**What you'll have at the end:** the git configuration the project
requires, the commit conventions, and an accurate picture of how review
works. **Time:** thirty minutes, most of it one-time setup.

NativeLink accepts contributions from anyone. The process is more specific
than most projects' (signed commits, a fork, one commit per pull request,
review in Reviewable rather than in GitHub's own UI), and the specificity is
deliberate: it keeps `main` a linear history of signed, reviewable commits.
<SourceLink file="CONTRIBUTING.md" symbol="CONTRIBUTING.md" />
is the authoritative version; this page is the orientation around it.

## What gets accepted [#what-gets-accepted]

| Welcome | With a caveat |
| --- | --- |
| Bug fixes, with a regression test | |
| Performance improvements | Bring reproducible numbers |
| New stores, schedulers, worker types | Open an issue and agree the approach first |
| Documentation fixes, including this page | Smaller is better |
| Tooling that makes development or operations easier | |

Three things reliably do not land: breaking changes to the Remote Execution
API surface, which is an upstream spec the project tracks rather than owns;
vendor-specific code paths only one deployment can use; and refactoring
proposed on its own merits rather than as part of a change that needed it.

If you want to work on an existing issue, **claim it by commenting first.**
That is the only mechanism preventing two people from writing the same patch.

## Git setup, once [#git-setup]

<Steps>
  <li>
    **Create distinct authentication and signing keys** in your
    [GitHub key settings](https://github.com/settings/keys). Two keys, not
    one reused.
  </li>
  <li>
    **Fork the repository** and clone your fork, not upstream:

    ```bash
    git clone git@github.com:yourusername/nativelink
    cd nativelink
    git remote add upstream git@github.com:TraceMachina/nativelink
    ```

    `git remote -v` should show `origin` pointing at your fork and
    `upstream` at `TraceMachina/nativelink`.
  </li>
  <li>
    **Configure git to sign every commit and tag.** In `~/.gitconfig`:

    ```ini
    [user]
        name = Your Full Name
        email = the-email-you-use-on-github@example.com
        signingkey = ~/.ssh/your_private_signing_key

    [gpg]
        format = ssh   # or gpg, if you use a GPG key

    [commit]
        gpgsign = true

    [tag]
        gpgsign = true
    ```
  </li>
</Steps>

<Callout type="warn" title="Signing is not a DCO sign-off">
  NativeLink requires cryptographically **signed** commits, SSH or GPG, via
  `commit.gpgsign`. It does not use a Developer Certificate of Origin
  sign-off, so `git commit -s` alone will not satisfy the requirement.
</Callout>

Direct commits and human-created branches on `TraceMachina/nativelink` are
not allowed, including for members of the organization. Everything arrives
as a pull request from a fork.

## The pull request loop [#the-pull-request-loop]

<Steps>
  <li>
    **Sync your fork**, then branch:

    ```bash
    git switch main
    git pull -r upstream main
    git push
    git switch -c some-feature
    ```
  </li>
  <li>
    **Make one commit.** The title rules are enforced by review, not by a
    hook, and they are short: start with a capital letter, use the
    imperative, no trailing period, keep it as short as it can be, and put
    detail in the body wrapped at 72 characters.

    ```text
    Add some feature          ← good
    Add some feature.         ← trailing period
    Adds some feature         ← not imperative
    Add X and also Y          ← "and" means this is two commits
    ```

    The word "and" in a title is the signal to split the commit.
  </li>
  <li>
    **Push and open the PR** from your fork, then click the purple
    **Reviewable** button on the GitHub page to add reviewers with
    `+@somereviewer`. Review conversation happens there.
  </li>
  <li>
    **Revise with `--amend`, not a follow-up commit.**

    ```bash
    git commit --amend
    git push -f
    ```

    Reviewable keeps the diff between commit versions visible, so nothing is
    lost by rewriting, and the branch stays one commit, which is what lands.
  </li>
</Steps>

## Before you push [#before-you-push]

```bash
bazel test //...                                  # what CI runs
pre-commit run -a                                 # hooks, vale, typos
bazel run --config=rustfmt @rules_rust//:rustfmt  # formatting
```

<Callout type="warn" title="Stable `cargo fmt` is not the formatter">
  `.rustfmt.toml` uses `group_imports` and `imports_granularity`, both
  nightly-only. Stable `cargo fmt` ignores them silently and produces output
  CI rejects. Use the Bazel target.
</Callout>

`cargo test --all --profile=smol` is the fast inner loop, but a green Cargo
run is not a green CI: `bazel test` attaches rustfmt and clippy aspects that
Cargo does not. See [the testing guide](/contribute/testing-guide) for what
the suites contain and which local failures are environmental.

## What reviewers look for [#what-reviewers-look-for]

**One change per pull request.** Small ones land; large ones get split.

**A body that explains why.** The diff already shows what changed. The body
is where the constraint, the incident or the measurement goes.

**Tests that match the change.** A bug fix wants a regression test; new
behaviour wants both unit and integration coverage. No coverage
percentage gate; the question is whether the new path is exercised, not
whether a number moved.

**Conventions followed rather than argued.** Most review feedback on a first
contribution is about the house rules: the task macros instead of
`tokio::spawn`, `#[expect(..., reason = "…")]` rather than a bare `#[allow]`,
the `BUILD.bazel` `srcs` entry. [Codebase internals](/contribute/codebase-internals) is the page that
gets you past those in one read.

## Licensing [#licensing]

Your contribution is licensed under the license that applies to the file or
module you change. Most of the repository is `FSL-1.1-Apache-2.0`; a handful
of files are Apache-2.0; and two areas,
`nativelink-util/src/metrics.rs` and everything under
`nativelink-worker/src/persistent_worker/`, are **Business Source
License**. Copy the header from the file next to the one you are creating
rather than from memory.

Meaningful contributors may be eligible for license waivers on the Business
Source License modules. Raise an issue or contact the maintainers before
depending on one.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Which dev workflow should I pick?" id="which-dev-workflow-should-i-pick">
    [Cargo](/contribute/cargo) for small localized changes, the fastest way
    in. [Bazel](/contribute/bazel) when you want exactly what CI runs.
    [Nix](/contribute/nix) for the pinned shell, required for LRE work and
    for the generated `bazelrc` fragments.
  </Accordion>
  <Accordion title="My Cargo tests pass but CI is red." id="cargo-green-ci-red">
    Two usual causes. A new source file missing from its crate's
    `BUILD.bazel` `srcs` list, since Bazel enumerates where Cargo globs; or
    formatting, because `bazel test` attaches a rustfmt aspect and stable
    `cargo fmt` produces different output.
  </Accordion>
  <Accordion title="Do I sign with `git commit -s`?" id="do-i-sign-with-dash-s">
    No. That adds a DCO trailer, which this project does not use. Configure
    `commit.gpgsign = true` with an SSH or GPG signing key so commits are
    cryptographically signed.
  </Accordion>
  <Accordion title="I need to change something after review started." id="change-after-review">
    Amend and force-push. `git commit --amend && git push -f`. Reviewable
    tracks the diff between commit versions, so reviewers can see exactly
    what you changed without a follow-up commit polluting the history.
  </Accordion>
  <Accordion title="How long until someone looks at my PR?" id="how-long-until-review">
    Every PR is reviewed by a core contributor, including PRs from core
    contributors. No turnaround is published; if a PR goes quiet,
    comment on it or ping in the Reviewable thread.
  </Accordion>
</Accordions>

<NextStep href="/contribute/repo-crate-map" title="Repository and crate map">
  Where your change actually goes: twelve crates, a strict layering, and the
  six places the first guess is wrong.
</NextStep>

# https://docs.nativelink.com/contribute/bazel

**Develop with Bazel**: Build, test, and run NativeLink with Bazel, the same dev loop CI uses.

The Bazel-based dev loop is the one our CI runs. Use this if you want
the most reproducible setup or if your editor integration prefers
Bazel.

## Prerequisites [#prerequisites]

- Bazelisk, which reads `.bazelversion` and fetches the right Bazel for
  you. CI runs 9.1.1 and also checks 8.7.0.
- A C++ toolchain (clang or gcc) for native dependencies.
- On macOS: XCode command-line tools.
- On Linux, inside the Nix shell, the LRE toolchains link with `mold`;
  outside it Bazel uses whatever the host toolchain provides.

## First build [#first-build]

```bash
git clone https://github.com/TraceMachina/nativelink
cd nativelink
bazel test //...
```

The first `bazel test` will take 10-20 minutes as it builds the
toolchain and dependencies. Subsequent builds finish in seconds.

<Callout type="info" title="Hooking up your own cache">
  `.bazelrc` already defines `--config=self_test` (remote cache at
  `grpc://127.0.0.1:50051`) and `--config=self_execute` (remote executor at
  `grpc://127.0.0.1:50052`) for a NativeLink running locally. For any other
  endpoint, put the flags in `user.bazelrc` at the repository root, which
  `.bazelrc` pulls in with `try-import` and git ignores:

  ```text
  build --remote_cache=grpc://localhost:50051
  build --remote_executor=grpc://localhost:50051
  ```

  With a NativeLink cluster running, subsequent builds will hit the cache
  for any action your team has already produced.
</Callout>

## Common commands [#common-commands]

```bash
# Run all tests.
bazel test //...

# Run one crate's tests/ directory.
bazel test //nativelink-store:integration

# Run the in-file `mod tests` suites, or the doc examples.
bazel test //:unit_tests
bazel test doctests

# Build the server binary. The binary is defined in the root BUILD.bazel,
# so its label is `//:nativelink`; there is no `//nativelink` package.
bazel build //:nativelink

# Run the built binary against a config.
bazel run //:nativelink -- /path/to/config.json5

# Format Rust code. The formatting config uses nightly-only options, so
# stable `cargo fmt` produces different output than CI expects.
bazel run --config=rustfmt @rules_rust//:rustfmt
```

See [the testing guide](/contribute/testing-guide) for what `unit_test`
and `integration` actually mean in this repository; they do not mean what
the names suggest.

## Editor integration [#editor-integration]

For `rust-analyzer`:

```bash
bazel run @rules_rust//tools/rust_analyzer:gen_rust_project
```

This generates a `rust-project.json` that VS Code / Helix / your
editor of choice will pick up.

## Common pitfalls [#common-pitfalls]

- **LRE flags are missing**: `.bazelrc` pulls in `lre.bazelrc`,
  `nativelink.bazelrc`, `nixos.bazelrc` and `darwin.bazelrc` with
  `try-import`, and the Nix dev shell is what generates them. Without
  `nix develop` they do not exist, and the `try-` means Bazel
  says nothing about it. See [Develop with Nix](/contribute/nix).
- **Slow first build with no cache**: expected. Subsequent builds
  reuse Bazel's local cache; pointing at a remote NativeLink cluster
  makes a fresh clone fast for everyone.
- **A new source file builds with Cargo and fails under Bazel**: Bazel
  enumerates `srcs` rather than globbing. Add the file to the crate's
  `BUILD.bazel`, alphabetically.

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Why does the first build take 10-20 minutes?" id="why-does-the-first-build-take-10-20-minutes">
    It bootstraps the toolchain and builds every dependency from source.
    Subsequent builds are seconds, and pointing `--remote_cache` at a
    NativeLink cluster (above) makes even a fresh clone fast.
  </Accordion>
  <Accordion title="Do I have to run the whole test suite every time?" id="do-i-have-to-run-the-whole-test-suite-every-time">
    No. Target what you touched
    (`bazel test //nativelink-store:integration`), then run the
    full `bazel test //...` once before submitting.

    For non-Bazel workflows, see
    [Develop with Cargo](/contribute/cargo) or
    [Develop with Nix](/contribute/nix).
  </Accordion>
</Accordions>

# https://docs.nativelink.com/contribute/cargo

**Developing with Cargo**: Pure Cargo workflow for NativeLink contributors who'd rather not run Bazel.

Every NativeLink crate compiles cleanly with plain Cargo. Use this
workflow if you want a smaller dev shell, faster incremental builds
during iteration, or just don't have Bazel installed.

## Prerequisites [#prerequisites]

- A recent stable Rust toolchain. No `rust-toolchain.toml` is checked in; the
  minimum is `rust-version` in the root `Cargo.toml`, currently 1.93.1,
  on edition 2024.
- On Linux: `mold` is optional and has to be configured by hand; see
  [faster incremental builds](#faster-incremental-builds).

You do **not** need `protoc`. The generated Rust for every proto is
checked into `nativelink-proto/genproto/`, and `nativelink-proto`'s
`[lib] path` points straight at it; there is no build script.

## First build [#first-build]

```bash
git clone https://github.com/TraceMachina/nativelink
cd nativelink
cargo build --workspace
```

First build is a few minutes (dependency compilation); subsequent
incremental builds are typically a few seconds.

## Common commands [#common-commands]

```bash
# Build everything.
cargo build --workspace

# Run all tests. `smol` keeps target/ near 1GB instead of ~12GB.
cargo test --all --profile=smol

# Run a specific crate's tests.
cargo test -p nativelink-store

# Run the server.
cargo run -p nativelink -- /path/to/config.json5

# Lint with the workspace lint set from the root Cargo.toml. The Cargo CI
# lane builds and tests with RUSTFLAGS=-Dwarnings; clippy itself runs in CI
# through the Bazel aspect, on the same lint set.
cargo clippy --all-targets -- -D warnings
```

<Callout type="warn" title="Don't format with stable `cargo fmt`">
  `.rustfmt.toml` uses `group_imports` and `imports_granularity`, both
  nightly-only. Stable `cargo fmt` silently ignores them and produces
  output CI rejects. Run `bazel run --config=rustfmt @rules_rust//:rustfmt`
  instead.
</Callout>

## Faster incremental builds [#faster-incremental-builds]

A few flags that help:

- `mold` as linker (Linux):

  ```toml
  # ~/.cargo/config.toml
  [target.x86_64-unknown-linux-gnu]
  linker = "clang"
  rustflags = ["-C", "link-arg=-fuse-ld=mold"]
  ```

- `CARGO_TARGET_DIR` set to a shared directory across checkouts to
  reuse incremental artifacts.

## Caveats [#caveats]

- Three directories are excluded from the workspace and build on their
  own: `nativelink-config/generate-stores-config`, `nativelink-test/fuzz`,
  and `tools/generate-bazel-rc`. The LRE flake outputs are Nix, not Cargo.
- Regenerating protos goes through Bazel
  (`bazel run nativelink-proto:update_protos`), not Cargo.
- CI runs Bazel, and `bazel test` attaches rustfmt and clippy aspects that
  a Cargo run does not. A green `cargo test` does not guarantee a green CI.
  Run `bazel test //...` before submitting.

For the Bazel-based workflow, see
[Develop with Bazel](/contribute/bazel). For what the test suites contain,
see [the testing guide](/contribute/testing-guide).

## FAQ [#faq]

<Accordions type="multiple">
  <Accordion title="Do I need Bazel installed at all?" id="do-i-need-bazel-installed-at-all">
    Not to build or test; the whole workspace builds with plain Cargo.
    You need Bazel for a pre-submit `bazel test //...`, for regenerating
    protos, and for formatting, since that's what CI runs.
  </Accordion>
  <Accordion title="Do I need protoc?" id="why-is-protoc-a-prerequisite">
    No. The generated Rust is committed under `nativelink-proto/genproto/`
    and there is no build script invoking `protoc`. If you change a `.proto`,
    regenerate with `bazel run nativelink-proto:update_protos` and commit the
    result. `//nativelink-proto:update_protos_test` is the CI gate for that.
  </Accordion>
</Accordions>

# https://docs.nativelink.com/contribute/nix

**Develop with Nix**: The dev shell that generates the bazelrc files the rest of the build assumes, and what is actually inside it.

**Who this is for:** anyone who wants the toolchain CI uses, and anyone
working on [LRE](/explanations/lre), which needs it. **What you'll have at
the end:** the shell, the five generated `bazelrc` files it produces, and an
accurate list of what it does and does not contain. **Time:** twenty
minutes, mostly download.

The flake in the repository root is where almost all of NativeLink's tooling
lives. Entering the shell is not just a convenience: the shell hook is what
writes the `bazelrc` fragments that `.bazelrc` pulls in with `try-import`,
and `try-import` means their absence is silent. A build outside the shell is
not a broken build; it is a *differently configured* one.

## Install Nix [#install-nix]

The
[next-gen Nix installer](https://github.com/NixOS/experimental-nix-installer)
is the easiest path:

```bash
curl -sSfL https://artifacts.nixos.org/nix-installer | sh -s -- install --enable-flakes
```

The `--enable-flakes` flag turns on the `nix-command` and `flakes`
experimental features, which the repository's flake needs.

<Callout type="warn" title="Three things that go wrong on first install">
  - **`nix` not found after installing.** Restart your terminal, or source
    the profile script:

    ```bash
    . /nix/var/nix/profiles/default/etc/profile.d/nix-daemon.sh
    ```

  - **`experimental Nix feature 'nix-command' is disabled`.** You used the
    standard installer. Add this to `~/.config/nix/nix.conf`:

    ```text
    experimental-features = nix-command flakes
    ```

  - **Disk space.** Toolchains, compiler binaries and build artifacts are
    large. Have **15 to 20 GB free** before the first `nix develop`.
</Callout>

## Enter the dev shell [#enter-the-dev-shell]

```bash
git clone git@github.com:yourusername/nativelink
cd nativelink
nix develop
```

First run downloads the toolchain, which takes a few minutes. After that,
entering is close to instant; everything is content-addressed in
`/nix/store`.

<Callout type="info" title="Install `direnv` and stop thinking about this">
  `CONTRIBUTING.md` recommends it strongly, and the reason is that the shell
  has to be re-entered after every branch switch or change to a `.nix` file.
  `direnv` does that for you:

  ```bash
  nix profile install nixpkgs#direnv
  # add the hook for your shell: https://direnv.net/docs/hook.html
  # then, in the repo: direnv allow
  ```

  To check the environment is active: `env | grep NIX` should print several
  `*NIX_*` variables.
</Callout>

## What entering the shell generates [#what-the-shell-generates]

This is the part that matters and the part that is invisible. The shell hook
writes five things into your working tree:

| Generated file | What it does |
| --- | --- |
| `.pre-commit-config.yaml` | Symlink enabling the repo's pre-commit hooks |
| `lre.bazelrc` | Points Bazel at the LRE toolchains |
| `nativelink.bazelrc` | Gives your builds access to NativeLink's read-only cache |
| `nixos.bazelrc` | Adds the required NixOS binary paths (NixOS only) |
| `darwin.bazelrc` | Configures Darwin libs and frameworks (macOS only) |

`.bazelrc` picks all four `bazelrc` fragments up with `try-import`, so
outside the shell Bazel silently proceeds without them. That is why a build
can succeed outside Nix and still not match CI.

## What is in the shell [#what-is-in-the-shell]

The package list is in
<SourceLink file="flake.nix" symbol="devShells.default" />.
Broadly:

**Rust and build**: a `bazel` wrapper around Bazelisk (it unsets `TMPDIR`
first), the pinned stable Rust toolchain from the LRE overlay,
`rust-analyzer`, `buck2`, `python3`, `git`, `pre-commit`, `git-cliff`.

**Web**: `bun`, `nodejs_22`, `lychee`, the Playwright driver. This is what
lets you work on the docs without installing anything.

**Infrastructure**: `kubectl`, `kubernetes-helm`, `kustomize`, `kubectx`,
`kind`, `cilium-cli`, `fluxcd`, `pulumi`, `skopeo`, `dive`, `cosign`,
`trivy`, `awscli2`, `google-cloud-sdk`, `docker-client`, `tektoncd-cli`,
`go`.

**Docs and prose**: `vale`, which is the style linter the docs are checked
against.

**Repo-specific helpers**: the toolchain generators, the local image
builders, `local-image-test`, and the LRE clang.

<Callout type="warn" title="Some things you might expect are not here">
  No `protoc` is provided; the generated Rust is checked into
  `nativelink-proto/genproto/` and nothing invokes `protoc` at build time.
  No prebuilt `nativelink` binary is in the shell; build it with Bazel
  or Cargo, or run `nix run .` which builds and runs the flake's default
  package. And `CONTRIBUTING.md` still advises installing Clang separately;
  the shell does now ship `lre.clang`, so try without first, and fall back to
  `nix profile install nixpkgs#clang` if a C++ toolchain error says
  otherwise.
</Callout>

## Running things inside the shell [#running-things-inside-the-shell]

Everything from the [Bazel](/contribute/bazel) and
[Cargo](/contribute/cargo) workflows works unchanged. The shell guarantees
the versions, and supplies the `bazelrc` fragments those workflows assume.

The two commands worth knowing that only exist here:

```bash
pre-commit run -a                       # the full pre-commit suite
nix build .#nativelinkCoverageForHost   # HTML coverage report in ./result
```

## LRE and worker images [#lre-and-worker-images]

The flake builds the LRE worker images rather than exposing an LRE shell.
The ones you are likely to want:

```bash
nix build .#nativelink-worker-lre-cc
nix build .#nativelink-worker-lre-rs
nix build .#nativelink-worker-lre-java
```

`lre.bazelrc`, generated on entering the shell, is what points a local Bazel
build at the matching toolchains. See [LRE](/explanations/lre) for what the
scheme is doing and why the `/nix/store` paths have to match on both sides.

## When Nix isn't worth it [#when-nix-isnt-worth-it]

For a typo fix, a doc tweak or a single unit test, the
[Cargo workflow](/contribute/cargo) is faster to get into and nothing will
go wrong. Reach for Nix when you are touching the build graph, working on
LRE, chasing a difference between your machine and CI, or working on the
docs, since `vale` and `bun` both come from here.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="What is Nix, in one sentence?" id="what-is-nix-in-one-sentence">
    A package manager that builds every package in isolation and addresses
    the result by a hash of all its inputs, which is why two machines
    running `nix develop` on this repo get bit-identical toolchains.
  </Accordion>
  <Accordion title="Is the dev shell required?" id="is-the-dev-shell-required">
    Not for building or testing; Cargo and Bazel both stand on their own.
    It is required for LRE work, and it is the only way to get the generated
    `bazelrc` fragments, so a build outside it is configured differently from
    CI even when it is green.
  </Accordion>
  <Accordion title="My Bazel build behaves differently outside the shell." id="different-outside-shell">
    Almost certainly the `try-import`ed fragments. `lre.bazelrc`,
    `nativelink.bazelrc`, `nixos.bazelrc` and `darwin.bazelrc` are generated
    by the shell hook and silently skipped when absent. Enter the shell, or
    accept that you are building without the LRE toolchains and without the
    read-only cache.
  </Accordion>
  <Accordion title="Do I have to re-enter the shell after switching branches?" id="reenter-after-branch-switch">
    Yes, whenever the `.nix` files change, which a branch switch can do
    without you noticing. This is the argument for `direnv`: it re-enters
    automatically on `cd` and on file changes.
  </Accordion>
</Accordions>

<NextStep href="/contribute/repo-crate-map" title="Repository and crate map">
  With a working shell, the next question is where your change goes. Twelve
  crates, a strict layering, and six places the first guess is wrong.
</NextStep>

# https://docs.nativelink.com/contribute/repo-crate-map

**Repository and crate map**: What every crate and top-level directory in the NativeLink repository is for, and which one your change belongs in.

**Who this is for:** anyone opening the NativeLink repository for the first
time and trying to work out where their change goes. **What you'll have at
the end:** the crate layering, the size of each piece, the directories that
are not crates, and the half-dozen places where the first guess is wrong.
**Time:** fifteen minutes.

NativeLink is one Cargo workspace of twelve crates plus a root binary, and
the layering is strict enough that you can usually name the crate a change
belongs in before you open an editor. The exceptions are worth learning
early, because two of them put code in a crate whose name actively points
somewhere else.

## The workspace has no members list [#the-workspace-has-no-members-list]

The root
<SourceLink file="Cargo.toml" symbol="Cargo.toml" />
declares `[workspace]` with only `exclude` and `resolver`. It has no
`members` key. Membership is inferred from the path dependencies of the root
`[package]`, which means **adding a crate to the workspace happens by
depending on it**, not by editing a list.

Three directories are explicitly excluded and build on their own:
`nativelink-config/generate-stores-config`, `nativelink-test/fuzz`, and
`tools/generate-bazel-rc`.

<Callout type="info" title="The root Cargo.toml is a package, not just a workspace">
  The version at the top of the root manifest is a `[package]` version, not
  a `[workspace.package]` version. Nothing inherits it; every crate carries
  its own copy, which is why a release bump edits the version in fourteen
  files (seventeen once the two lock files and the changelog are counted).
  See [releases and versioning](/contribute/release-versioning).
</Callout>

## The crates [#the-crates]

Source sizes exclude each crate's `tests/` directory; the test column counts
files directly under `tests/`.

| Crate | Source | Tests | What lives here |
| --- | --- | --- | --- |
| `nativelink` (root `src/`) | 3 files, ~1.4k lines | none | `main()`, runtime setup, service wiring, TLS and listener construction; also the `redis_store_tester` and `cas_speed_check` binaries |
| `nativelink-store` | 39 files, ~18.4k | 28 | Every store implementation and the composition machinery |
| `nativelink-util` | 29 files, ~11.9k | 20 | The shared hub: store traits, scheduler traits, task spawning, fs, health, telemetry |
| `nativelink-worker` | 12 files, ~9.0k | 7 | The worker runtime: action execution, materialization, upload |
| `nativelink-scheduler` | 20 files, ~8.3k | 10 | Queueing, matching, state managers, worker bookkeeping |
| `nativelink-service` | 13 files, ~5.4k | 13 | The gRPC service implementations |
| `nativelink-config` | 7 files, ~4.7k | 3 | The config schema, the source of truth for the generated reference, plus the `build-schema` binary |
| `nativelink-proto` | 20 files, ~18.3k | none | Vendored protos and their **checked-in** generated Rust |
| `nativelink-redis-tester` | 5 files, ~0.9k | none | Fake Redis, mock pub-sub, read-only Redis |
| `nativelink-error` | 1 file, ~0.6k | 2 | `Error`, `Code`, `ResultExt`, the error macros |
| `nativelink-metric` (with the nested `nativelink-metric-macro-derive`) | 2 files, ~0.8k | none | The `MetricsComponent` trait, its derive, and the `publish!`/`group!` macros |
| `nativelink-macro` | 1 file, 99 lines | none | `#[nativelink_test]` |

## The layering [#the-layering]

Every internal dependency points down this stack. It has no cycles, and
the shape is worth memorising because it tells you what a change can reach.

<Mermaid>{`graph TD
  proto["nativelink-proto"]
  macroderive["nativelink-metric-macro-derive"]
  metric["nativelink-metric"]
  error["nativelink-error"]
  config["nativelink-config"]
  util["nativelink-util<br/>the hub"]
  redis["nativelink-redis-tester"]
  store["nativelink-store"]
  sched["nativelink-scheduler"]
  worker["nativelink-worker"]
  service["nativelink-service"]
  bin["nativelink<br/>root binary"]
  macroderive --> metric
  proto --> error
  metric --> error
  error --> config
  config --> util
  proto --> util
  metric --> util
  redis --> store
  util --> redis
  util --> store
  store --> sched
  store --> worker
  sched --> service
  store --> service
  sched --> bin
  service --> bin
  worker --> bin
`}</Mermaid>

`nativelink-util` is the hub. Almost every trait that two crates need to
agree on lives there rather than in the crate that looks like it owns the
concept: `Store` and `StoreDriver` are in `nativelink-util`, not
`nativelink-store`, and the four scheduler state-manager traits are in
`nativelink-util`, not `nativelink-scheduler`. This is deliberate: the
implementations depend on the traits, and the traits must not depend on the
implementations.

<Callout type="warn" title="`nativelink-redis-tester` is a production dependency">
  Its name says test utility; `nativelink-store` depends on it in
  `[dependencies]`, not `[dev-dependencies]`. Treat changes to it as changes
  to shipping code.
</Callout>

## Where the first guess is wrong [#where-the-first-guess-is-wrong]

Six of these, and each one has cost somebody an afternoon.

**Worker bookkeeping lives in the scheduler.** `worker.rs`,
`worker_registry.rs` and `api_worker_scheduler.rs` are in
`nativelink-scheduler`. The scheduler's *model* of a worker (its
properties, its consumed resources, its liveness) belongs to the scheduler.
`nativelink-worker` is only the runtime that executes actions.

**`nativelink-test/` contains only `fuzz/`.** The `#[nativelink_test]`
attribute macro is in `nativelink-macro`. The directory name is a leftover.

**`nativelink-metric` does not depend on `nativelink-error`.** It cannot;
`nativelink-error` depends on `nativelink-metric`, and the source carries a
comment explaining that the direction was chosen to break the cycle. A
metrics change that wants to return a `nativelink_error::Error` is a change
that needs rethinking.

**Generated proto code is checked in.** `nativelink-proto/genproto/` holds
committed Rust, and the `[lib]` path points at it. You do not need `protoc`
to build; you do need to regenerate and commit when a proto changes. See
[how to extend](/contribute/how-to-extend#adding-a-grpc-service).

**Store composition is not in one file.** A `verify` store wrapping a
`fast_slow` over a `filesystem` is three files in `nativelink-store`, and
the thing that decides what wraps what is
`default_store_factory.rs`. [The store model](/explanations/store-model)
explains the semantics.

**Two metric systems coexist.** Which one you want depends on whether you
are introspecting structure or graphing a number. See
[codebase internals](/contribute/codebase-internals#two-metric-systems).

## Directories that are not crates [#directories-that-are-not-crates]

| Directory | What it is |
| --- | --- |
| `deploy/` | Kustomize overlays (`dev`, `kubernetes-example`, `lre-manual`, `chromium-example`) that assemble the pieces under `kubernetes/` |
| `deployment-examples/` | Reference deployments you copy and adapt: `docker-compose`, `metrics`, `persistent-workers`, `rhel` |
| `kubernetes/` | The Kustomize building blocks (`components/`, `resources/`, `workers/`) the overlays reference; Kubernetes deployments proper, including the Helm charts, ship with [NativeLink Enterprise](https://enterprise.nativelink.com) |
| `integration_tests/` | Shell-driven end-to-end suites plus BuildStream, Buck2 and Mongo harnesses |
| `local-remote-execution/` | The Nix-based LRE toolchain generation; see [LRE](/explanations/lre) |
| `toolchain-examples/`, `templates/` | Starting points for downstream projects; both are `.bazelignore`d |
| `tools/` | Nix modules, pre-commit hooks, toolchain definitions, CI helpers |
| `web/` | The docs and marketing sites: Bun workspace, **not** part of the Bazel build |
| `assets/` | The logo SVGs |

<Callout type="info" title="`deploy/`, `deployment-examples/` and `kubernetes/` are three different things">
  They are not alternative spellings of each other. `deploy/` is the
  overlays, `deployment-examples/` is copyable manifests, and `kubernetes/`
  is the building blocks the overlays compose.
</Callout>

## Conventions you will notice immediately [#conventions-you-will-notice-immediately]

Every manifest except the nested macro-derive crate's starts with
`#:schema tools/cargo-with-detailed-deps.json`, which is what gives you
completion and validation in a TOML-aware editor.

Every crate except `nativelink-proto` (generated code) and
`nativelink-metric-macro-derive` has `lints.workspace = true` and no lint
configuration of its own; all of it is centralised in the root
`[workspace.lints]`.

Three license headers are in use. FSL-1.1-Apache-2.0 is the default and
covers the overwhelming majority of files; a handful are Apache-2.0; and two
areas are **Business Source License**:
`nativelink-util/src/metrics.rs` and all of
`nativelink-worker/src/persistent_worker/`. Copy the header from the file
next to the one you are creating rather than from memory.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="I want to add a store. Which crates do I touch?" id="which-crates-for-a-store">
    Two, plus a Bazel file: the spec goes in `nativelink-config`, the
    implementation and its factory arm go in `nativelink-store`, and the new
    file has to be added to `nativelink-store/BUILD.bazel`. The step-by-step
    is in [how to extend](/contribute/how-to-extend#adding-a-store).
  </Accordion>
  <Accordion title="Why is there no `members` list in the workspace?" id="no-members-list">
    Membership is derived from the root package's path dependencies. It
    works, and it means the dependency graph and the workspace definition
    cannot drift apart, but it does mean a crate nothing depends on is
    silently not in the workspace.
  </Accordion>
  <Accordion title="Can I build without Bazel?" id="build-without-bazel">
    Yes. `cargo build --all` builds the whole workspace. CI runs Bazel as
    well, and a few checks exist only there, so run `bazel test //...` before
    submitting. See [the testing guide](/contribute/testing-guide).
  </Accordion>
  <Accordion title="Where does `main()` live?" id="where-is-main">
    `src/bin/nativelink.rs`, in the root package. It builds the Tokio runtime
    by hand, constructs the stores, then assembles the gRPC routes. It is the
    single best file to read first if you want to see how the pieces connect.
  </Accordion>
</Accordions>

<NextStep href="/contribute/codebase-internals" title="Codebase internals">
  The traits, the error model, the async rules and the two metric systems:
  the conventions that a reviewer will expect you to already know.
</NextStep>

# https://docs.nativelink.com/contribute/codebase-internals

**Codebase internals**: The traits, the error model, the async rules and the two metric systems: the conventions a reviewer will assume you already know.

**Who this is for:** anyone about to write NativeLink code rather than read
it. **What you'll have at the end:** the four things that are enforced
rather than suggested: the store trait split, the error model, the
spawning rules, and which metric system to use. **Time:** thirty minutes.

Most of NativeLink's review feedback is about conventions, not logic. The
conventions are strong, mostly mechanically enforced, and each one exists
for a reason you can point at in the source. This page is the set of them
that a first contribution reliably trips over.

The repository targets Rust edition 2024 with `rust-version = "1.93.1"`.

## The three store traits, and which one you implement [#the-three-store-traits]

Three of them exist, and they are not interchangeable.

**`StoreDriver`**
(<SourceLink file="nativelink-util/src/store_trait.rs" symbol="StoreDriver" />)
is the one you implement. Its supertraits are
`Sync + Send + Unpin + MetricsComponent + HealthStatusIndicator + 'static`,
so a new store is observable and health-checkable by construction, not by
remembering to add it.

**`Store`** is a `#[repr(transparent)]` newtype over `Arc<dyn StoreDriver>`.
It is what everything else holds.

**`StoreLike`** is the caller-facing surface: two required methods, every
convenience built on top of them as a default, and blanket impls so that
both `Store` and `impl StoreDriver` satisfy it. You call `StoreLike`; you
implement `StoreDriver`.

The `StoreDriver` methods you must supply are `post_init`,
`has_with_results`, `update`, `get_part`, `inner_store`, `as_any`,
`as_any_arc` and `register_remove_callback`.

<Callout type="warn" title="`inner_store` is a barrier, not a getter">
  It is how a caller resolves an optimization ("can I hardlink this?")
  through a stack of wrappers. A routing wrapper forwards it; a wrapper that
  changes the bytes must return `self`, because what is underneath is not the
  blob. Getting this wrong is a **runtime** error, not a compile error: the
  trait checks with `addr_eq` and fires an `error_if!` if a store returns
  `self` while advertising an optimization it cannot honour. [The store
  model](/explanations/store-model) covers the semantics from the operator's
  side.
</Callout>

`StoreKey` is a two-variant enum, `Str` or `Digest`, with a variant-salted
`Hash` so that a string key and a digest key cannot collide.

Register health with the `default_health_status_indicator!(YourType);`
macro as the last line of the file, unless the store has something real to
report. Health checks across all components run **concurrently**
(`buffer_unordered(usize::MAX)`), with a comment in
<SourceLink file="nativelink-util/src/health_utils.rs" symbol="health_utils.rs" />
explaining that a serial sweep would blow the kubelet's probe timeout.

## The scheduler traits are split by caller [#the-scheduler-traits-are-split-by-caller]

No `ActionScheduler` trait exists. The state layer is split into
`ClientStateManager`, `WorkerStateManager` and `MatchingEngineStateManager`
in
<SourceLink file="nativelink-util/src/operation_state_manager.rs" symbol="operation_state_manager.rs" />,
plus `WorkerScheduler`, `KnownPlatformPropertyProvider` and
`AwaitedActionDb`. The split by *caller* is what allows the in-memory and
Redis-backed implementations to be swapped wholesale. See
[scheduler internals](/explanations/scheduler-internals#there-is-no-actionscheduler-trait).

## The error model [#the-error-model]

`Code` **is** `tonic::Code`, re-exported from
<SourceLink file="nativelink-error/src/lib.rs" symbol="Error" />.
No separate internal code enum and no mapping layer exist, which is why
an error raised deep in a store arrives at a client with a sensible gRPC
status without anyone writing a conversion.

An `Error` carries a code, a **`Vec<String>` of messages**, and a context.
The messages are a stack, joined with `" : "` when displayed. This is the
whole point of the model: each layer appends what it was trying to do, so
the final message reads outside-in.

```rust
some_store
    .get_part_unchunked(key, 0, None)
    .await
    .err_tip(|| "while loading the action result")?;
```

Three macros do most of the work: `make_err!`, `make_input_err!` and
`error_if!`. `ResultExt`, and therefore `err_tip`, is implemented for
both `Result` and `Option`, so an `Option` can become a well-described
error without an intermediate `ok_or_else`.

Around twenty-five `From<…> for Error` impls exist for the common foreign
error types, so `?` usually just works.

<Callout type="info" title="`err_tip` closures are lazy on purpose">
  `err_tip` takes a closure so the string is only built on the error path.
  Passing an already-formatted `String` compiles and costs an allocation on
  every success.
</Callout>

`clippy::todo` is denied. `clippy::unwrap_used` is currently `allow`, with a
`TODO` in the manifest recording the intent to flip it; write new code as
though it were already denied.

## Async: `tokio::spawn` is banned [#async-tokio-spawn-is-banned]

The Tokio runtime is built by hand in `main()`, not with `#[tokio::main]`:
the builder installs an `on_thread_start` hook (macOS QoS), and the process
signal handlers and the shutdown broadcast are spawned onto it before
`inner_main` runs. Each of those calls carries an
`#[expect(clippy::disallowed_methods, reason = …)]`, because the same lint
that bans `tokio::spawn` everywhere else bans the runtime builder too.

Fourteen entries in
<SourceLink file="clippy.toml" symbol="clippy.toml" />
make the standard spawning API a lint error. Use the three macros in
<SourceLink file="nativelink-util/src/task.rs" symbol="task.rs" />
instead:

| Macro | Returns | Use when |
| --- | --- | --- |
| `spawn!` | `JoinHandleDropGuard`, which **aborts on drop** | The task's life is tied to the caller's |
| `background_spawn!` | A bare `JoinHandle` | The task should outlive the caller |
| `spawn_blocking!` | `JoinHandleDropGuard` around a blocking-pool task | The work blocks a thread |

The first argument to every one of them is a task name, which becomes a
`tracing::error_span!`. That name is how a task is identified in logs and
traces, so make it specific. OpenTelemetry context propagates into spawned
tasks automatically.

<Callout type="warn" title="`spawn!` aborts when its guard is dropped">
  This is a feature (it makes cancellation the default) and a trap if you
  spawn work and then let the handle fall out of scope. If the task must
  survive, use `background_spawn!` and say why in a comment.
</Callout>

File I/O goes through `nativelink_util::fs`, which serialises open file
descriptors behind a global semaphore (`DEFAULT_OPEN_FILE_LIMIT` is 24576).
Opening a file with `std::fs` or `tokio::fs` directly bypasses the limit and
will eventually exhaust the process's descriptors under load.

Shutdown is coordinated by
<SourceLink file="nativelink-util/src/shutdown_guard.rs" symbol="ShutdownGuard" />
with three priorities: `P0`, `P1`, `LeastImportant`. Awaiting `wait_for` at
a given priority blocks until everything less important has drained. `SIGTERM`
drains and then exits `143`; `SIGINT` is a hard exit `130`.

## Two metric systems [#two-metric-systems]

They are not redundant, and picking the wrong one is a common review
comment.

**System A: `nativelink-metric`.** A `#[derive(MetricsComponent)]` plus
`#[metric(help = …, kind = …, group = …, handler = …)]` on fields, with
`RootMetricsComponent` marking a tree root. Roughly 190 `#[metric]`
attributes exist. Publishing a tree walks the live object graph and emits
one `tracing` span per component and one event per field, on the target
`nativelink_metric`, so what it describes is the *structure* of a running
process. Nothing in the v1.6.5 binary serves that tree over HTTP; the old
metrics collector went away with the OpenTelemetry migration, so today the
derive is what keeps every store and scheduler describable, and the trait
bound on `StoreDriver` is what forces new ones to be.

**System B: `nativelink-util/src/metrics.rs`.** OpenTelemetry instruments
in eight `LazyLock` bundles (`CACHE_METRICS`, `EXECUTION_METRICS`,
`WORKER_METRICS`, `RPC_METRICS`, `SCHEDULER_METRICS`, `STORE_TIER_METRICS`,
`HEALTH_METRICS`, `CONNECTION_METRICS`), wired up in
`telemetry.rs::init_tracing` and exported over OTLP (`NL_OTEL_ENDPOINT`
overrides the endpoint). Attribute sets for the hot paths are
**pre-computed** into `CacheMetricAttrs` and `ExecutionMetricAttrs` structs
specifically to bound cardinality.

Choose A when you are exposing the *structure* of a component: how many
stores, which config, what state. Choose B when you want a number someone
will graph or alert on.

<Callout type="warn" title="Two hazards in the OTel path">
  `nativelink-util/src/metrics.rs` carries the **Business Source License**
  header, not the FSL header the rest of the crate uses; do not paste the
  wrong one in. And never attach a per-action or per-worker identifier to a
  hot instrument; the pre-computed attribute structs exist precisely to stop
  that, and adding a high-cardinality dimension will take down a Prometheus
  before it takes down NativeLink.
</Callout>

Instrument names are dotted OpenTelemetry style, with `.with_description`
and a `.with_unit` that is either a UCUM unit (`"By"`, `"s"`, `"ms"`) or a
braced count such as `"{entry}"`, `"{action}"` or `"{worker}"`.

## Lints, formatting and the small stuff [#lints-formatting-and-the-small-stuff]

`clippy::all`, `clippy::nursery` and `clippy::pedantic` are all denied at
the workspace level. `std_instead_of_core` is denied too, which is why you
will see `use core::…` throughout for anything that does not need `std`.

Suppress a lint with `#[expect(lint, reason = "…")]` rather than a bare
`#[allow]`; `expect` fails the build once the suppression is no longer
needed, which is how they get cleaned up. `#[allow]` still appears where the
lint only fires on some targets or feature sets, and it should carry a
`reason` too. Nothing enforces this mechanically; reviewers do.

`.rustfmt.toml` uses `group_imports = "StdExternalCrate"` and
`imports_granularity = "Module"`, both nightly-only options. Do not run
stable `cargo fmt` and expect it to match; run the Bazel target instead:

```bash
bazel run --config=rustfmt @rules_rust//:rustfmt
```

`unsafe` is rare: a few dozen blocks in the whole tree, mostly FFI, a few
measured optimizations, and environment-variable writes in tests. Most carry
a `// SAFETY:` comment, and a new one without a justification will be asked
for one.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Why can't I use `tokio::spawn`?" id="why-not-tokio-spawn">
    Because a bare spawn has no name, no tracing span, no OpenTelemetry
    context and no cancellation story. The three macros in
    `nativelink-util::task` supply all four, and the clippy config makes the
    alternative a build failure rather than a review comment.
  </Accordion>
  <Accordion title="Should my new metric be a derive or an OTel instrument?" id="derive-or-otel">
    If it describes what a component *is* (its config, its children, its
    internal counters as part of the object tree), derive; the
    `MetricsComponent` bound on `StoreDriver` requires the impl anyway. If a
    dashboard would graph it or an alert would fire on it, OTel. A few things
    legitimately want both; most want exactly one.
  </Accordion>
  <Accordion title="My store compiles but panics at runtime about `inner_store`." id="inner-store-runtime-error">
    You returned something other than `self` from `inner_store` while
    advertising an optimization your store cannot honour, or the reverse.
    A wrapper that changes the bytes on the way through must return `self`;
    a wrapper that only routes should forward to its inner store.
  </Accordion>
  <Accordion title="Which license header do I put on a new file?" id="which-license-header">
    Copy it from the file next to yours. FSL-1.1-Apache-2.0 is the default;
    `nativelink-util/src/metrics.rs` and everything under
    `nativelink-worker/src/persistent_worker/` are Business Source License,
    and a handful of files are Apache-2.0.
  </Accordion>
  <Accordion title="`cargo fmt` keeps fighting with CI. Why?" id="cargo-fmt-fights-ci">
    The formatting config uses nightly-only options. Stable `cargo fmt`
    silently ignores them and produces different output. Run
    `bazel run --config=rustfmt @rules_rust//:rustfmt`, which is what CI runs.
  </Accordion>
</Accordions>

<NextStep href="/contribute/how-to-extend" title="How to extend NativeLink">
  The four recipes (a store, a config field, a metric, a gRPC service),
  each with the complete list of files a working change touches.
</NextStep>

# https://docs.nativelink.com/contribute/how-to-extend

**How to extend NativeLink**: Complete file-by-file recipes for adding a store, a config field, a metric, or a gRPC service.

**Who this is for:** anyone adding a new component rather than changing an
existing one. **What you'll have at the end:** four recipes, each listing
every file a working change touches, including the ones that compile fine
when you forget them and fail in CI. **Time:** use as a checklist.

Each recipe below is derived from the smallest existing example in the tree.
Copy that example rather than starting from a blank file: it already has the
right license header, the right derives, and the right last line.

Read [codebase internals](/contribute/codebase-internals) first if you have
not; the trait split and the spawning rules are assumed here.

## Adding a store [#adding-a-store]

Model: `nativelink-store/src/noop_store.rs`, the smallest complete
`StoreDriver` in the tree.

<Steps>
  <li>
    **Add the spec** to `nativelink-config/src/stores.rs`. A new variant on
    `StoreSpec` (the JSON5 key is the snake_case form of the variant name;
    box the payload if it is large) plus a spec struct modelled on
    `MemorySpec`:

    ```rust
    #[derive(Serialize, Deserialize, Debug, Clone)]
    #[serde(deny_unknown_fields)]
    #[cfg_attr(feature = "dev-schema", derive(JsonSchema))]
    pub struct MyStoreSpec {
        /// What this field is for.
        /// Default: 0
        pub some_field: usize,
    }
    ```

    The `dev-schema` `cfg_attr` is not optional; without it your store never
    appears in the generated
    [configuration reference](/reference/nativelink-config).
  </li>
  <li>
    **Write the store** at `nativelink-store/src/my_store.rs`: the license
    header copied from a neighbouring file, the struct, a `MetricsComponent`
    impl, `pub fn new(...) -> Arc<Self>`, `#[async_trait] impl StoreDriver`,
    and `default_health_status_indicator!(MyStore);` as the last line.
  </li>
  <li>
    **Declare the module**: `pub mod my_store;` in
    `nativelink-store/src/lib.rs`.
  </li>
  <li>
    **Wire the factory**: an import and a match arm in
    `nativelink-store/src/default_store_factory.rs`. The match is exhaustive,
    so this one is a compile error if you forget it. It is the only step that
    is.
  </li>
  <li>
    **Add the file to `nativelink-store/BUILD.bazel`**, in the `srcs` list,
    **alphabetically**. Cargo does not care. Bazel does, and CI runs Bazel;
    this is the single most common way a store PR goes red after a green
    local run.
  </li>
  <li>
    **Add a test** at `nativelink-store/tests/my_store_test.rs` and register
    it in the crate's `rust_test_suite`. See
    [the testing guide](/contribute/testing-guide).
  </li>
</Steps>

<Callout type="warn" title="The store examples file is generated from your doc comments">
  `nativelink-config/examples/stores-config.json5` is produced by the
  `generate-stores-config` tool from the ```` ```json ```` fenced blocks in
  the doc comments in `stores.rs`, via a pre-commit hook. And
  `tests/json5_test.rs` parses every example, so a malformed JSON fence in a
  doc comment is not a cosmetic problem; it breaks the build.
</Callout>

Two semantics to get right, both covered in
[the store model](/explanations/store-model):

**`inner_store` must return `self`** if your store changes the bytes on the
way through, and forward to the inner store if it only routes. Returning the
wrong one is a runtime error, not a compile error.

**`update` must drain its reader** even when it is discarding the data.
`NoopStore` does this; a store that returns early without draining will hang
its writer.

## Adding a config field [#adding-a-config-field]

Config fields are documentation. The doc comment becomes the description in
the generated reference, and the ```` ```json ```` fences in `stores.rs` are
also harvested into the examples file, which is why they have to be valid
JSON rather than illustrative fragments.

<Steps>
  <li>
    **Add the field with a doc comment ending in a default line:**

    ```rust
    /// How long to wait before giving up on a connection.
    /// Default: 5 (seconds)
    #[serde(default, deserialize_with = "convert_duration_with_shellexpand")]
    pub connection_timeout_s: u64,
    ```

    The `Default:` line is parsed out of the description by the docs
    generator. Without it the reference shows no default.
  </li>
  <li>
    **Pick the right `deserialize_with` helper** from
    <SourceLink file="nativelink-config/src/serde_utils.rs" symbol="serde_utils.rs" />.
    They all expand shell variables, which is how a config file references
    the environment.
  </li>
  <li>
    **Regenerate the reference** if you want to see the result:
    `bun --filter @nativelink/docs gen:config-reference` from `web/`.
  </li>
</Steps>

The helpers that exist:

| Helper | For |
| --- | --- |
| `convert_numeric_with_shellexpand` | Any integer type |
| `convert_optional_numeric_with_shellexpand` | `Option<N>` |
| `convert_string_with_shellexpand` | `String` |
| `convert_optional_string_with_shellexpand` | `Option<String>` |
| `convert_vec_string_with_shellexpand` | `Vec<String>` |
| `convert_boolean_with_shellexpand` | `bool` |
| `convert_data_size_with_shellexpand` | Byte sizes; accepts `1GB`, `512MiB` |
| `convert_optional_data_size_with_shellexpand` | `Option<size>` |
| `convert_duration_with_shellexpand` | Durations |
| `convert_duration_with_shellexpand_and_negative` | Durations that may be negative |

<Callout type="info" title="There is no `default_as_true` helper">
  A `bool` that should default to `true` needs its own
  `fn default_true() -> bool { true }` and `#[serde(default = "default_true")]`.
  Several fields do this; there is no shared helper.
</Callout>

To rename a field without breaking existing configs, add a shim in
`backcompat.rs`; both existing shims are `#[serde(untagged)]` enums that
accept the old and new shapes and emit a `tracing::warn!` on the deprecated
one.

## Adding a metric [#adding-a-metric]

Decide which system first; see
[two metric systems](/contribute/codebase-internals#two-metric-systems).

**The derive path** touches one file. Add the field, annotate it:

```rust
#[metric(help = "Number of blobs currently resident")]
resident_blobs: AtomicU64,
```

The struct already derives `MetricsComponent`, and the value appears in the
introspection tree automatically.

**The OpenTelemetry path** touches three:

<Steps>
  <li>
    Declare the instrument in the appropriate `LazyLock` bundle in
    `nativelink-util/src/metrics.rs` (`CACHE_METRICS`, `EXECUTION_METRICS`,
    `WORKER_METRICS`, `RPC_METRICS`, `SCHEDULER_METRICS`,
    `STORE_TIER_METRICS`, `HEALTH_METRICS` or `CONNECTION_METRICS`). Use a
    dotted name, a `.with_description`, and a `.with_unit` that is a UCUM
    unit (`"By"`, `"s"`, `"ms"`) or a braced count such as `"{entry}"`.
    Then regenerate the metrics reference with
    `bun --filter @nativelink/docs gen:metrics-reference` from `web/`, which
    reads this file.
  </li>
  <li>
    Record it at the call site, using the pre-computed attribute struct
    where one exists (`CacheMetricAttrs` / `ExecutionMetricAttrs`) rather
    than building attributes inline.
  </li>
  <li>
    Extend `nativelink-util/tests/metrics_test.rs`.
  </li>
</Steps>

<Callout type="warn" title="Two things to not do here">
  `nativelink-util/src/metrics.rs` is **Business Source License**; keep its
  header, do not paste the FSL one in. And never add an action digest or a
  worker ID as an attribute on a hot instrument: the cardinality will kill
  the metrics backend long before it affects NativeLink.
</Callout>

## Adding a gRPC service [#adding-a-grpc-service]

Model: `nativelink-service/src/fetch_server.rs`: 165 lines, and it has
every convention in it.

<Steps>
  <li>
    **Write the server** at `nativelink-service/src/my_server.rs`. Import
    the generated trait and server type with the standard alias:

    ```rust
    use nativelink_proto::…::my_server::{My, MyServer as Server};
    ```

    Hold per-instance state in a `HashMap<String, _>` keyed by instance
    name. Constructor signature:

    ```rust
    pub fn new(
        configs: &[WithInstanceName<MyConfig>],
        store_manager: &StoreManager,
    ) -> Result<Self, Error>
    ```
  </li>
  <li>
    **Provide `into_service`**, exactly this name, because the
    `service_setup!` macro calls it:

    ```rust
    pub fn into_service(self) -> Server<Self> {
        Server::new(self)
    }
    ```
  </li>
  <li>
    **Split each RPC in two.** A private `inner_<rpc>` returning
    `nativelink_error::Error` holds the logic; the public
    `#[tonic::async_trait]` method wraps it, converts to `Status`, and
    carries the instrumentation:

    ```rust
    #[instrument(
        err(level = Level::WARN),
        ret(level = Level::INFO),
        skip_all,
        fields(request = ?grpc_request.get_ref())
    )]
    ```
  </li>
  <li>
    **Register it** in the `Routes::builder().routes()` chain in
    `src/bin/nativelink.rs`, with `.add_optional_service(...)` and
    `service_setup!`. The macro applies the message-size limit (4 MiB by
    default) and compression settings, so registering without it silently
    opts out of both.
  </li>
</Steps>

If the service needs new protos: add the package to `PROTO_NAMES` **and** to
the `gen_rs_protos` `srcs` list in `nativelink-proto/BUILD.bazel`, then
regenerate and commit the output:

```bash
bazel run nativelink-proto:update_protos
```

`//nativelink-proto:update_protos_test` is the CI gate that catches a
forgotten regeneration.

<Callout type="warn" title="Worker-facing services belong on their own port">
  The CAS server carries an explicit note about this: the `worker_api`
  service must not share a listener with client-facing services, because it
  has no authentication of its own. Anything you add with worker-level
  privileges inherits that requirement; see
  [security hardening](/operate/security-hardening).
</Callout>

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="My store builds with Cargo but CI fails. Why?" id="cargo-green-bazel-red">
    Almost certainly the `srcs` list in `nativelink-store/BUILD.bazel`. Cargo
    globs; Bazel enumerates. Add the file, keep the list alphabetical, and
    run `bazel build //nativelink-store:nativelink-store` before pushing.
  </Accordion>
  <Accordion title="My new config field doesn't show up in the reference." id="field-missing-from-reference">
    Check that the spec struct has
    `#[cfg_attr(feature = "dev-schema", derive(JsonSchema))]`. The generated
    reference is produced from the schemars schema, and a struct without that
    derive is invisible to it. If the field is there but has no default,
    add a `/// Default: …` line to the doc comment.
  </Accordion>
  <Accordion title="A doc comment broke the build. How?" id="doc-comment-broke-build">
    Doc comments in `nativelink-config` are load-bearing twice over: fenced
    JSON blocks are extracted into
    `nativelink-config/examples/stores-config.json5` by a pre-commit hook,
    and `tests/json5_test.rs` parses every generated example. Malformed JSON
    in a fence fails that test.
  </Accordion>
  <Accordion title="Do I need protoc installed to add a proto?" id="need-protoc">
    No. Generated Rust is checked into `nativelink-proto/genproto/`, and
    regeneration goes through `bazel run nativelink-proto:update_protos`.
    Commit the regenerated files with your change.
  </Accordion>
  <Accordion title="Where does a new store's default go?" id="where-do-store-defaults-go">
    In the implementation, not the config crate; the spec should describe
    intent, and the store applies its own fallback. Record the value in the
    field's `/// Default:` line so the reference stays honest.
  </Accordion>
</Accordions>

<NextStep href="/contribute/testing-guide" title="Testing guide">
  How the suites are organised, what `#[nativelink_test]` does for you, and
  which local environment problems produce which confusing failure.
</NextStep>

# https://docs.nativelink.com/contribute/testing-guide

**Testing guide**: How NativeLink's tests are organised, what `#[nativelink_test]` gives you, and which local environment problems cause which confusing failure.

**Who this is for:** anyone writing a test, or trying to work out why a
test that passes in CI fails on their laptop. **What you'll have at the
end:** the commands, the conventions, the fixtures that already exist, and
a diagnosis list for the local failures that are not your code's fault.
**Time:** twenty minutes.

NativeLink has two test runners over the same tests. Cargo is faster to
iterate with; Bazel is what CI runs and what gates a merge. Both matter,
and the vocabulary differs between them in a way that trips people up.

## The commands [#the-commands]

```bash
# Cargo: fastest loop. `smol` keeps target/ around 1GB instead of ~12GB.
cargo test --all --profile=smol
cargo test -p nativelink-store

# Bazel: what CI runs.
bazel test //...              # everything, plus rustfmt and clippy aspects
bazel test //:unit_tests      # in-file `mod tests` across all crates
bazel test doctests           # doc examples
bazel test //nativelink-store:integration   # one crate's tests/ directory
```

<Callout type="info" title="Bazel's `unit_test` and `integration` do not mean what you expect">
  In this repository, `unit_test` means *tests written inside a source file*
  and `integration` means *tests in the crate's `tests/` directory*. Neither
  implies anything about scope or about talking to a network. The
  genuinely end-to-end suites are the shell scripts in `integration_tests/`,
  which are separate again.
</Callout>

`bazel test` also attaches rustfmt and clippy aspects, which is why a
formatting mistake shows up as a test failure rather than a separate lint
step.

## Where tests live [#where-tests-live]

The convention is a separate `tests/` directory per crate, not inline
modules; only nine `mod tests` blocks exist in the whole tree.

| Crate | Test files under `tests/` |
| --- | --- |
| `nativelink-store` | 28 |
| `nativelink-util` | 20 |
| `nativelink-service` | 13 |
| `nativelink-scheduler` | 10 |
| `nativelink-worker` | 7 |
| `nativelink-config` | 3 |
| `nativelink-error` | 2 |

A new test file has to be added to its crate's `rust_test_suite` in
`BUILD.bazel`. Cargo picks it up automatically; Bazel does not.

## `#[nativelink_test]` [#nativelink-test]

Around 780 tests use it. It is defined in
<SourceLink file="nativelink-macro/src/lib.rs" symbol="nativelink_test" />
and it is not a thin wrapper; it expands to five things:

- `#[tokio::test]`, so the test body can be `async`
- `#[::tracing_test::traced_test]`, capturing logs for assertions
- an `error_span!` named after the test function
- `reseed_rng_for_test()`, so randomised tests are reproducible
- a **log-redaction assertion** that fails the test if any captured log line
  contains `" data: b"` (with one carve-out for an `aws_runtime` line that
  legitimately does)

That last one is the surprising one. It exists to catch blob contents
leaking into debug output. If your test starts failing with
`Non-redacted data in "..."` and you did not change any assertion, something
you touched is now logging a `Debug` of a buffer.

The forms:

```rust
#[nativelink_test]                          // the usual case
#[nativelink_test("crate")]                 // inside nativelink-util itself
#[nativelink_test(flavor = "multi_thread")] // args forwarded to tokio::test
#[nativelink_test(start_paused = true)]     // virtual clock, see below
```

## Fixtures that already exist [#fixtures-that-already-exist]

Reach for these before writing your own.

**Redis**: `nativelink-redis-tester` provides a fake Redis, a `MockPubSub`,
and a read-only Redis wrapper. Note that it is a regular dependency of
`nativelink-store`, not a dev-dependency.

**Time**: `MockInstantWrapped` plus `mock_instant::MockClock::advance` lets
you move a store's clock forward without sleeping. Combine with
`#[nativelink_test(start_paused = true)]` for Tokio's virtual time.

**Scheduler helpers**: `nativelink-scheduler/tests/utils/scheduler_utils.rs`,
pulled in with an inline `mod utils { pub(crate) mod scheduler_utils; }`
block rather than as a crate.

**Mongo**: `nativelink-store/tests/mongo_runner/` downloads and starts a
real `mongod`. This is why the store crate's Bazel timeout is `moderate`
rather than `short`, and why that suite needs network on a cold cache.

**Temp paths**: `make_temp_path` honours `TEST_TMPDIR`, so tests behave
under Bazel's sandbox.

## The end-to-end suites [#the-end-to-end-suites]

```bash
./run_integration_tests.sh
```

It refuses to run as root, requires Bazel, Docker and `sudo`, and waits for
ports 50051, 50052 and 50071 before starting. It drives
`simple_cache_test.sh`, `simple_remote_execution_test.sh`,
`simple_tls_test.sh` and `chunking_cache_test.sh`. You can pass a pattern to
run one of them.

Beyond those, `integration_tests/` also holds BuildStream, Buck2 and Mongo
suites driven from Nix, and `nativelink-test/fuzz` holds a single fuzz
target (`cas_config.rs`) run by ClusterFuzzLite.

## Coverage [#coverage]

```bash
nix build .#nativelinkCoverageForHost
```

The `result` symlink contains an HTML report. **No coverage
threshold gate**; no PR is blocked on a percentage. Coverage is a tool for
finding untested paths, not a target to hit.

## When it fails locally but not in CI [#when-it-fails-locally-but-not-in-ci]

Most of these are environment, not code.

| Symptom | Cause | Fix |
| --- | --- | --- |
| Namespace/worker tests fail on Ubuntu 24.04 | `kernel.apparmor_restrict_unprivileged_userns` is on by default | Disable it for the session, or skip those tests locally and let CI run them |
| `Too many open files` | `ulimit -n` too low for the store tests | Raise it; NativeLink itself defaults to a 24576 fd budget |
| `run_integration_tests.sh` exits immediately | Running as root, or Docker not reachable without `sudo` | Run as a non-root user with Docker access |
| The Mongo store suite times out | It downloads a real `mongod` on a cold Bazel cache | Give it network, and don't run it on a plane |
| Two tests pass alone, fail together | Both touch process environment; they are marked `#[serial(env)]` for a reason | Don't parallelise them, and mark new env-touching tests the same way |
| `Non-redacted data in "..."` | The `#[nativelink_test]` log assertion caught new `Debug` output containing a buffer | Redact the log line rather than the assertion |
| Formatting failures from `bazel test` | Stable `cargo fmt` ignores the nightly-only rustfmt options | `bazel run --config=rustfmt @rules_rust//:rustfmt` |

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Do I have to run the whole suite before submitting?" id="run-whole-suite">
    Run the crate you touched with Cargo while iterating, then
    `bazel test //...` once before pushing. CI runs Bazel, and the aspects
    attached to `bazel test` catch formatting and clippy problems that a
    Cargo run does not.
  </Accordion>
  <Accordion title="Why do doc examples need their own target?" id="why-doctests">
    Doc comments in `nativelink-config` are the source text for the generated
    [configuration reference](/reference/nativelink-config), so their examples
    are user-facing documentation. `bazel test doctests` is what stops them
    rotting.
  </Accordion>
  <Accordion title="My test needs to wait for something. Should I sleep?" id="should-i-sleep">
    No. Use `#[nativelink_test(start_paused = true)]` with Tokio's virtual
    clock, or `MockInstantWrapped` with `MockClock::advance` where a store's
    own clock is what matters. A real sleep makes the suite slower and
    flakier at the same time.
  </Accordion>
  <Accordion title="What coverage do I need for a PR to be accepted?" id="coverage-requirement">
    No numeric gate exists. What reviewers ask for is a regression test for
    a bug fix and both unit and integration coverage for new behaviour; see
    [contribution guidelines](/contribute/guidelines).
  </Accordion>
  <Accordion title="Where do I add a new test file so Bazel sees it?" id="new-test-file-bazel">
    In the crate's `BUILD.bazel`, in the `srcs` list of its `rust_test_suite`
    (named `integration`). Cargo discovers `tests/*.rs` automatically, so a
    test that runs locally and never runs in CI is usually this.
  </Accordion>
</Accordions>

<NextStep href="/contribute/release-versioning" title="Releases and versioning">
  How a version becomes a tag, what the tag triggers, and what compatibility
  NativeLink does and does not promise.
</NextStep>

# https://docs.nativelink.com/contribute/release-versioning

**Releases and versioning**: How a NativeLink version becomes a signed tag, what the tag triggers, and what compatibility the project does and does not promise.

**Who this is for:** maintainers cutting a release, and anyone who needs to
know what a version number means before pinning one. **What you'll have at
the end:** the fourteen files a version lives in, the release sequence, what
each workflow produces, and the compatibility policy, which is mostly the
absence of one. **Time:** twenty minutes.

NativeLink's release process is deliberately semi-manual. Tags are signed by
a human, and release notes are written by a human, because both are things
an automated changelog does badly. Everything downstream of the tag is
automated.

## The version lives in fourteen files [#the-version-lives-in-fourteen-files]

No single source of truth for the version exists, because the root
`Cargo.toml` declares a `[package]` version rather than a
`[workspace.package]` version, so nothing inherits it.

| File | Count |
| --- | --- |
| Root `Cargo.toml` | 1 |
| `MODULE.bazel` | 1 |
| Each crate's `Cargo.toml` | 11 |
| `nativelink-metric/nativelink-metric-macro-derive/Cargo.toml` | 1 |

<Callout type="warn" title="The nested crate is the one that gets missed">
  The macro-derive crate is nested one level deeper inside
  `nativelink-metric/` and does not match a `nativelink-*/Cargo.toml` glob.
  `CONTRIBUTING.md` calls it out by name and gives a sanity check: a standard
  release commit touches 17 files, the fourteen above plus `Cargo.lock`,
  `nativelink-test/fuzz/Cargo.lock` and `CHANGELOG.md`.
</Callout>

Several things that contain a version are **generated** and must not be
hand-edited: `CHANGELOG.md` (git-cliff, configured by `cliff.toml`),
`web/apps/docs/lib/config-versions.ts`, the per-version configuration
reference pages, and the lockfiles.

## Cutting a release [#cutting-a-release]

The authoritative sequence is the numbered list under "Creating releases" in
<SourceLink file="CONTRIBUTING.md" symbol="CONTRIBUTING.md" />.
The shape of it:

<Steps>
  <li>
    **Bump the version** in all fourteen files.
  </li>
  <li>
    **Prepend the new changelog section**, after `git fetch upstream --tags`
    so `--unreleased` means what it should:
    `git cliff --unreleased --tag=v1.x.y --prepend CHANGELOG.md`. Do not
    regenerate the whole file; that rewrites earlier entries and drops manual
    curation. Adjusting `cliff.toml` is expected when a commit lands in the
    wrong section.
  </li>
  <li>
    **Open the release PR**, titled `Release NativeLink v1.x.y`, and merge it.
  </li>
  <li>
    **Create a signed tag on the release commit**: `git tag -s v1.x.y`, with
    the tag message equal to the tag name. Use the `v<major>.<minor>.<patch>`
    form: the image workflow fires on any tag, but the config-reference
    workflow only fires on `v1.*`.
  </li>
  <li>
    **Push the tag to your fork first.** That triggers the image workflow in
    your own repository, which is the dry run. Only push to `upstream` once
    it is green.
  </li>
  <li>
    **Confirm the config reference regenerated.** Pushing the tag upstream
    triggers `config-reference.yaml`, which opens an auto-merging PR against
    `main`. If it did not fire, re-run it from the Actions tab or regenerate
    by hand with
    `bun --filter @nativelink/docs gen:config-reference v1.x.y`.
  </li>
  <li>
    **Write the release notes**, attributing every entry to its author, then
    publish. Publishing is what triggers the signed-artifacts workflow.
  </li>
  <li>
    **Verify the assets landed**: for each target, a `.tar.gz` plus its
    `.sig` and `.pem`, an `.spdx.json` SBOM, and an `.intoto.jsonl`
    provenance file.
  </li>
</Steps>

## What the tag triggers [#what-the-tag-triggers]

| Workflow | Fires on | Produces |
| --- | --- | --- |
| `tagged_image.yaml` | Any tag push | Four OCI images pushed to GHCR |
| `image.yaml` | Push to `main` (and pull requests, for the build) | A `YYYY-MM-DD-<sha>` image plus a Trivy scan |
| `config-reference.yaml` | `v1.*` tags, and weekly | Regenerated config reference, auto-merged |
| `release.yaml` | Publishing a GitHub Release | Signed binaries, SBOMs, SLSA provenance |

`release.yaml` builds three targets (`x86_64-unknown-linux-musl`,
`aarch64-unknown-linux-musl` and `aarch64-apple-darwin`), signs each with
keyless Sigstore cosign, and attaches SLSA Build Level 3 provenance.

<Callout type="info" title="Cosign is pinned to v2.5.3 on purpose">
  Cosign v3 stopped emitting the separate `.sig` and `.pem` files. The
  OpenSSF Scorecard `Signed-Releases` check reads the GitHub Releases API and
  looks for exactly those, so the pin is what keeps that check passing. The
  cosign signatures on the GHCR container images are invisible to it.
</Callout>

Verify any asset locally:

```bash
slsa-verifier verify-artifact nativelink-1.x.y-x86_64-unknown-linux-musl.tar.gz \
  --provenance-path nativelink-1.x.y.intoto.jsonl \
  --source-uri github.com/TraceMachina/nativelink \
  --source-tag v1.x.y
```

## What the version number promises [#what-the-version-number-promises]

Less than you would assume, and it is better to say so than to imply
otherwise.

**No SemVer or compatibility policy is stated anywhere in the
repository.** `MODULE.bazel` declares `compatibility_level = 0`, which is
Bazel's way of saying no compatibility guarantee has been asserted. The
security policy states only that the most recent tagged version is
supported.

In practice this means: read the release notes for breaking changes rather
than inferring from the version number; pin an exact version in production
rather than a range; and treat a minor bump as something to test, not
something to assume.

**Nothing is published to crates.io.** The crates exist for the workspace's
own use. Consume NativeLink as a binary, a container image, or a Bazel
module, not as a library dependency.

Backward-compatible config changes do get shims: see the `#[serde(untagged)]`
helpers in `nativelink-config`'s `backcompat.rs`, which accept the old shape
and emit a deprecation warning. That is a convention, not a promise.

## Regenerating the configuration reference [#regenerating-the-configuration-reference]

The reference pages under `reference/nativelink-config/` are generated from
the Rust config crate and must never be hand-edited. The chain is:
`cargo run --bin build-schema --features dev-schema` produces a JSON schema,
`web/apps/docs/scripts/gen-config-reference.mjs` and `scripts/lib/schema-to-mdx.mjs`
turn it into MDX, and the whole thing is invoked as:

```bash
cd web
bun --filter @nativelink/docs gen:config-reference v1.x.y
```

Passing a tag updates `lib/config-versions.ts` (which is what the docs UI
reads to decide the current version), rewrites the current reference from
that tag, and archives the previous one as a versioned page. Historical
versions do not need regenerating.

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Does a minor version bump mean no breaking changes?" id="does-minor-mean-no-breaking">
    It is not guaranteed to. No SemVer policy is stated, and
    `compatibility_level = 0` in `MODULE.bazel` explicitly asserts none. Read
    the release notes, which do call out breaking changes and include
    migration instructions.
  </Accordion>
  <Accordion title="Why sign tags manually instead of automating it?" id="why-manual-tags">
    A signed tag attests that a human made the release. Automating it would
    move the signing key into CI, which is exactly what the signature is
    supposed to rule out.
  </Accordion>
  <Accordion title="Can I depend on `nativelink-store` from my own crate?" id="depend-on-crates">
    Not from crates.io; nothing is published there. You can path- or
    git-depend on it, but with no compatibility policy you are pinning a
    commit, not an interface.
  </Accordion>
  <Accordion title="A release shipped without signed assets. Can it be fixed?" id="backfill-signed-assets">
    Yes. Re-run `Signed release artifacts` from the Actions tab with the tag
    as the input; it rebuilds, re-signs, and attaches the assets to the
    existing release.
  </Accordion>
  <Accordion title="Which version do the docs describe?" id="which-version-do-docs-describe">
    The configuration reference is generated per release and version-switched
    in the UI. Everything else describes the most recent tagged version. If a
    page's claim and your binary disagree, the binary wins, and the page is
    a bug worth reporting.
  </Accordion>
</Accordions>

<NextStep href="/contribute/docs" title="Working on documentation">
  The other half of a contribution: how these pages are built, what the
  components are, and the rules a docs PR is reviewed against.
</NextStep>

# https://docs.nativelink.com/contribute/docs

**Working on documentation**: How these docs are built: the four page archetypes, the components, the anchor and snippet lints, and what a docs PR is reviewed against.

**Who this is for:** anyone writing or editing a page on this site. **What
you'll have at the end:** the local dev loop, the archetype your page has to
satisfy, the components you can use without importing them, and the lints
that will fail your PR if you skip them. **Time:** thirty minutes, once.

The docs are an MDX corpus under
<SourceLink file="web/apps/docs/content/docs" dir symbol="content/docs/" />,
rendered by Fumadocs on Next.js. Frontmatter at the top, Markdown plus JSX
below. The only build steps are the two generators the dev server runs on
start (`gen:changelog` and `gen:llms`).

Two things make this site different from most docs sites, and both are
enforced rather than encouraged: **every page declares an archetype and is
reviewed against that archetype's contract**, and **every heading carries an
explicit anchor** so that a link cited by a human or cached by an agent keeps
resolving after the heading is reworded.

## The local loop [#the-local-loop]

```bash
cd web
bun install
bun dev:docs        # docs only: http://localhost:3001/docs
```

Edit any `.mdx` file under `apps/docs/content/docs/` and the dev server picks
it up on save. To preview the marketing site and the docs together (the
marketing app rewrites `/docs` to the docs app in dev), run `bun dev` and use
`http://localhost:3000/docs`.

<Callout type="info" title="`web/` is not part of the Bazel build">
  The docs are a Bun workspace. `bazel test //...` does not touch them, and
  `bun` is the only toolchain you need to work on a page. See
  [contribution guidelines](/contribute/guidelines) for the rest of the repo.
</Callout>

## Pick the archetype first [#pick-the-archetype-first]

Every page is exactly one of four kinds, and the kind determines the shape.
Mixing two is the most common structural review comment, because a tutorial
that pauses to explain tradeoffs stops being a guaranteed happy path, and an
explanation that turns into a procedure stops being readable out of order.

| Archetype | Answers | Shape |
| --- | --- | --- |
| **Tutorial** | "Walk me through it" | One guaranteed happy path. Numbered steps, each a command and its expected output. No options. |
| **How-to** | "I need to do this specific thing" | The complete working artifact first, then field-by-field on the parts that matter, then troubleshooting. |
| **Explanation** | "Why is it shaped this way?" | The mental model in three to five sentences, a diagram, then the tradeoffs. Facts alone are reference. |
| **Reference** | "What exactly does this field do?" | Exhaustive, one anchor per entry, no narrative. Generated where possible. |

The templates live in `web/apps/docs/templates/`. Copy the one you need,
and keep the comment it puts directly under the frontmatter:

```mdx

```

That comment is not decoration. It tells a reviewer, and an agent
generating a page, which contract to check the page against.

## The opening block every page has [#the-opening-block]

Directly under the archetype comment, before anything else:

```mdx
**Who this is for:** the reader this page assumes. **What you'll have at the
end:** the concrete outcome. **Time:** an honest estimate.
```

It exists so a reader who landed here from a search result can decide in five
seconds whether to keep reading. "Time" is a real estimate; rounding it down
to look welcoming is how a page loses trust in its first paragraph.

Pages on the reading path state what they assume on the next line:

```mdx
<Prerequisites>
  A cache serving hits to your build. If you don't have one yet, start with
  [Getting started](/getting-started).
</Prerequisites>
```

<Callout type="warn" title="`<Prerequisites>` belongs on reading-path pages only">
  The reading path is Getting started, Remote execution, Configuration and
  How-to guides, with Why NativeLink before it and Operate after it. Pages
  under `explanations/`, `reference/` and `contribute/` are read in any order
  and must not state prerequisites, because that block is also what tells a
  reader the corpus has an ordering.
</Callout>

## Frontmatter and navigation [#frontmatter-and-navigation]

Two required fields, one optional:

```mdx
---
title: My page
description: One sentence, written for a search result rather than for a reader who already opened the page.
---
```

`full: true` removes the right-hand table of contents. Use it on landing
pages and wide generated references, not on ordinary pages.

Sidebar order comes from each section's `meta.json`:

```json
{
  "title": "Section name",
  "pages": ["first", "second", "third"]
}
```

The top-level `content/docs/meta.json` orders the sections themselves; each
section's `meta.json` gives it the title the sidebar shows. Do not list
`index` in `pages`: a section's `index.mdx` becomes the link on the section
title itself, and listing it would show the same page twice.

<Callout type="warn" title="Three sections have no index page">
  `explanations/`, `reference/` and `contribute/` are groupings, not
  landing pages. Linking to `/explanations` or `/contribute` produces a 404;
  link to a specific page inside them instead.
</Callout>

## Components [#components]

All of these are provided globally. **Never write an import in an `.mdx`
file**; an import statement is the usual reason a page compiles locally and
fails in the MDX check.

| Component | Use |
| --- | --- |
| `<Callout type="info\|warn\|success\|error" title="…">` | An aside worth interrupting for |
| `<Steps>` with `<li>` children | An ordered procedure |
| `<Tabs items={["A","B"]}>` with `<Tab value="A">` | Alternatives the reader picks between |
| `<Accordions type="multiple">` with `<Accordion title="…" id="…">` | The per-page FAQ |
| `<Mermaid>` | Diagrams |
| `<SourceLink file="…" symbol="…" />` | A permalink into the source at the pinned ref |
| `<MinVersion v="1.6.0" />` | An inline badge for something that needs a minimum version |
| `<VerifyBlock>` | The "you did it right if…" box that closes a tutorial |
| `<Prerequisites>` | What a reading-path page assumes; see above |
| `<NextStep href="…" title="…">` | The handoff at the bottom of a reading-path page; `kind="aside"` marks an optional detour rather than the path |

`<ConfigVersionSwitcher>` is also registered, but only the generated
configuration reference uses it.

Two syntax details that catch people:

**`<SourceLink>` takes no children.** It is always self-closing, and the
label is the `symbol` prop. Pass `dir` instead of `file` to link a directory.

```mdx
<SourceLink file="nativelink-util/src/store_trait.rs" symbol="StoreDriver" />
```

**`<Mermaid>` takes a template literal**, and literal angle brackets inside
node text break the MDX parse; write `&lt;` and `&gt;`, or reword.

```mdx
<Mermaid>{`graph LR
  a["Client"] --> b["CAS"]
`}</Mermaid>
```

## Anchors are explicit, and a lint enforces it [#anchors-are-explicit]

Fumadocs derives a heading's id from its text. That is convenient and it is
exactly the problem: the id is a function of the prose, so rewording a
heading silently repoints every link that cited it. A human notices a dead
in-page link and scrolls. An agent follows the cached URL, lands at the top
of the page, and quotes whatever is there.

So every heading carries its id, and every FAQ entry carries an `id`:

```mdx
## The three store traits [#the-three-store-traits]

<Accordion title="Why can't I use `tokio::spawn`?" id="why-not-tokio-spawn">
```

Once written, an anchor is a URL. Reword the heading freely; do not change
the anchor without adding a redirect.

```bash
cd web
bun --filter @nativelink/docs lint:anchors          # check
bun --filter @nativelink/docs lint:anchors --fix    # freeze what exists
```

`--fix` writes the anchor Fumadocs would have generated anyway, so running it
never moves an existing link; it only pins it. Generated pages are skipped.

## Snippets are checked against the real schema [#snippets-are-checked]

The configuration reference cannot drift from the binary, because it is
generated from the same Rust types the binary deserializes. Prose pages can:
a how-to showing a renamed field keeps showing it, and nothing fails until a
reader copies it and the binary answers `unknown field`.

`lint:snippets` closes that gap. It harvests every field name out of the
generated reference and checks every `json`/`json5` fence under
`content/docs/` against it.

```bash
bun --filter @nativelink/docs lint:snippets
```

For a fence that legitimately is not NativeLink config (a `meta.json`, a
Bazel lockfile, an API response), put the escape hatch on the line before it:

```mdx

```

Keys nested under a free-form map (`properties`, `platform_properties`,
`env`, …) are skipped automatically, since those names are the operator's to
choose.

## Generated pages: do not edit [#generated-pages]

| Surface | Generated by |
| --- | --- |
| `reference/nativelink-config/` | `gen:config-reference` from the Rust config crate |
| `reference/metrics` | `gen:metrics-reference` from `nativelink-util/src/metrics.rs` and its call sites |
| `reference/changelog` | `gen:changelog` from `CHANGELOG.md` |
| `public/llms.txt`, `public/llms-full.txt` | `gen:llms` from `meta.json` plus page frontmatter |

Each MDX page among them opens with an `AUTOGENERATED` comment, which is
also how the anchor lint knows to skip it. Fix the generator or the source doc comment; an edit to the
output is overwritten on the next run.

`gen:llms` is the one to remember when you add a page: it builds the machine
reader's index from `meta.json` and frontmatter, so a page missing from a
`meta.json` is invisible to agents even if it renders fine.

```bash
bun --filter @nativelink/docs gen:llms
```

## Style [#style]

Vale (`.vale.ini`, error level only) runs over the MDX in pre-commit and in
the `Vale` CI check; `typos` excludes `web/`. The house rules on top of that
are short:

**Write prose, not bullet fragments.** A bulleted list of six three-word
items is a table that has not admitted it yet. Use a real table, or write
sentences.

**One idea per paragraph**, and lead with the claim rather than building up
to it.

**Every code block gets a language tag.** Markdown needs it for
highlighting, and the snippet lint needs it to find the fence.

**No em-dashes or en-dashes.** Use a comma, a colon, parentheses, or two
sentences instead.

**Say the surprising thing plainly.** If a default is wrong for most people,
if a name points at the wrong crate, if a policy does not exist, write that
down. A page that reads as a brochure is a page that stops being consulted.

**Link the canonical page rather than restating it.** Field-level truth lives
in the generated reference; a reading-path page states intent and tradeoffs
and links out for the types and defaults.

## Before you open the PR [#before-you-open-the-pr]

<Steps>
  <li>
    **Run the lints** from `web/`. These are what the `Docs lint` workflow
    runs on every PR that touches `web/apps/docs/`, and `gen:llms` must
    leave the tree unchanged:

    ```bash
    bun --filter @nativelink/docs lint:anchors
    bun --filter @nativelink/docs lint:links
    bun --filter @nativelink/docs lint:snippets
    bun --filter @nativelink/docs lint:boundaries
    bun --filter @nativelink/docs lint:navigation
    bun --filter @nativelink/docs gen:llms
    ```
  </li>
  <li>
    **Check that it compiles and typechecks.** `bun --filter @nativelink/docs build`
    catches an MDX parse error or a stray import that the dev server tolerated.
  </li>
  <li>
    **Run pre-commit** over what you changed,
    `pre-commit run --files <paths>`, for Vale.
  </li>
  <li>
    **Attach a screenshot** of the rendered page. Reviewers read the MDX, but
    a broken table or an unrendered component only shows up in the render.
  </li>
</Steps>

Then the normal flow: signed commits, an imperative title with no trailing
period, a force-pushed amend rather than a follow-up commit. See
[contribution guidelines](/contribute/guidelines).

## Common questions [#common-questions]

<Accordions type="multiple">
  <Accordion title="Where do FAQs go?" id="where-do-faqs-go">
    At the bottom of the page they belong to, under a `## Common questions`
    heading, in an `<Accordions type="multiple">` block. No
    standalone FAQ page exists, deliberately: a question is only useful next to the
    material that raised it. Every `<Accordion>` needs an `id`.
  </Accordion>
  <Accordion title="My page renders in dev but the build fails." id="renders-in-dev-build-fails">
    Three usual causes, in order of likelihood: an `import` statement in the
    MDX (components are global; remove it), a literal `<` or `>` inside
    Mermaid node text, or children passed to `<SourceLink>`, which takes none.
  </Accordion>
  <Accordion title="Do I have to add an anchor to every heading?" id="anchor-every-heading">
    Yes, on every `##` and `###`, and an `id` on every `<Accordion>`. Run
    `lint:anchors --fix` and it writes the ids Fumadocs was already
    generating, so nothing moves; it just stops the ids from tracking the
    wording.
  </Accordion>
  <Accordion title="I need to move or rename a page." id="moving-a-page">
    Add the redirect in `next.config.mjs` in the same PR, update the section's
    `meta.json`, and grep the corpus for the old path. A moved page with no
    redirect breaks every external link and every agent that cached the URL,
    which is most of the reason the old URLs are still routable.
  </Accordion>
  <Accordion title="Can I add a new component?" id="adding-a-component">
    Yes, in `web/apps/docs/components/`, registered in the global MDX
    components map so pages do not import it. Prefer reusing what is there:
    a component used on one page is a styling choice, and styling choices
    belong in CSS.
  </Accordion>
  <Accordion title="Which pages need a `<NextStep>`?" id="which-pages-need-nextstep">
    Every reading-path page: Getting started through How-to guides, Why
    NativeLink and Operate, and the explanation and contribute chains. Reference pages are lookup surfaces and end where they
    end. A reading-path page with no handoff is a dead end, and dead ends are
    where readers leave.
  </Accordion>
</Accordions>

<NextStep href="/contribute/guidelines" title="Contribution guidelines">
  The rest of the contribution flow: forks, signed commits, commit message
  conventions, and how review works.
</NextStep>

# https://docs.nativelink.com/

**For agents**: How an AI agent should read these docs end to end, where every entry point is, how to cite and verify a claim, and what the docs guarantee to a machine reader.

**Who this is for:** an AI agent (Claude Code, Cursor, Copilot, a homegrown
tool) that has been pointed at NativeLink, and the person pointing it.
**What you'll have at the end:** the entry points, a procedure for reading
the whole corpus in order, the conventions that make pages citable and
verifiable, and a task index into the rest of the docs. **Time:** five
minutes to read, one fetch to load everything.

## Read this first if you are an agent [#read-this-first-if-you-are-an-agent]

1. Fetch **[`/llms-full.txt`](/llms-full.txt)**. It is the entire documentation
   corpus, every page body verbatim, in reading order, in one plain-text file
   (under 1 MB). One fetch gives you everything a human would find by
   clicking through the sidebar.
2. If you only want the map, fetch **[`/llms.txt`](/llms.txt)** instead: one
   line per page with its canonical URL and a one-sentence description, in
   the same order.
3. Each page in `llms-full.txt` starts with a line of the form
   `# https://docs.nativelink.com/<path>`. That is the page's canonical URL.
   Cite it, and cite headings with the explicit `#anchor` you see in the
   source (`## Heading [#anchor]`), because those anchors never move.
4. Three pages are generated from the source code and are linked rather than
   inlined, because they are long and change on a different cadence. Fetch
   them directly when you need them:
   - [`/reference/nativelink-config`](/reference/nativelink-config): every
     configuration field, its type and default, for the latest release;
     older releases live at `/reference/nativelink-config/v1.x.y` and the
     development branch at `/reference/nativelink-config/main`.
   - [`/reference/metrics`](/reference/metrics): every OpenTelemetry
     instrument, its attributes, and whether the binary actually emits it.
   - [`/reference/changelog`](/reference/changelog): the release history.
5. When a page states how the binary behaves, look for a source link: in the
   page source it is `<SourceLink file="path/in/repo" symbol="name" />`, and
   in rendered HTML an `<a data-source-link="path/in/repo" data-source-ref="vX.Y.Z">`.
   It resolves to `https://github.com/TraceMachina/nativelink/blob/<ref>/<path>`.
   Follow it to check the claim against the code at that tag before you rely
   on it.
6. The current release these docs describe is the `data-source-ref` value,
   and the config reference and metrics reference both name the version and
   commit they were generated from in a comment at the top of the page.

Nothing here requires an account, an API key, or JavaScript: every URL above
serves plain text or static HTML.

## The three entry points [#the-three-entry-points]

**`AGENTS.md` at the repo root** <SourceLink file="AGENTS.md" /> is for an
agent working *on the code*. It maps every `nativelink-*` crate to what it
owns, says where config, store, scheduler and worker logic actually live,
gives the build and test commands, and carries a table of "you changed X, so
this doc has to follow" pairs that are often missed.

**[`/llms.txt`](/llms.txt)** is for an agent working *with the docs*. It
follows the [llms.txt convention](https://llmstxt.org/): a link-first index of
every published page with a one-line description, in reading order. It is
generated from the navigation and page frontmatter by
<SourceLink file="web/apps/docs/scripts/gen-llms.mjs" />, runs as part of
`dev` and `build`, and is gitignored, so it cannot be edited into
disagreement with the sidebar, because there is no committed copy to edit.

**[`/llms-full.txt`](/llms-full.txt)** is the same corpus with bodies
included, concatenated in the same order, for agents that would rather fetch
once than crawl every page. Page bodies are reproduced verbatim, MDX
components included, so nothing is lost in transcription.

## How the corpus is ordered [#how-the-corpus-is-ordered]

The sidebar, `llms.txt` and `llms-full.txt` all use the same order, defined
once in the `meta.json` files under `content/docs/`:

| Section | What it is | Read it when |
| --- | --- | --- |
| [Why NativeLink](/use-cases) | What a remote cache and remote execution do, and who benefits | Deciding whether to use it at all |
| [Getting started](/getting-started) | A cache running, connected to a build, proven to be hitting | First contact |
| [Remote execution](/remote-execution) | Workers running the actions the cache misses | After the cache works |
| [Configuration](/configuration) | The config model: stores, servers, services, scheduler, workers | Writing a config rather than copying one |
| [How-to guides](/how-to) | Task recipes: every storage backend, TLS, multiple workers, upgrades, migration | A specific task |
| [Operate](/operate) | Production shape, deployment, observability, scaling, runbooks, troubleshooting | Running it for other people |
| Concepts (`/explanations/...`) | How it is built: architecture, stores, scheduler, worker, correctness, LRE, history | Understanding a behaviour |
| [Reference](/reference/nativelink-config) | Generated config and metrics references, CLI and environment, protocol, glossary, licence, changelog | Looking something up |
| Contribute (`/contribute/...`) | Building, testing, the crate map, extending, releasing, working on these docs | Changing the code or the docs |

Pages in the first six sections open with a `<Prerequisites>` block
(`data-prerequisites` in HTML) stating what they assume you already have, and
end with one primary `<NextStep>` (`data-next-step="next"`) naming the page
that follows; detours are `data-next-step="aside"`. Concepts, Reference and
Contribute carry no position: enter them from a search or a link.

## Task index [#task-index]

The shortest path from a question to the page that answers it.

| If the task is | Read |
| --- | --- |
| Run NativeLink for the first time | [Quickstart](/getting-started/quickstart), then [Verify your cache](/getting-started/verify-your-cache) |
| Point Bazel, Buck2, Siso, Chromium, Pants, BuildStream or CMake at it | [Connect your build](/getting-started/connect-your-build) |
| Share one cache across a team | [Shared cache](/getting-started/shared-cache) |
| Add workers and run actions remotely | [Remote execution](/remote-execution), [Your first remote action](/remote-execution/first-remote-action) |
| Route actions to the right workers | [Platform properties](/remote-execution/platform-properties) |
| Hermetic toolchains, containers, persistent workers | [Toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity), [Containers and images](/remote-execution/containers-and-images), [Persistent workers](/remote-execution/persistent-workers) |
| Understand or write a config file | [Configuration](/configuration), then the [generated reference](/reference/nativelink-config) |
| Pick and configure a storage backend | [Storage backends](/how-to/stores) |
| Compose stores (tiers, verification, dedup, compression, sharding) | [Compose stores](/how-to/stores/compose-stores), [Store model](/explanations/store-model) |
| TLS, mTLS, identity headers | [TLS and auth](/how-to/tls-and-auth) |
| Upgrade or migrate | [Upgrade versions](/how-to/upgrade-versions), [Migrate an existing cache](/how-to/migrate-an-existing-cache) |
| Production shape, deployment, security | [Production configuration](/operate/production-config), [Docker Compose](/operate/deploy-docker-compose), [Bare metal](/operate/deploy-bare-metal), [Security hardening](/operate/security-hardening) |
| Metrics, dashboards, alerts | [Observability](/operate/observability), [Metrics reference](/reference/metrics) |
| Scale workers or the CAS | [Scaling workers](/operate/scaling-workers), [Scaling the CAS](/operate/scaling-cas), [Autoscaling reference](/operate/autoscaling-reference) |
| Something is wrong | [Troubleshooting](/operate/troubleshooting), [Runbooks](/operate/runbooks) |
| Every flag, environment variable and CLI argument | [CLI and environment](/reference/cli-and-env) |
| Which RPCs and protocols are implemented | [Protocol and API](/reference/protocol-api) |
| What is open source and what is licensed differently | [Open source and Enterprise](/reference/oss-and-enterprise) |
| How the scheduler, worker and stores work internally | [Architecture](/explanations/architecture), [Scheduler internals](/explanations/scheduler-internals), [Worker execution](/explanations/worker-execution) |
| Build, test or change the code | [Contribution guidelines](/contribute/guidelines), [Repository and crate map](/contribute/repo-crate-map), [Codebase internals](/contribute/codebase-internals), [How to extend](/contribute/how-to-extend) |
| Change these docs | [Working on the documentation](/contribute/docs) |

## What these docs guarantee to a machine reader [#what-these-docs-guarantee-to-a-machine-reader]

Each of these is enforced by something in the repository, not by an author
remembering. Where it is a script, the script is named.

**One canonical page per concept.** If two pages would explain the same
thing, one of them links the other. An agent that finds an answer has found
*the* answer.

**Stable anchors.** Every heading and every FAQ entry carries an explicit
`[#anchor]` or `id`, so a citation to a specific claim keeps resolving after
the prose around it is reworded.
<SourceLink file="web/apps/docs/scripts/lint-anchors.mjs" />
fails the check if one is missing, and its `--fix` writes the anchor Fumadocs
would have derived from the heading text anyway, so pinning an anchor never
moves a link that already worked.

**Redirects on every moved URL.** This site's structure changed
substantially; agents hold cached URLs longer than humans hold bookmarks, so
old paths redirect rather than 404. They live in
<SourceLink file="web/apps/docs/next.config.mjs" />. Follow redirects rather
than treating a 301 as a failure.

**Doc-to-source permalinks.** Behavioural claims carry a link to the code
that implements them, pinned to a release tag rather than to `main`. A
permalink that drifts is worse than no link, because the reader can't tell.
The pinned ref is a single constant in
<SourceLink file="web/apps/docs/lib/source-ref.ts" />.

**Generated reference.** The configuration reference, the metrics reference
and the changelog are generated from the source of truth in the repo rather
than transcribed from it. An agent reading the docs and an agent reading the
code get the same answer by construction. Config snippets in narrative pages
are checked against that generated reference by
<SourceLink file="web/apps/docs/scripts/lint-snippets.mjs" />, so a snippet
cannot quietly reference a field that no longer exists.

**Every internal link resolves, and every page on the path has a next step.**
<SourceLink file="web/apps/docs/scripts/lint-links.mjs" /> and
<SourceLink file="web/apps/docs/scripts/lint-navigation.mjs" /> run on every
change, so a crawl never hits a dead internal link or a dead end.

**Machine-detectable structure.** Verification blocks carry
`data-verify-block`; prerequisite blocks carry `data-prerequisites`; next
steps carry `data-next-step`; version badges carry `data-min-version`; source
links carry `data-source-link` and `data-source-ref`. Ordering and
provability are readable without parsing prose.

## What is not guaranteed [#what-is-not-guaranteed]

Being specific about the edges is more useful than a blanket claim.

**Prose is not generated.** Everything outside the generated reference is
written by a human and checked against the source at the pinned release, not
by a script. A behavioural claim carrying a `<SourceLink>` has been checked
against that file at the pinned tag; a claim without one has not been
mechanically checked against anything.

**The pinned ref lags releases.** `<SourceLink>` targets a release tag, so
immediately after a release lands the links point at the previous one until
the reference is regenerated. That is the deliberate trade: a stale-but-honest
permalink over a `main` link that silently means something different next
week.

**Third-party tools are described from their own documentation.** Bazel,
Buck2, Siso, Pants, BuildStream and recc flags are taken from those projects'
docs at the time of writing; NativeLink's source cannot vouch for them.

<NextStep href="/getting-started" title="Getting started">
  If you're here to actually run NativeLink rather than to point something
  else at it, Getting started is the start of the reading path.
</NextStep>