# NativeLink: abridged documentation corpus > A remote build cache and remote execution service in one Rust binary, for > build systems that speak the Remote Execution API: Bazel, Buck2, Siso, > Pants, BuildStream, and CMake via recc. The cache, the scheduler and the > worker are the same executable started with different configuration. These docs are ordered as a path, not a catalogue: get a cache serving hits, then put workers behind the misses, then own the config file, then reach for the how-to guides. Each section assumes the previous one is working, and every page on the path opens by stating what it assumes. An agent reading this file in order learns the same sequence a human learns from the sidebar. Getting started, then Remote execution, then Configuration, then How-to guides. Why NativeLink comes before that path, and Operate comes after it. Concepts, Reference and Contribute are outside the ordering: enter them from a search or a link at any point. This file is generated from the navigation (`meta.json`) and page frontmatter by `web/apps/docs/scripts/gen-llms.mjs`, so it cannot drift from the sidebar. Every page below is delimited by a heading of the form `# https://docs.nativelink.com/`, giving that page's canonical URL. This is the abridged corpus: each page keeps its headings and the prose that opens each section, and drops code samples, tables, components and navigation. It is meant to fit in a context window alongside the question being asked. When a section here is the one you need, fetch its page, or the whole corpus at /llms-full.txt. # https://docs.nativelink.com/ **Introduction**: NativeLink is a remote build cache and execution service in one Rust binary. This is the documentation for running it yourself. NativeLink is a **remote build cache** and a **remote execution** service in one program. Your build tool asks it "have you already built this?" and, when the answer is no, "can you build it for me over there?" Your source code doesn't change and your build files don't change: you point your build tool at a URL, and work that used to happen on your machine stops happening on your machine. ## The reading path The first four sections are ordered, and each one assumes the last. You can skip ahead, but the pages will tell you what they expected you to have. ## Start where you are ### I'm deciding whether this is for us - [Why NativeLink](/use-cases): the mental model, and the 30-second test for whether your build benefits. - [AI coding agents](/use-cases/ai-coding-agents): why an agent's build loop makes cache hit rate a token-cost question. - [Silicon and large builds](/use-cases/silicon-and-large-builds): EDA flows, large C++ and LLVM trees, ML graphs. ### I want to get something running - [Quickstart](/getting-started/quickstart): a local cache in under ten minutes. - [Connect your build tool](/getting-started/connect-your-build): Buck2, Siso, Chromium, Pants, BuildStream, CMake. - [Shared cache](/getting-started/shared-cache): the single-server deployment that survives a team-wide rollout. ### I want to configure it - [The config model](/configuration): how a NativeLink config is put together, in one page. - [Stores](/configuration/stores): declaring, naming and composing the storage a cache is built from. - [Scheduler and workers](/configuration/scheduler-and-workers): how work gets matched to machines. ### I want to run it in production - [Production configuration](/operate/production-config): the three-process split a real cluster runs in. - [Deploy with Docker Compose](/operate/deploy-docker-compose) or [on bare metal](/operate/deploy-bare-metal): the two shapes, and where state lives in each. ### I want to understand how it works - [Architecture](/explanations/architecture): scheduler, CAS, action cache, workers, and what each is responsible for. - [Architecture deep dive](/explanations/architecture-deep-dive): the same system at the level of crates, startup order, and where an action's path forks. ### I need to look something up - [Configuration reference](/reference/nativelink-config): every field, generated from the source. - [Protocol and API surface](/reference/protocol-api): every gRPC service, which RPCs are implemented, and what `Capabilities` advertises. - [Metrics](/reference/metrics): every instrument, its type, and whether it's actually emitted. ### I want to contribute - [Contribution guidelines](/contribute/guidelines): what gets accepted and how review works. - Build from source with [Bazel](/contribute/bazel), [Cargo](/contribute/cargo), or [Nix](/contribute/nix). - [Writing docs](/contribute/docs): the archetypes and conventions this site is built on. ## FAQ Every page here ends with an FAQ for its own topic. These are the questions we get most overall. # https://docs.nativelink.com/use-cases **Why NativeLink**: What NativeLink is, what a remote cache and remote execution actually do, and a 30-second check for whether your build is one that benefits. **Who this is for:** anyone deciding whether to put a remote cache and execution layer in front of their build, including people meeting the idea for the first time. **What you'll have at the end:** a working mental model of what NativeLink does, and a clear yes or no for your own situation. **Time:** about ten minutes. ## What NativeLink is NativeLink is a **remote build cache** and a **remote execution** service in one program. Your build tool asks it "have you already built this?" and, when the answer is no, "can you build it for me over there?" Nothing about your source code changes. You point your build tool at a URL, and work that used to happen on your machine stops happening on your machine. ## The problem it exists to solve Every build system computes the same thing twice. Once on your laptop, once in CI. Once for you, once for the colleague who pulled the same commit. Once before lunch, once after you reverted the change that broke it. The work is deterministic, the inputs are identical, and the machine does it again anyway. ## What a remote cache is A build is a graph of actions. Each action has inputs (source files, compiler flags, the compiler itself) and produces outputs. A **remote cache** stores those outputs on a server keyed by a hash of all of the inputs. ## What remote execution is Caching only helps with work that has already been done once. Remote execution is what happens on the miss. ## Why NativeLink specifically The protocol is open and there are other implementations. What's particular here: ## Is this for you? You will get something out of NativeLink if any of these are true: ## The three cases worth their own page Most people arrive with one of three problems. Each of these is the long-form version of a bullet above; read the one that's yours and skip the others. ## Common questions Same instinct, different scope. `ccache` caches compiler invocations on one machine, and `sccache` extends that to a shared bucket. NativeLink caches every action in the build graph rather than just compiles, and it can also *run* the actions that miss. If a shared compile cache is all you need, you may not need this. # https://docs.nativelink.com/use-cases/ai-coding-agents **AI coding agents**: An agent's edit-build-test loop runs far more often than a human's, so build time sets both its wall clock and a large part of its token bill, and a shared cache turns most of those iterations into downloads. **Who this is for:** anyone running coding agents against a codebase big enough that building it is not instant. **What you'll have at the end:** a clear model of why build infrastructure is agent infrastructure, and what specifically to put in place. **Time:** about ten minutes. ## The loop is the unit of work A coding agent does not write a patch and stop. It edits, builds, reads the error, edits again, builds again, and repeats until the tests pass. That loop is the agent's entire mechanism for finding out whether it was right. ## Why agents hit caches more than people do Here is the part that is rarely noticed: agents are unusually good cache clients, and the reason is that most of what they do is repetitive in exactly the way content-addressing rewards. ## Determinism is the other half Content-addressing gives agents something beyond speed, and it may matter more. ## The token argument Wall clock is the visible cost. The token bill is the hidden one, and for teams running agents at scale it is frequently the larger. ## What to actually put in place In rough order of return: ## Common questions No: that removes the main benefit. Cache entries are keyed by the hash of their inputs, so two agents cannot corrupt each other's results; the worst case is that they store separate entries for genuinely different work. Isolation you might want for other reasons is a network and access-control question, not a cache-partitioning one. # https://docs.nativelink.com/use-cases/autonomous-operation **Autonomous operation**: What it takes for a build farm to grow and shrink with demand without anyone watching it: which properties NativeLink gives you, which pieces you assemble yourself, and where the seams are. **Who this is for:** whoever would otherwise be paged about the build farm. **What you'll have at the end:** an accurate picture of how close NativeLink gets you to a hands-off farm, and exactly which parts you build. **Time:** about fifteen minutes. ## The shape of the problem Build load is spiky in a way that punishes static capacity. It is near zero at night, ragged through the morning, and vicious in the hour before a release. A farm sized for the peak is idle most of the time and expensive all of the time. A farm sized for the average has a queue during every moment anybody cares about. ## Why workers can be treated as disposable Autoscaling only works if adding and removing a machine is uneventful. Three properties of the worker make it so. ## Draining, for the shutdown you know about Retries handle the machines that die without warning. Machines that give you notice deserve better, and there's a specific hook for it. ## What the scheduler remembers, and what it forgets This is the constraint that shapes a hands-off design, so it's worth being precise. ## The signal you scale on An autoscaler needs a number that says "more work is waiting than there is capacity for." That number is queue depth: actions sitting in the `queued` stage rather than `executing`. ## What the repository ships, and what it doesn't Being direct, because this is the part most likely to be assumed: ## What "no human in the loop" actually means A realistic end state, in the order the pieces earn their keep: ## Common questions More carefully. Workers are stateless and cheap to churn; the CAS holds the data everything else is asking for. Stateless CAS replicas in front of one shared backing store scale on request rate, but a local filesystem tier is warm state that a new replica does not have, so scaling in throws away cache locality. Scale it slowly, and prefer bigger replicas over more of them. # https://docs.nativelink.com/use-cases/silicon-and-large-builds **Silicon and large builds**: EDA flows, large C++ and LLVM trees, and ML graphs are the workloads remote execution was invented for: thousands of actions, hardware that differs from action to action, and a long tail that decides the wall clock. **Who this is for:** anyone whose build is measured in thousands of actions and tens of minutes, on hardware that isn't uniform. **What you'll have at the end:** an understanding of why these builds are the hard case, and how NativeLink routes actions to the machines that suit them. **Time:** about fifteen minutes. ## What makes a build "large" in the way that matters Size in gigabytes is not the interesting axis. Three properties are, and a build that has all three is the shape remote execution was designed around. ## Why caching alone runs out On a small build, the cache is most of the story. On a build like this it is still enormously valuable (it is still the thing you deploy first, and Getting started is still where to start), but it stops being sufficient, for a specific reason. ## Routing actions to the hardware that fits The mechanism is **platform properties**: key-value pairs an action declares as requirements, matched against properties a worker declares about itself. The scheduler will not assign an action to a worker that does not satisfy them. ## Special hardware, and what NativeLink actually knows about it NativeLink has no GPU support, and that phrasing is deliberate rather than a shortcoming. No GPU-aware code exists anywhere in the codebase: no CUDA integration, no device discovery, no special-cased scheduling. GPUs are routed the same way a specific kernel version or a licensed tool is: as plain platform properties that you name and the scheduler matches. ## Running a heterogeneous fleet The pool is not one shape of machine. It's several, each a worker process with its own configuration, all registering with the same scheduler. ## Toolchains are the hard part, not the scheduling For this class of build the routing is the straightforward half. The half that takes the time is making the actions hermetic enough to run somewhere else at all. ## What a build like this looks like when it works One published account is worth reading rather than paraphrasing: [Reid Kleckner's write-up](https://reidkleckner.dev/posts/llvm-recc-nativelink/) describes building LLVM with `recc` executing remotely against NativeLink and taking a build from roughly seventeen minutes to roughly four. ## Common questions Partly. `recc` puts a CMake-based build on the remote cache by intercepting compiler invocations, and it's the documented path for large C++ trees. The setup in these docs is cache-only (`RECC_CACHE_ONLY=1`, so misses compile locally); recc can also forward compiles to remote workers once you have them, one compiler invocation at a time. # https://docs.nativelink.com/getting-started **Getting started**: The first section of the reading path: a cache running, connected to your build, proven to be hitting. **Who this is for:** anyone who has decided to try NativeLink and wants the shortest path to a real result. **What you'll have at the end of this section:** a cache your build system is talking to, hits you can point at, and, if you want it, the same cache shared with your team. **Time:** ten minutes for the first win, an afternoon for the shared one. ## Why caching first NativeLink does two things: it caches build outputs, and it executes build actions remotely. It is tempting to reach for the second one immediately, because it's the more impressive half. Do the first one first anyway. ## Start here, or skip ahead **Start here** if NativeLink isn't running yet. Take [Quickstart](/getting-started/quickstart) in order; it's a guaranteed happy path and every step tells you what output to expect. ## The order to read them in 1. **[Quickstart](/getting-started/quickstart)**: install NativeLink and get a cache answering on localhost. 2. **[Verify your cache](/getting-started/verify-your-cache)**: build twice and read the count of remote cache hits. 3. **[Connect your build tool](/getting-started/connect-your-build)**: the flags for Buck2, Siso, Pants, BuildStream and recc. 4. # https://docs.nativelink.com/getting-started/quickstart **Quickstart**: Install NativeLink and point your build system at it in under 10 minutes. This guide takes you from zero to your first cached build. Pick the installation method that matches your platform, then wire your build system to the running cluster. ## Install NativeLink The fastest way is the prebuilt container image. It runs anywhere Docker runs; the configuration it needs is a single JSON5 file from the repository. ## Verify it's running In another terminal: ## Point your build system at it Add to your `.bazelrc`: ## Confirm cache hits Re-run your build a second time without changing any source files. Every action should land in the cache and report a hit. With Bazel: ## Troubleshooting The NativeLink server isn't listening on the port your client is pointing at. Confirm with `lsof -i :50051` (macOS / Linux) or `netstat -ano | findstr 50051` (Windows). Check the Docker container is still running with `docker ps`. ## FAQ Every action's inputs (source digests, the compiler binary, the exact command line, declared headers) are hashed together, and the action's outputs are stored under that hash. Anyone whose action hashes identically gets the stored outputs back in milliseconds instead of redoing the work. # https://docs.nativelink.com/getting-started/verify-your-cache **Verify your cache**: Prove the cache is actually serving hits rather than quietly falling back to local work, using your build tool's own accounting, not a status page. **Who this is for:** anyone who has pointed a build at NativeLink and wants evidence it is working, rather than the absence of an error. **What you'll have at the end:** a measurement you can repeat, a number you can put in a message to your team, and the ability to tell a cold cache from a broken one. **Time:** about ten minutes. ## The measurement The test is a build, a clean, and the same build again. The first populates the cache; the second must not do the work. ### Build once, cold Read the last line of the output. It looks like this: ### Discard the local results Not `bazel clean --expunge`, which also throws away the external repositories and turns the next step into a ten-minute fetch that tells you nothing about the cache. ### Build again, warm Now the same line should read differently: ## Reading the number honestly A hit rate below 100% on an unchanged tree is normal. Bazel counts several kinds of work in that line, and only some of it is cacheable: ## Getting more detail than the summary line When the summary is not enough, ask Bazel for the per-action record: ## Other build tools The measurement is the same everywhere: build, clean, rebuild, and read the tool's own accounting. Only the vocabulary changes. Buck2 shows where each action ran in `buck2 log what-ran`; Pants counts remote cache requests in its end-of-run stats when `[stats].log` is on; recc logs an `Action Cache hit` line per compile with `RECC_LOG_LEVEL=info`. # https://docs.nativelink.com/getting-started/connect-your-build **Connect your build tool**: Use NativeLink without Bazel: Buck2, Siso, Pants, BuildStream, and CMake with recc. NativeLink speaks the standard Remote Execution API. If your build tool speaks the same protocol, you can point it at NativeLink without rewriting your build files. ## Options - [Buck2](/getting-started/connect-your-build/buck2): use Buck2's built-in remote execution client with NativeLink as CAS, Action Cache, and executor. - [Siso](/getting-started/connect-your-build/siso): configure Chromium's Ninja replacement to use NativeLink through the RE API. - [Pants](/getting-started/connect-your-build/pants): enable remote cache reads and writes from `pants.toml`. ## Common NativeLink endpoint The examples below assume a NativeLink server exposing CAS, Action Cache, execution, capabilities, and ByteStream on port `50051` with instance name `main`. ## FAQ Yes: they speak the same protocol against the same endpoint. Each tool hashes its actions differently, so tools won't share entries with each other, but every client of the same tool and toolchain will. # https://docs.nativelink.com/getting-started/connect-your-build/buck2 **Buck2**: Configure Buck2 to use NativeLink for remote cache and execution. Buck2 has a native Remote Execution API client. NativeLink can serve the CAS, Action Cache, capabilities, and execution services that Buck2 expects. ## Start NativeLink Use a config that exposes these services on `localhost:50051`: ## Configure Buck2 In your Buck2 project, add `.buckconfig`: ## Register an execution platform Buck2 needs an execution platform that allows remote actions. The integration test registers one with `remote_enabled = true` and `local_enabled = true`: ## Run a build The integration test uses a small staged build graph that mixes local-only and remote-capable actions, then verifies the output with `diff`. That shape is useful when validating a new cluster because it proves both upload and download paths are working. ## Troubleshooting - **Buck2 cannot connect.** Confirm NativeLink is listening on `50051` and Buck2 is not expecting TLS. - **Everything runs locally.** Check that your selected execution platform sets `remote_enabled = True`. - **Instance-name errors.** Make sure Buck2's `instance_name` matches the NativeLink config. The integration test uses `main`. ## FAQ `engine_address`, `action_cache_address`, and `cas_address` are separate settings because Buck2 lets you split those services across hosts. NativeLink serves them all from one listener, so all three point at the same `host:port`. # https://docs.nativelink.com/getting-started/connect-your-build/siso **Siso**: Point Siso at NativeLink for Chromium-style remote caching and execution. [Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md) is Chromium's Ninja replacement. It runs build actions on RBE natively, and NativeLink can be the RE API backend for Chromium and other Siso-driven builds. ## Local endpoint setup Start NativeLink on `localhost:50051` with instance name `main`, then export the environment variables Siso reads: ## Chromium-style setup Chromium checks in Siso configs. Point them at NativeLink by enabling remote execution in GN: ## Production TLS settings For a secured NativeLink endpoint, set the Siso TLS variables instead of `RBE_service_no_security=true`: ## Confirm it is working On a warm cache, cache hits should climb and network errors should stay at zero. If actions fall back to local execution, compare the platform properties Siso requests with the properties your NativeLink workers advertise. ## FAQ Only for TLS-free local development. Against any shared endpoint, use the TLS variables shown above instead. # https://docs.nativelink.com/getting-started/connect-your-build/chromium **Chromium**: Building Chromium with NativeLink as the Siso backend, a worked example for the largest public consumer. Chromium's build system shells out to [Siso](https://chromium.googlesource.com/build/+/refs/heads/main/siso/README.md), which speaks the Remote Execution API. Pointing it at a NativeLink cluster is a configuration change, with no patches to the Chromium tree. ## Prerequisites - A NativeLink cluster reachable from your build machines. Cache-only is fine to start; remote execution drops build times further. - A Chromium checkout with `depot_tools` on `$PATH`. - The Chromium source checkout (`fetch chromium`). ## Configure Siso Siso reads its server settings from environment variables. Add to your build shell: ## Generate Chromium build files In your Chromium checkout: ## Build Watch the build go. On a fresh cluster the first build will be mostly misses; subsequent builds hit the cache and run in a fraction of the time. ## What good looks like - **Cache hit rate**: on a warm cache the hit rate for an incremental build should be high and climb from build to build; watch it in the NativeLink cache metrics. - **Local fallback rate**: close to zero. Anything higher means workers are rejecting actions (platform mismatch, capacity). - **Network errors**: zero. Anything else is a problem. ## Worker requirements Chromium's Siso config takes its remote platform properties from `build/config/siso/backend_config/backend.star` (copy `template.star` and edit it, or point the `reapi_backend_config_path` gclient custom var at your own file). ## Troubleshooting - **All actions falling back to local.** Likely a platform-property mismatch. Compare what Chromium is requesting (in the siso log) against what your workers advertise. - **Siso is not being selected.** If the output directory was already generated for Ninja, run `gn clean out/Default`, regenerate with `use_siso=true`, and build again. ## FAQ No. Pointing Siso at NativeLink is configuration only: GN args, the Siso environment variables, and a local `backend.star` for the platform properties. The same applies to Chromium forks. # https://docs.nativelink.com/getting-started/connect-your-build/pants **Pants**: Enable Pants remote caching with NativeLink. Pants can read and write remote cache entries through the Remote Execution API. NativeLink can provide the remote store and Action Cache for Pants goals without changing BUILD files. ## Start NativeLink Start a NativeLink server that exposes CAS and Action Cache on `localhost:50051` with instance name `main`. The Docker quickstart in [Getting Started → Quickstart](/getting-started/quickstart) is enough for local cache validation. ## Configure Pants Add the remote cache settings to `pants.toml`: ## Run a goal Run any Pants goal as usual: ## Remote execution Pants remote execution uses the same Remote Execution API, but it needs a worker environment that matches the actions Pants sends. Start with remote caching, then add remote execution once you have worker platform properties and toolchains aligned. ## Troubleshooting - **No cache hits on the second run.** Check that `remote_cache_write` and `remote_cache_read` are both enabled. - **Instance-name errors.** Match `remote_instance_name` to the NativeLink config. The local examples use `main`. - **Connection failures.** Pants expects a `grpc://` URL in `remote_store_address` for a plaintext endpoint, or `grpcs://` for TLS. ## FAQ To let you split roles: CI can populate the cache (`remote_cache_write = true`) while developer laptops only read from it. For a single shared setup, enable both. # https://docs.nativelink.com/getting-started/connect-your-build/buildstream **BuildStream**: Configure BuildStream to use NativeLink for artifact storage and remote execution. BuildStream can use a Remote Execution API backend for artifact storage, Action Cache, CAS, and remote execution. NativeLink can serve those endpoints from the same `localhost:50051` listener used by the other local examples. ## Start NativeLink Use a config that exposes CAS, Action Cache, execution, capabilities, ByteStream, fetch, and push services on `localhost:50051`. ## Configure BuildStream Add a BuildStream config file that points artifact storage and remote execution services at NativeLink: ## Configure a project A minimal BuildStream project needs a project name, a supported BuildStream version, and an element directory: ## Add an element This small element stages local source files and runs a build command: ## Run a build Run BuildStream with the config file: ## Troubleshooting - **BuildStream cannot connect.** Match the `http://localhost:50051` endpoint in `buildstream.conf` to the NativeLink listener. The local test uses an insecure HTTP URL. - **Uploads are missing.** Set `push: true` under `artifacts.servers` so BuildStream can write artifacts back to NativeLink. ## FAQ They configure different BuildStream subsystems: artifact push/pull storage versus the execution, action-cache, and storage services used for remote builds. Pointing every URL at the same NativeLink listener is fine; it exposes all of those services on one port. # https://docs.nativelink.com/getting-started/connect-your-build/cmake-recc **CMake with recc**: Accelerate CMake builds with NativeLink as a remote cache, using recc as the bridge. This tutorial follows the approach Reid Kleckner laid out in [Distributed builds of LLVM with CMake, recc, and NativeLink](https://reidkleckner.dev/posts/llvm-recc-nativelink/). Read his post for the LLVM-scale walkthrough and the reasoning behind each piece. This page is the short, hands-on version for your own projects. ## Prerequisites - Docker - CMake 3.16+ - A C/C++ compiler (clang or gcc) ## 1. Start NativeLink The image is multi-arch (x86_64 and ARM64) as of v1.6.0, so it runs natively on Linux and Apple Silicon alike. ## 2. Install recc Works on Linux too if you have [Homebrew on Linux](https://docs.brew.sh/Homebrew-on-Linux). ## 3. Create the example Drop these two files into a fresh project directory and `cd` into it. You can also copy them from ## 4. Point recc at NativeLink What each one does: ## 5. Build it You should see `Hello, NativeLink!`. ## 6. Confirm the cache is live Wipe the build directory and rebuild. This time the compile output should come straight out of NativeLink: ## Teardown ## FAQ Not in this setup: `RECC_CACHE_ONLY=1` means misses compile locally and only the outputs travel through NativeLink. Drop that variable once you have remote workers configured and recc will dispatch compiles to them. ## Going further - Point `RECC_SERVER` at a shared NativeLink deployment to share the cache across your team. - Drop `RECC_CACHE_ONLY` once you have remote workers configured to actually offload compile work. See [Remote execution](/remote-execution). - For PCH, LTO, and other gnarly C++ build features at scale, read [Reid Kleckner's writeup](https://reidkleckner.dev/posts/llvm-recc-nativelink/). # https://docs.nativelink.com/getting-started/shared-cache **Shared cache**: Self-host NativeLink on hardware you control, with the operational checklist a team rollout actually needs. The source-available release of NativeLink is designed to run on your own infrastructure. This page covers what changes between the [10-minute quickstart](/getting-started/quickstart) and a deployment that serves your team without 3 AM pages. ## When to self-host Pick on-prem when one of these is true: ## What ships in the box A NativeLink deployment is composed of four roles. The same binary serves all of them; the JSON5 config decides which subset to run. ## The rollout checklist
  • **Pick the storage backend.** The quickstart's `basic_cas.json5` keeps everything under `/tmp` on one machine, which is fine for a 10-minute demo and useless for a team. Pick one before rolling out: - **Filesystem**: single-node clusters, sub-100GB caches. - **Cloud object storage**: anything multi-node. ## Container registry Official images are published to GitHub Container Registry. Pull a specific version tag; there is no `latest` tag to fall back on, and pinning is what you want in production anyway. ## Backups & recovery The CAS is the only stateful piece you can't trivially rebuild from clients. Snapshot strategy depends on the backend: ## FAQ Yes, for permitted use. Most of the monorepo is `FSL-1.1-Apache-2.0`: internal use, modification, and redistribution for non-competing purposes are allowed. Metrics and remote persistent workers are Business Source License modules: fine for an individual developer's cache, but shared, production, or commercial use of those modules needs NativeLink Enterprise or an intentionally inexpensive com… # https://docs.nativelink.com/remote-execution **Remote execution**: Workers that run the actions your cache misses. **Who this is for:** anyone with a working cache who wants the misses to stop running locally. **What you'll have at the end of this section:** a scheduler, at least one worker, and a build whose cache misses execute on the farm instead of on your machine. **Time:** an hour for the first remote action, longer if your toolchain isn't hermetic yet. ## What this section adds A cache answers the question "has this exact action already been run?" When the answer is no, something still has to run it, and with caching alone, that something is your laptop. Remote execution changes the answer to that second question. ## What to expect the first time The first remote action almost never works on the first try, and it's nearly always one of two walls. ## The pages, in order **Understand the change** ## Start here, or skip ahead **Start here** if you have a cache serving hits and haven't run anything remotely yet. # https://docs.nativelink.com/remote-execution/from-cache-to-execution **From cache to execution**: You have a cache serving hits. This is what changes when the misses stop running on your laptop: in the deployment, in the build invocation, and in what breaks first. **Who this is for:** anyone who finished Getting started and is deciding whether to take the next step. **What you'll have at the end:** an accurate picture of what remote execution adds, what it costs, and which wall you'll hit first. **Time:** ten minutes of reading, no commands. ## What execution buys you Two things, and it's worth being clear that they're separate. ## What's new in the deployment Three components, and one of them is a security boundary. ## What's new in the build invocation Less than you'd think. Two flags and a declaration. ## What stays exactly the same Your CAS and your action cache. The same stores, the same digests, the same hits you were already getting. ## What to expect the first time The first remote action rarely works on the first attempt, and it is nearly always one of two walls. Both are ordinary; both have a page. ## FAQ No. One `nativelink` process can host the CAS, the action cache, a scheduler, and a worker at once. That's exactly what # https://docs.nativelink.com/remote-execution/first-remote-action **Your first remote action**: Run a scheduler and a worker on your own machine, point Bazel at them, and watch a build action execute somewhere other than where you typed the command. **Who this is for:** anyone with a working cache who wants to see remote execution actually run. **What you'll have at the end:** a build whose actions Bazel reports as `remote`, running against a scheduler and worker you started yourself. **Time:** about fifteen minutes. ## Get the config The repo ships a config that is exactly this tutorial: CAS, action cache, scheduler, and one worker, all in a single process on localhost. ### Start the cluster Or, if you'd rather use the container image, mount the config in and publish both ports: ### Confirm the worker registered The worker connects *outward* to the scheduler on `50061` a moment after startup. In the server's log you should see it announce itself before you run anything, at the default `INFO` level: ### Point a build at it From any Bazel workspace: ### Read the result Ask Bazel where each action ran rather than trusting the wall-clock number: ## What just happened Four processes' worth of work happened inside one: ## When it doesn't work Your action's platform properties don't match any registered worker, so the scheduler is holding it in the queue. This is the single most common first-run failure and it is silent by design: the scheduler keeps waiting in case a suitable worker shows up later. ## FAQ Because it's the fewest moving parts that still exercises the real code paths: the worker here uses the same gRPC worker API, the same scheduler, and the same store implementations a production cluster does. Splitting into separate processes changes the deployment, not the protocol. See [Production configuration](/operate/production-config) for the split version. # https://docs.nativelink.com/remote-execution/platform-properties **Platform properties**: How the scheduler decides which worker gets an action, why a mismatch queues forever instead of failing, and how to read the matching diagnostics it already emits. **Who this is for:** anyone whose actions are queueing, and anyone about to run more than one kind of worker. **What you'll have at the end:** an accurate model of the matching rules, plus the log lines that tell you which rule you tripped. **Time:** fifteen minutes. ## The three places a property appears A property key has to be declared in all three, and they have to agree. ## The four property types Most documentation you'll find elsewhere describes three of these. `ignore` exists and is the only one that exempts a worker from carrying the key. ### Why `priority` is not the escape hatch people assume The matching loop checks key presence *before* it checks value compatibility. For every property the action requests, the scheduler looks the key up on the worker; if it isn't there, the worker is rejected, regardless of type. `priority` only tells the scheduler to stop caring about the *value*. ### `minimum` is a consumable resource, not a filter This is the most consequential and least visible behaviour in the system. ## What happens when it doesn't match The two failure modes are asymmetric, and knowing which one you're in tells you where to look. ## Reading the scheduler's own diagnostics The scheduler has a built-in periodic dump of exactly this. It's on by default at a ten-second interval: ## A worked example: routing to a GPU pool Say you have two worker pools and want tests tagged for GPU to land only on the GPU machines. ## About `container-image` It appears in nearly every example config and it is worth being precise about what it does, because the name suggests something that isn't happening: **NativeLink does not pull or launch a container per action.** No code sits behind this key. It is an ordinary string property like any other. ## Troubleshooting ## FAQ `priority` if you have one OS in the fleet, `exact` the moment you have two. The example configs use `priority` with an empty value because a homogeneous fleet gains nothing from enforcing it, but that's a default for a tutorial, not a recommendation for a mixed fleet. With `exact` you get a real routing guarantee; with `priority` you get a label. # https://docs.nativelink.com/remote-execution/toolchains-and-hermeticity **Toolchains and hermeticity**: Your action ran on a worker and failed on a missing compiler. Here is why that happens, and how to pin the toolchain so local and remote builds hash identically. **Who this is for:** anyone whose actions execute remotely but fail on the worker for reasons that never happen locally. **What you'll have at the end:** a Nix-pinned toolchain that produces the same action digests on your machine and on a worker. **Time:** thirty minutes for the walkthrough. ## The problem An action is a closed description of some work: input digests, a command line, an environment, and platform properties. If the compiler your command invokes isn't in that description, the action is a lie: it works on your machine because your machine happens to have the compiler, and it fails on a worker that doesn't. ## What ships in `@local-remote-execution` Three toolchain families, and they do not behave the same way once you leave `x86_64-linux`. ## Set it up
  • ### Install Nix with flakes enabled The [next-gen installer](https://github.com/NixOS/experimental-nix-installer) is the shortest path. If you already have Nix, make sure `experimental-features = nix-command flakes` is in your `nix.conf`. ### Pull the template This writes a `flake.nix` importing `nativelink.flakeModules.lre` and pinning `lre = { inherit (pkgs.lre.lre-cc.meta) Env; }`, plus a `hello-world.cpp` example, its `BUILD.bazel`, and `platforms/BUILD.bazel`. ### Initialise git before you enter the shell This step is not optional and it fails quietly. ### Enter the dev shell This fetches the Nix-pinned toolchain and writes `lre.bazelrc` (a symlink into the Nix store). On `aarch64-darwin` the generated file looks like this, abbreviated; your store paths will differ: ### Point `user.bazelrc` at your cluster `nix flake init` writes a `user.bazelrc` with the literal string `TODO` as every value. That is what lands on disk, not a placeholder left in this page: ### Build With a real endpoint and a worker running the `nativelink-worker-lre-cc` image, this compiles `src/hello-world.cpp` on that worker and returns the binary through the same CAS your local Bazel reads from. Run it again and it's a cache hit. ## Adding LRE to a project you already have The template bootstraps a *new* workspace. For an existing one you need the flake-parts module wiring, the `bazel_dep` override, and a `try-import` of the generated `lre.bazelrc`. ## Troubleshooting ## FAQ You can get *reproducibility across a fleet* by baking the toolchain into one worker image and routing every action to it. That's path 1 at the top of this page, and for many teams it's enough. What you don't get is matching action digests between a developer's laptop and the fleet, because the laptop isn't running that image. Nix pinning is what closes that gap. # https://docs.nativelink.com/remote-execution/containers-and-images **Containers and images**: What the container-image property actually does (and doesn't), plus how to build worker images and keep what your build requests in sync with what your fleet advertises. **Who this is for:** anyone whose actions need a specific environment, and anyone running more than one kind of worker image. **What you'll have at the end:** worker images that carry your toolchain, and a routing scheme that sends each action to the right one. **Time:** twenty minutes. ## What `container-image` is not Start here, because the name is misleading and the misunderstanding is expensive. ## The actual model One image per worker *pool*. Every action a given worker runs shares that worker's environment. Heterogeneity comes from running several pools and routing between them, not from varying the image per action. ## Build a worker image A NativeLink worker image is any image that contains your toolchain and can run the `nativelink` binary with a worker config. The repo's Nix helper, `createWorker`, turns a toolchain image into a `nativelink-worker-*` image: ## Wire up the routing
  • ### Declare the key on the scheduler Use `exact` once you have more than one image. The examples ship `priority`, which matches any value: fine for a single-pool tutorial, useless as a routing mechanism. ### Advertise it on each worker Pin a tag you can change deliberately. `:latest` makes the advertised string stable while the actual contents drift, which defeats the entire point: the scheduler will keep matching, and your builds will silently change behaviour. ### Request it from the build Globally, if your whole build wants one environment: ## Rolling an image without stranding actions Order matters, because the two sides are matched on a literal string. ## Troubleshooting ## FAQ Not meaningfully. A worker process runs in one environment; the property is an assertion about that environment. You can advertise several values for a `priority` key, but that's a routing label, not a second toolchain. Run one pool per image. # https://docs.nativelink.com/remote-execution/persistent-workers **Persistent workers**: Keep a JVM or Node compiler process warm across actions instead of paying its startup cost every time, and understand exactly what NativeLink gives up to do it. **Who this is for:** anyone whose remote builds are dominated by JVM or TypeScript compile actions. **What you'll have at the end:** compile actions reusing a warm process, and a clear-eyed view of the isolation you trade for it. **Time:** an hour, most of it in your build rules rather than in NativeLink. ## Two things to know before you start **Almost nothing needs configuring in NativeLink.** No field anywhere in [the configuration model](/configuration) turns this on, tunes the pool, or selects a protocol; `LocalWorkerConfig` has no persistent-worker section at all, and every pool parameter is a compiled-in constant. The feature is driven by two platform properties on each action. ## Opt an action in The worker looks at two of the action's platform properties (the REAPI `Platform` on the `Action`): `supports-workers` must be exactly `1`, and `requires-worker-protocol` picks the wire format. Nothing on the worker's own `platform_properties` is involved. ## What makes two actions share a process Actions are pooled by a key derived from the command line: ## The isolation you give up This is the part to read twice. NativeLink applies real isolation to ordinary actions and **applies none of it to persistent workers**. The persistent path returns before any of it runs. ## The pool's actual behaviour Every parameter below is a compiled-in constant. None is configurable. ## Steps
  • ## When it doesn't work Either the properties are not reaching the action's `Platform`, or every action is getting its own key. Check `supports-workers` is exactly `"1"` and is sent as a platform property (see the callout above), and that the scheduler declares both keys; an undeclared key leaves the action queued forever. # https://docs.nativelink.com/remote-execution/local-testing **Local testing**: Keep a NativeLink cluster on your own machine to test config changes, reproduce scheduler behaviour, and prove a client works, in the two shapes the repo itself uses. **Who this is for:** anyone changing a config, debugging a client, or reproducing something a shared cluster did. **What you'll have at the end:** a local cluster in whichever of the two shapes matches what you're testing, and a reliable way to tell what actually ran where. **Time:** ten minutes for the single-process shape. ## Which shape to run ## Shape 1: one process `--remote_cache` and `--remote_executor` both point at `127.0.0.1:50051`. `50061` exists only so the worker can register with the scheduler; Bazel never talks to it. ## Shape 2: the compose split When you need separate processes (a CAS process and a scheduler process talking over `grpc` stores, which is what a real deployment looks like), the repo ships that too, and its own integration tests run against it. ## Verify what actually ran Wall-clock time is a bad signal and Bazel's summary line is only a rough one. Ask the build event log directly: ## Troubleshooting At least one worker's `platform_properties` must satisfy what your actions request, and every requested key must be declared in the scheduler's `supported_platform_properties`. Starting with just `cpu_count: "minimum"` on the scheduler and `cpu_count: { values: ["1"] }` on the worker unblocks most local setups. ## FAQ Yes: the config exposes standard Remote Execution API services, so Buck2, Pants, Siso, and recc point at `127.0.0.1:50051` the same way. The Bazel flags are just the worked example; [Connect your build](/getting-started/connect-your-build) has the others. # https://docs.nativelink.com/remote-execution/examples-and-templates **Examples and templates**: Runnable configs, deployment topologies, and project scaffolds you can copy, plus the three client-side patterns most teams actually adopt. **Who this is for:** anyone who wants a working starting point rather than a blank file. **What you'll have at the end:** the right example for your situation, and a client-side pattern that matches how you want work split between laptops and a fleet. **Time:** varies; the point of this page is to find the thing, not to read it. ## Server configs is the largest and most useful collection. Every file is a complete, loadable config. ## Deployment topologies holds deployment material rather than single config files: the `docker-compose/` stack, a RHEL 8 `Dockerfile` (`rhel/`), the metrics pipeline configs and dashboards (`metrics/`), and the persistent-worker rule sketches (`persistent-workers/`). ## Project templates The flake ships templates for the client side: a project already wired to a Nix-pinned toolchain. ## Three client-side patterns Independently of which config you start from, there are three ways teams wire their builds. Most adopt them in this order. ### Pattern A: Cache only Every action runs locally; results land in the shared cache. No worker fleet needed, which makes it the cheapest thing that helps. ### Pattern B: Cache plus execution Actions ship to a worker fleet; the cache underneath stores everything. ### Pattern C: Hybrid Small actions local, everything else remote, decided by policy. ### Choosing The speedup numbers you'll see quoted for these vary by an order of magnitude between workloads, and mostly reflect how much of a given build is cache-missing compilation. Measure your own before committing to a shape. ## FAQ `s3_backend_with_local_fast_cas.json5` for the store layer and the `docker-compose/` topology for the process split; that combination is the shape [Production configuration](/operate/production-config) describes at full size. Don't start from `local_rbe_self_test.json5`; it's optimised for being readable in one screen, not for running a fleet. # https://docs.nativelink.com/configuration **Configuration**: The config model on one page, so you can read and write a NativeLink config instead of copying one. **Who this is for:** anyone running a cluster they got working by copying an example, who now needs to change it. **What you'll have at the end of this section:** the ability to read any NativeLink config and write one from an empty file. **Time:** an afternoon, most of it on the capstone. ## One file, six keys A NativeLink deployment of any shape (one process on a laptop, a sharded fleet across three regions) is described by a single JSON5 file. It has exactly six top-level keys, and only two of them are required. ## How a request traverses them Read that diagram as: **the `servers` array is the only entry point, and everything else is reached from it by name.** A service doesn't contain a store, it names one. A worker doesn't contain a scheduler, it dials one over the worker API. The names are the wiring. ## Everything is a named array Every top-level collection is an **array of objects**, not a map keyed by name. Stores and schedulers carry their `name` beside the type key; workers carry it inside the `local` block; servers have an optional `name`: ## A typo is a startup error, not a silent default Nearly every struct in the config crate is `#[serde(deny_unknown_fields)]`. Misspell a key and the process refuses to start and tells you which key it didn't recognise, rather than ignoring it and behaving strangely three hours later. ## The pages, in order **Learn the mechanics** ## Start here, or skip ahead **Start here** if you have a cluster running from a copied example and want to understand what you copied. # https://docs.nativelink.com/configuration/config-file **The config file**: JSON5 mechanics, environment substitution, how the binary is pointed at a config, and how to find out a config is wrong before it takes a cluster down with it. **Who this is for:** anyone about to edit a NativeLink config for the first time. **What you'll know at the end:** what dialect the file is written in, how to keep secrets and addresses out of it, how the binary finds it, and how to check one is valid without waiting for a production rollout to tell you. **Time:** twenty minutes. ## It's JSON5, not JSON NativeLink parses its config with `serde_json5`, not `serde_json`. The difference matters every time you edit a file: ## Environment substitution Any string field that is deserialized through the config crate's shellexpand helpers accepts shell-style variable references, including the default-if-unset form: ## How the binary finds the config One way only, and it is positional: ## Checking a config before you trust it Here is the honest answer, because the alternative wastes an afternoon: **there is no `--validate` flag and no dry-run mode.** The binary takes a config path and starts. ## Editor validation, if you want it The config crate can emit a JSON Schema for the whole `CasConfig` type, which editors will use for completion and inline errors: ## FAQ Not with an include directive; the parser reads exactly one file. What you do instead is split by *role*: one config for the CAS process, one for the scheduler, one for the workers, each a complete file naming the others by network address. That's the same split described in [the section overview](/configuration), and it's how the docker-compose deployment is laid out. # https://docs.nativelink.com/configuration/stores **Stores**: Declaring stores, naming them, and the idea that unlocks the rest of the config: stores compose, and one store refers to another by name. **Who this is for:** anyone who has seen a NativeLink config with a store nested five levels deep and wanted to know why. **What you'll know at the end:** how to declare a store, how to name one, how composition works, and which layers are worth reaching for. **Time:** thirty minutes. ## A store is one entry in an array The `stores` array holds every storage backend the process can use. Each entry is an object with a `name` and exactly one type key: ## Backing it with disk Swap the type key and the same two stores survive a restart: ## Stores compose Here is the idea the rest of this page is about. **Most store types don't store anything.** They wrap another store and change its behaviour, and the store they wrap is written inline, as a full store spec: ## `ref_store`: naming instead of nesting Nesting alone can't express *sharing*. If two stores should write to the same place, inlining the backend twice gives you two independent stores that happen to have identical settings. ## The layers worth knowing You don't need all eighteen store types to write a good config. These are the ones that appear in nearly every non-trivial one: ## What keys the store A store is content-addressed, so what goes in a key is the digest, and every server, worker, and build client sharing a cache must compute that digest the same way. SHA-256 and BLAKE3 produce different keys for identical bytes, so uploads made under one function are cache *misses* for clients using the other. ### Which function to pick We compared both algorithms on an Apple M2 Max with 12 CPU cores and 32 GB of memory, running macOS 15.6 and Bazel 9.1.1. Each of two alternating runs built `//:nativelink` against a fresh local NativeLink filesystem cache with hash and size verification enabled. Uploads were synchronous, and the Bazel output and NativeLink cache directories were isolated by algorithm. ## FAQ `eviction_policy.max_bytes` caps it, and once full the least-recently-used entries are evicted to make room. An evicted CAS entry is just a future cache miss; the client rebuilds and re-uploads. An evicted entry that a cached action result still points at is the problem `completeness_checking` exists to solve. # https://docs.nativelink.com/configuration/servers-and-services **Servers and services**: Listeners, the eleven services, instance names, and the port split that keeps the worker API off your public interface. **Who this is for:** anyone deciding what a NativeLink process should expose, and on which port. **What you'll know at the end:** what a listener is, what each of the eleven services does, how `instance_name` partitions a cache, and why the worker API must not share a port with anything else. **Time:** thirty minutes. ## A server is a listener plus a set of services The `servers` array is the only entry point into a NativeLink process. Each entry binds one socket and decides what is reachable on it: ## The eleven services A cache-only deployment needs the first four. Adding execution adds `execution` and `worker_api`. The rest are opt-in. ## Instance names Most services are declared as an **array**, and each element carries an `instance_name`. That field is a routing key: it selects which store (or scheduler) serves a request. The instance name is not part of the storage key, so two instance names pointing at *different* stores are fully isolated, and two pointing at the *same* store share every blob. ## The port split The `worker_api` service is different from every other service on this page, and the difference is a security boundary rather than a design preference. ## TLS, when you need it TLS is a property of the listener: ## FAQ Yes; the array is unbounded, and separating concerns by port is the intended use. `deployment-examples/docker-compose/local-storage-cas.json5` serves the same cache twice, plain on `:50051` and TLS on `:50071`. A third listener for build events, or a plain-HTTP health endpoint a load balancer can probe without TLS, follows the same pattern. # https://docs.nativelink.com/configuration/scheduler-and-workers **Schedulers and workers**: The minimum config for each, the properties contract between them, and the fields that decide what happens when something goes wrong. **Who this is for:** anyone adding execution to a config that currently only caches. **What you'll know at the end:** the minimum scheduler and worker blocks, how the two are wired to each other, and which of the many optional fields actually change behaviour you'll notice. **Time:** thirty minutes. ## The minimum scheduler A scheduler is one entry in the `schedulers` array, with a name and a type: ## The minimum worker A worker is one entry in the `workers` array. `local` is currently the only type; it means the worker runs actions in this process, on this machine: ## How they find each other Nothing in the scheduler names a worker. Workers are not declared to the scheduler at all; they connect to it, announce their platform properties, and are added to the pool. Removing a worker means stopping it. ## The fields that change what you'll notice `SimpleSpec` has ten optional fields. Most can stay at their defaults forever. These four are the ones worth setting deliberately: ## Putting the pieces together Adding execution to a cache-only config takes four edits, all of which you've now seen: ## FAQ Yes, and that's what `nativelink-config/examples/basic_cas.json5` is: one process with stores, a scheduler, a worker, a public server on `:50051` and a private one on `:50061`. It's the right shape for a laptop and for testing. Splitting into separate processes is a deployment change, not a config rewrite; the same blocks move into separate files. # https://docs.nativelink.com/configuration/your-first-full-config **Your first full config**: An empty file to a running cache-and-execution cluster, one block at a time, checking after each. **Who this is for:** anyone who has read the rest of this section and wants to prove it by writing a config from nothing. **What you'll have at the end:** a single-process NativeLink serving a cache and executing actions, from a file you wrote yourself and can explain line by line. **Time:** about an hour. ## Before you start You need the `nativelink` binary and a writable directory. This walkthrough uses `/tmp/nativelink`; if you use somewhere else, keep the **content path and the work directory on the same filesystem**, because the worker hard-links between them. ## Step 1: Two stores and a listener Start a file called `my-config.json5`. The smallest useful NativeLink is a cache: two stores and one server. ## Step 2: Point a build at it Nothing about the config changes here, but proving the cache works before adding execution keeps the next failure unambiguous. ## Step 3: Add a scheduler Execution needs a queue. Add a `schedulers` array beside `stores`: ## Step 4: Expose execution, on two ports Now wire the scheduler in. Add `execution` to the public server, give `capabilities` a `remote_execution` block, and add a **second server** for the worker API: ## Step 5: Add a worker A worker needs a `fast_slow` store with a filesystem fast tier. Replace `CAS_MAIN_STORE` with one: ## Step 6: Execute something Point the build at execution rather than just the cache: ## The finished file Compare it with `nativelink-config/examples/basic_cas.json5` in the source tree. The differences are instructive: the shipped example registers every service twice, under `instance_name: ""` and `"main"`, so clients that omit the instance name also work, and it declares nineteen platform properties rather than three. ## What this is not yet This config is correct and it is not production. Four things are missing, and each has a page: # https://docs.nativelink.com/how-to **How-to guides**: Pick the task you're doing, not the artifact you're reading about. **Who this is for:** anyone with a running NativeLink who now needs it to do one specific thing differently. **What you'll have at the end:** the one page that does your task, rather than a tour. **Time:** as long as the task takes. ## How this section works Every page here is a task, phrased as the thing you are trying to do. None of them is a tour of a subsystem (that's what [Concepts](/explanations/architecture) is for), and none of them is an exhaustive field list, which is what [Reference](/reference/glossary) is for. ## Pick your task **I need my cache stored somewhere else.** [Storage backends](/how-to/stores) compares every place NativeLink can put bytes ([filesystem](/how-to/stores/filesystem), [S3 and compatibles](/how-to/stores/s3-and-compatible), [GCS](/how-to/stores/gcs), [Azure Blob](/how-to/stores/azure), [OCI](/how-to/stores/oci-object-storage), [Redis](/how-to/stores/redis) and [Mongo](/how-to/stores/mongo)) on durability, latency and c… # https://docs.nativelink.com/how-to/stores **Storage backends**: Every place NativeLink can put bytes, compared on durability, latency and cost, and the composition almost every real deployment lands on. **Who this is for:** anyone whose cache currently lives somewhere they can't keep it. **What you'll have at the end:** the one backend page that matches your durability and latency requirements, and the shape you'll wrap it in. **Time:** ten minutes to choose, then as long as the backend page takes. ## Everything here is a store entry Nothing on this page is a new concept. A backend is one entry in the `stores` array with a name and a type, exactly as described in [Stores](/configuration/stores): ## Pick a backend `memory`, `grpc` and `noop` have no page of their own because there is nothing to decide: `memory` takes an eviction policy, `grpc` takes an upstream address, and `noop` takes nothing at all. All three are covered in [Compose stores](/how-to/stores/compose-stores). ## The shape almost everyone lands on Very few production deployments name a cloud bucket as their CAS store directly. The read pattern of a build cache (many small objects, requested repeatedly, in bursts) is exactly what object storage prices and latency are worst at. The shape that works is a local fast tier in front of a durable slow tier: ## Shrinking what you move Two pages in this group are not backends at all. They change how many bytes travel between the client and NativeLink, whichever backend is underneath: # https://docs.nativelink.com/how-to/stores/filesystem **Filesystem store**: Put the cache on local disk: the two paths that must share a block device, the eviction policy you cannot skip, and the tuning knobs that matter on real hardware. **Who this is for:** anyone who wants the cache to survive a restart without paying for object storage, and anyone building the fast tier of a larger composition. **What you'll have at the end:** a `filesystem` store sized to your disk, with eviction that actually runs. **Time:** fifteen minutes. ## The recipe Create the parent directory and start NativeLink. The store scans `content_path` on boot and adopts every file whose name parses as a store key and generation (`--` under `d2/`, or `-` under `s2/`); a file whose name does not parse is deleted. The scan reads metadata, not content, so it does not check that bytes still match their digest. ## The two paths `content_path` holds the data. `temp_path` holds objects mid-upload and mid-delete, while their contents cannot yet be trusted. ## The eviction policy you cannot skip `eviction_policy` is optional in the schema and mandatory in practice. Without it, nothing is ever removed and the store grows until the disk is full. ## Tuning that matters on real hardware **`max_concurrent_writes`** (default `0`, unlimited). Every write streams into a temp file and calls `sync_all()`. Enough concurrent writes will saturate disk I/O and start blocking the async runtime. On a busy shared CAS, set this to something bounded (a few dozen) rather than leaving it open. ## If this store feeds a worker A worker's `cas_fast_slow_store` must have a `filesystem` store as its `fast` half. The worker builds each action's input tree by hard-linking files out of that store, and hard links require real files on a real filesystem; a memory or object store cannot supply them. ## Steps
  • ## When it doesn't work An upload that encounters a missing file during duplicate comparison removes the stale index entry and stores the uploaded content. A read removes only the entry whose file was missing. In a `fast_slow` composition, that read can fall through to the slow store and refill the fast tier. # https://docs.nativelink.com/how-to/stores/s3-and-compatible **S3 and compatible**: Back the cache with Amazon S3, Cloudflare R2 or NetApp ONTAP S3: one store type, three providers, and the shared options that decide cost and failure behaviour. **Who this is for:** anyone who wants durability they don't operate, on S3 or something that speaks S3. **What you'll have at the end:** a bucket-backed CAS behind a local fast tier, with credentials that aren't in the config file. **Time:** thirty minutes, most of it bucket and IAM setup. ## Amazon S3 `region` and `bucket` are the only fields you have to set. Credentials come from the standard AWS chain (environment variables, the shared credentials file, the EC2/ECS/EKS instance or task role), so nothing secret goes in the config. ## Cloudflare R2 R2 has no regions from the client's point of view: the endpoint is derived entirely from your account ID. ## NetApp ONTAP S3 On-premises S3 needs an explicit endpoint, the storage VM serving it, and usually a private CA bundle: ## The options all three share Every provider flattens the same block of common options. These are the ones worth setting deliberately: ## Steps
  • ## When it doesn't work For `aws`, the credential chain found nothing. Confirm the process can see `AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY`, or that the instance/task role is attached and reachable. For `r2`, set `access_key_id` and `secret_access_key` explicitly, because R2 tokens are not in the AWS chain. # https://docs.nativelink.com/how-to/stores/gcs **Google Cloud Storage**: Back the cache with a GCS bucket: authentication, the resumable-upload chunk size, and the timeout fields whose names disagree with their units. **Who this is for:** anyone running NativeLink on Google Cloud who wants the durable tier to be a GCS bucket. **What you'll have at the end:** a bucket-backed CAS behind a local fast tier, authenticated by the ambient service account. **Time:** twenty minutes, most of it IAM. ## The recipe `bucket` is the only field you have to set. No region is needed: a GCS bucket's location is a property of the bucket, decided when you create it, and the client reaches it the same way wherever it lives. Put the bucket in the region your workers run in; cross-region reads are both slower and billed. ## Authentication Credentials come from the ambient Google credential chain: a credentials file named by `GOOGLE_APPLICATION_CREDENTIALS` (or inline JSON in `GOOGLE_APPLICATION_CREDENTIALS_JSON`), the application-default credentials written by `gcloud auth application-default login`, or otherwise the metadata server on GCE, GKE or Cloud Run. Nothing secret goes in the config file. ## Options worth setting Objects under 5 MB with a known size go up in one request; everything else uses a resumable upload, sent chunk by chunk with `resumable_chunk_size` bytes per request. The value is rounded to a multiple of 256 KiB and capped at the 2 MB default, so it can only be lowered, which trades more round trips per object for less memory held per in-flight upload. No setting raises it. ## Steps
  • ## When it doesn't work The credential chain found something, but it can't act on the bucket. Confirm the service account attached to the instance or workload identity has object read and write on it. If it can create but not overwrite, re-uploads of a digest that already exists fail, because GCS needs the delete permission to replace an object. # https://docs.nativelink.com/how-to/stores/azure **Azure Blob Storage**: Back the cache with an Azure Blob container: account and container naming, the endpoint override for Azurite, and when a SAS URL replaces everything else. **Who this is for:** anyone running NativeLink on Azure who wants the durable tier to be a Blob Storage container. **What you'll have at the end:** a container-backed CAS behind a local fast tier. **Time:** twenty minutes, most of it access configuration. ## The recipe `account_name` and `container` are the pair you have to set. From them the store builds the endpoint `https://{account_name}.blob.core.windows.net/{container}`. The container is a blob container inside the storage account. Create it before starting NativeLink; the store does not create it for you. ## Three ways to point at a container **Account plus container.** The default above. The endpoint is derived, and the store authenticates with Entra ID Workload Identity and nothing else: it reads `AZURE_TENANT_ID`, `AZURE_CLIENT_ID` and `AZURE_FEDERATED_TOKEN_FILE` (the variables AKS workload identity injects into a pod) and fails at startup if they are missing. A VM managed identity, a storage account key or an `az login` session are not tried. ## Options worth setting Blobs under 5 MiB with a known size are uploaded in one request; anything larger, or of unknown size, is staged as blocks of at least 5 MiB and committed with a block list. `retry` and `consider_expired_after_s` behave as described in [the shared options section](/how-to/stores/s3-and-compatible); `insecure_allow_http` and `disable_http2` are accepted but not read by the Azure store. ## Steps
  • ## When it doesn't work Either the workload identity lacks Storage Blob Data Contributor on the container, or a `sas_url` has expired. A SAS expiry surfaces as an authentication failure, so check the signature's window before assuming the role assignment is wrong. If the store did not start at all with a message about the Workload Identity credential, the `AZURE_*` variables are missing. # https://docs.nativelink.com/how-to/stores/redis **Redis**: Back the cache with Redis (standalone, sentinel or cluster) and the concurrency, chunking and timeout settings that keep it from timing out under a real build. **Who this is for:** anyone who wants several NativeLink processes to share one low-latency store, or who needs the scheduler's pub/sub subscriptions. **What you'll have at the end:** a Redis-backed store with concurrency and timeouts set so it survives a burst. **Time:** twenty minutes. ## The recipe `addresses` is the only required field. The URL carries everything Redis needs: scheme, optional credentials, host, port and database index. `redis://username:password@redis-server-url:6380/99` selects port 6380 and database 99. Keep credentials out of the file by writing `redis://${REDIS_USER}:${REDIS_PASSWORD}@redis:6379/` and exporting the variables; see [The config file](/configuration/config-file). ## Standalone, sentinel or cluster `mode` takes `standard` (the default), `sentinel` or `cluster`. ## Sizing it so it doesn't time out Redis is single-threaded per node, so the settings that matter are the ones bounding how much work you can have in flight. ## Where Redis actually earns its place Rather than as a bulk CAS, Redis is usually the fast or index half of a composition: ## Steps
  • ## When it doesn't work Too much in flight. Lower `max_client_permits` so work queues in NativeLink instead of piling onto a single-threaded Redis. Only after that, consider raising `command_timeout_ms`. # https://docs.nativelink.com/how-to/stores/mongo **MongoDB**: Back the cache with MongoDB: collections, write concern, timeouts, and what the change-streams flag is for. **Who this is for:** anyone who already operates MongoDB. **What you'll have at the end:** a Mongo-backed CAS behind a local fast tier. **Time:** twenty-five minutes. ## The recipe `connection_string` is the only required field and takes either form Mongo accepts, `mongodb://localhost:27017` or `mongodb+srv://cluster.mongodb.net` for Atlas. It carries the credentials, so keep it in the environment and reference it with `${VAR}`; see [The config file](/configuration/config-file). ## Change streams and the scheduler The Mongo store implements the scheduler-store interface, and `enable_change_streams: true` is what that interface needs: the store refuses to hand out scheduler subscriptions without it. In v1.6.5, though, the `simple` scheduler's `experimental_backend` accepts only `memory` and `redis`, so there is no config path that puts a Mongo store behind the scheduler. ## Options worth setting Write concern is exposed as three separate fields mirroring Mongo's own: `write_concern_w` (a number like `1`, or the string `"majority"`), `write_concern_j` (whether to wait for the journal), and `write_concern_timeout_ms`. All three are optional and unset means the deployment's own default applies, but `write_concern_j` or `write_concern_timeout_ms` without `write_concern_w` is rejected at startup. ## Steps
  • ## When it doesn't work Not in v1.6.5. The store implements the scheduler-store interface behind `enable_change_streams`, but the `simple` scheduler's `experimental_backend` only accepts `memory` and `redis`, so there is nothing to point at it. Use [Redis](/how-to/stores/redis) for a shared scheduler. # https://docs.nativelink.com/how-to/stores/oci-object-storage **Oracle Cloud (OCI) Object Storage**: Back the cache with OCI Object Storage through its S3 Compatibility API: Customer Secret Keys, the two adjustments the store makes for you, and what has actually been verified. **Who this is for:** anyone running NativeLink on Oracle Cloud who wants the durable tier to be an OCI bucket. **What you'll have at the end:** a bucket-backed CAS and Action Cache authenticated with a Customer Secret Key. **Time:** thirty minutes, most of it namespace and policy setup. ## How it works [OCI Object Storage](https://www.oracle.com/cloud/storage/object-storage/) exposes an [S3 Compatibility API](https://docs.oracle.com/en-us/iaas/Content/Object/Tasks/s3compatibleapi.htm), and the `oci` provider is a thin adapter pointing the S3 store at it. Two OCI-specific adjustments are applied automatically, and you do not configure either: ## Before you start You need three things. ## Authentication The S3 Compatibility API authenticates with Customer Secret Keys, a static access-key and secret-key pair used as `access_key_id` and `secret_access_key`, signed with AWS SigV4. ## The recipe A minimal CAS and Action Cache sharing one bucket, segmented by `key_prefix`: ### Fields OCI also accepts `us-east-1` as a `region` value to target the tenancy home region, which is occasionally useful when a tool insists on an AWS region name. ## Steps
  • ## Operational notes **Reap incomplete multipart uploads.** When a multipart upload fails mid-flight the store issues `AbortMultipartUpload` to clean up, and OCI honors both `ListMultipartUploads` and `AbortMultipartUpload`. A process killed *before* it can abort still leaves orphaned parts, which is what the lifecycle rule is for. ## When it doesn't work Something is re-adding the default trailing checksum the store removes. This is the failure the checksum downgrade exists to prevent, and it shows up first on Action Cache entries because they are small single-`PUT` objects. Confirm you are on a NativeLink version that carries the `oci` provider rather than pointing the `aws` provider at an OCI endpoint by hand. # https://docs.nativelink.com/how-to/stores/compose-stores **Compose stores**: The wrapper vocabulary (fast_slow, verify, dedup, existence_cache, size_partitioning, shard and the rest) and the order the layers have to go in. **Who this is for:** anyone who has picked a backend and now needs it to be fast, cheap and trustworthy. **What you'll have at the end:** a layered store you can explain line by line, and the vocabulary to read anyone else's config. **Time:** thirty minutes. ## The two ways stores refer to each other A wrapper can hold its inner store inline: ## The wrappers ### `fast_slow`: a tier in front of a tier The single most important wrapper. Reads try `fast` first and fall through to `slow` on a miss, populating `fast` as they go. Writes go to both. ### `verify`: refuse to store corruption Checks the size and hash of an upload against the digest it is filed under, and rejects the write on a mismatch. Reads pass straight through to the backend unchecked, so this protects what enters the store, not what a backend later hands back. ### `completeness_checking`: the Action Cache wrapper The AC's counterpart. Before returning an action result as a hit, it confirms every output digest that result references still exists in the CAS. Without it, a client gets a cache hit pointing at blobs that have since been evicted. ### `existence_cache`: stop asking the same question Caches the answer to "does this digest exist" so `FindMissingBlobs` doesn't hit the backend for every blob of every build. ### `compression`: smaller on the backend Compresses blobs with LZ4 before they reach the wrapped store. ### `dedup`: store the changed parts only Splits blobs into content-defined chunks, stores each chunk once, and keeps a per-blob index of which chunks it is made of. ### `size_partitioning`: route by blob size Sends everything under `size` to one store and everything at or above it to another. ### `shard`: spread across several backends Distributes blobs across stores by digest, weighted. ### `cache_metrics`: measure one layer Wraps a store to record hit rates and timings under a label: ### The three with nothing to configure `memory` takes an `eviction_policy` and nothing else. It is the right fast half for a short-lived process and the wrong one for anything that must survive a restart. ## The order the layers go in Reading a real config outside-in, each layer answers a different question: ## A full example The shipped GCS and Azure examples both use this shape, and it is a good default for a cloud-backed CAS: ## Steps
  • ## When it doesn't work Almost every store spec is strict about unknown fields, so this is usually a field placed at the wrong nesting level: `eviction_policy` on the wrapper rather than on the store, or a backend option one layer too high. Check what the innermost brace you're inside actually is. # https://docs.nativelink.com/how-to/stores/compression **Remote cache compression**: Cut remote cache transfer bytes for compressible artifacts with REAPI zstd wire compression and Bazel's --remote_cache_compression. **Who this is for:** anyone whose clients reach NativeLink across a real network and whose artifacts compress well. **What you'll have at the end:** zstd on the wire, with stores still holding raw bytes. **Time:** ten minutes. ## Requirements - A Bazel client with `--remote_cache_compression` (available since Bazel 5). - Wire compression is **optional and off until it is enabled somewhere in the NativeLink process**. Without the configuration below or an explicitly enabled CAS `grpc` store, NativeLink does not advertise zstd, rejects `compressed-blobs/zstd` requests with `InvalidArgument`, and keeps its own outgoing transfers on the identity path,… ## Enabling it Set `remote_cache_compression: true` on the capabilities service for the instance. The capabilities service advertises zstd to clients, and the ByteStream and CAS services accept and serve compressed payloads for that instance: ### NativeLink-to-NativeLink transfers A `grpc` store with `store_type: "cas"` can compress its own uploads and full-blob downloads to the upstream with `experimental_remote_cache_compression`. Blobs of 64 KiB and above go as `compressed-blobs/zstd`; smaller blobs and ranged reads stay on the identity path. The upstream instance must have `remote_cache_compression` on its capabilities service, or compressed requests fail with `InvalidArgument`. ## Semantics worth knowing - Resource names and digests always refer to the **uncompressed** blob; only the bytes on the wire are compressed. ByteStream compressed writes report `committed_size` in compressed wire bytes, as the REAPI extension specifies. ## When it helps, and when it doesn't Compression pays off when clients reach the cache across a real network and the artifacts are compressible: uncompressed archives, binaries with debug info, and text-heavy outputs commonly shrink 2-10x on the wire. It does little on same-rack links (the zstd CPU cost buys nothing when the wire isn't the bottleneck) and nothing for already-compressed artifacts. ## Steps
  • ## When it doesn't work NativeLink only advertises zstd when `remote_cache_compression` is true on the capabilities service **for that instance name**. An instance-name mismatch between the capabilities entry and the CAS or ByteStream entries is the usual cause. # https://docs.nativelink.com/how-to/stores/chunking-and-dedup **Content-defined chunking**: Cut remote cache transfer bytes by 80-90% for incrementally changing artifacts with the REAPI SplitBlob/SpliceBlob extension and Bazel's --experimental_remote_cache_chunking. **Who this is for:** anyone re-transferring large artifacts that change a little at a time: container layers, linked binaries, archives. **What you'll have at the end:** clients uploading and downloading only the chunks that changed. **Time:** twenty minutes. ## Requirements - **Bazel 9.1.1+ or 8.7.0+** on the client, with `--experimental_remote_cache_chunking`. Avoid 9.1.0: it has a client bug that corrupts outputs when the chunking flag is combined with `--disk_cache` (fixed in 9.1.1). - Chunking is **optional and off by default**. ## Enabling it Add an `experimental_chunking` block to the CAS service and give it a small store for chunk layouts. The index store must not verify content digests and must not be the CAS store itself (naming the same store for both is rejected at startup): ## When it helps, and when it doesn't Chunking pays off when clients reach the cache across a real network (WAN, metered links, cross-region) and artifacts change incrementally: uncompressed archives, linked binaries, and container layers typically save 80 to 90% of transfer bytes per change. ## Steps
  • ## When it doesn't work Bazel 9.1.0 has a client bug that corrupts outputs when the chunking flag is combined with `--disk_cache`. Use 9.1.1+ or 8.7.0+. # https://docs.nativelink.com/how-to/run-multiple-workers **Run multiple workers**: Add workers to a running NativeLink: how they share the CAS, how the scheduler matches them, and the two things about the shipped compose example that will mislead you. **Who this is for:** anyone with remote execution working against one worker who now wants several. **What you'll have at the end:** three workers pulling from one queue, and a clear idea of how to make it thirty. **Time:** forty minutes. ## The topology Three roles, three kinds of process, on one network: ## What a worker config looks like `servers: []` is not an oversight. A worker listens on nothing; it dials out to the scheduler's worker API and to the CAS. That is why the worker tier needs no ingress, no certificates and no load balancer, and why it is safe to run many of them on machines that are otherwise unreachable. ## How the scheduler picks a worker The scheduler will only send an action to a worker whose advertised `platform_properties` satisfy the action's platform requirements, and it will only consider property keys listed in its own `supported_platform_properties`: ## Scaling the pool Workers are stateless, so growing the pool is adding processes. Three things scale with them and are worth watching before you double the count. ## Steps
  • ## The shipped example, and where its README misleads `deployment-examples/docker-compose/` contains a working three-worker deployment: `docker-compose-multi-worker.yml` with `cas-server-multi-worker.json5`, `scheduler-multi-worker.json5` and `worker-shared-cas.json5`. Start there. Two things in and around it will send you the wrong way. ## When it doesn't work Almost always the platform properties. Compare the keys the worker advertises against `supported_platform_properties` on the scheduler and against what the client requests: a key the scheduler doesn't list can't be matched on, and a key the worker doesn't advertise excludes it from anything requiring that key. Check the scheduler log for the queue depth: actions queuing while workers idle is the signature. # https://docs.nativelink.com/how-to/tls-and-auth **TLS and authentication**: Put TLS on the listeners clients reach, use mutual TLS as the access control, and understand exactly what NativeLink does and does not authenticate. **Who this is for:** anyone about to expose NativeLink beyond a single trusted network. **What you'll have at the end:** encrypted listeners, client certificates as the gate, and an accurate picture of what is still unguarded. **Time:** an hour, most of it spent on certificates. ## The three listeners have different trust levels A production deployment exposes three surfaces, and they do not want the same treatment. ## Turn on TLS Every entry in `servers` may carry a `tls` block on its `listener.http`: ## Make the client prove who it is Adding `client_ca_file` changes the listener from encrypted to gated: ## Point clients at the new scheme Bazel needs the CA that signed the server certificate, and its own certificate if you turned on mutual TLS: ## What `experimental_identity_header` actually does You will find this on `ServerConfig`, and its name invites a wrong conclusion: ## Credentials to an upstream service When NativeLink proxies to something that does want a token (a hosted REAPI cache, a gateway in front of one), a `grpc` store can attach headers: ## Steps
  • ## When it doesn't work The key file holds more than one key (a file with none fails earlier, with `Could not extract key(s) from file`). A concatenated bundle is the usual cause. Split it, and leave the chain in `cert_file` where multiple PEM blocks are fine. # https://docs.nativelink.com/how-to/upgrade-versions **Upgrade to a newer version**: Read the changelog for the things that actually break, upgrade the three process types in the right order, and know what happens to the cache you already have. **Who this is for:** anyone running a pinned version who wants a newer one. **What you'll have at the end:** a rehearsed upgrade, a rollback that works, and your existing cache intact. **Time:** thirty minutes for the rehearsal, minutes for the upgrade itself. ## What is stable and what isn't The **client-facing API** is the Remote Execution API. It is a published, versioned protocol, and Bazel talking to NativeLink over `:50051` and `:50052` is the most stable surface here. You do not generally need to upgrade clients in step with the server. ## Read the changelog for four things `CHANGELOG.md` is generated per release, grouped into features, bug fixes, documentation, testing and CI, miscellaneous, and bumps and version updates, with a `compare/v1.6.4..v1.6.5` link in each release heading that shows every commit between the two. ## Rehearse the config before you touch production No `--validate` flag exists. The binary takes one positional argument, the config file (plus the usual `--help` and `--version`), and parses it at startup, which is exactly what you want, because the check is just "does the new binary accept this file": ## Upgrade in the right order
  • ## Rolling back Rollback is the same operation in reverse, with one asymmetry worth internalizing. ## When it doesn't work Exactly the failure this design intends. The message names the field. Find it in the new version's config reference. Usually it was renamed, occasionally it moved under a different parent, rarely it is gone because the feature it controlled is. Fix the file, do not try to make the parser lenient. # https://docs.nativelink.com/how-to/migrate-an-existing-cache **Migrate from another REAPI cache**: Put NativeLink in front of your existing cache as a read-only diode, let real builds warm it, and cut over when the hit rate says you can. **Who this is for:** anyone running Buildbarn, Buildfarm, bazel-remote or a hosted REAPI cache who wants to move to NativeLink without a flag day. **What you'll have at the end:** NativeLink serving your builds, backed by the old cache, with a cutover you choose rather than one you're forced into. **Time:** an hour to set up; days to weeks of running both, at your pace. ## The diode A `fast_slow` store whose `slow` half points at the old cache, marked read-only: ## What crosses the boundary and what doesn't **Digests must match.** If your old cache holds SHA-256 entries and your builds now ask for BLAKE3, nothing is found, not because of a bug but because the key of a blob is its hash, and you are asking a different question. ## Run both while you cut over Nothing forces a single switch date. Give the two caches separate config groups in `.bazelrc`, so a build can pick: ## Steps
  • ## When it doesn't work Check the digest function first; a mismatch produces exactly this and no error. Then check `instance_name`, since an old cache scoped to a named instance returns nothing for a differently-named one. Then confirm the address is reachable from the NativeLink process rather than from your laptop. # https://docs.nativelink.com/operate **Operate**: What to watch, what to tune, and what to do at 3am, once other people depend on your cluster. **Who this is for:** whoever gets paged when the build farm is slow. **What you'll have at the end of this section:** a production-shaped deployment, a metrics pipeline that tells you when it's unhealthy, and a written answer for each of the ways it fails. **Time:** ongoing. ## The shift this section represents Everything up to the How-to guides is about making NativeLink do what you want. This section is about making it keep doing that while you aren't looking. The questions change shape: not "how do I configure a filesystem store" but "what happens when that filesystem fills up overnight, how will I know, and what do I do about it." ## The arc **Get the shape right.** The production configuration is not the quickstart configuration with bigger numbers. It has a fast tier over a durable one, eviction policies that match your disk, and a deliberate split between the public listeners and the worker API. ## Start here, or skip ahead **Start here** if you're moving from a working setup to one people depend on. ## Current pages - [Production configuration](/operate/production-config): the three-process shape, the port surface, and the pre-launch checklist. - [Tuning](/operate/tuning): the lever table (the signal you see, the knob that moves it, and what moving it costs). # https://docs.nativelink.com/operate/production-config **Production configuration**: The shape a real NativeLink cluster runs in: three processes, a split port surface, and what to turn on before other people depend on it. **Who this is for:** you have NativeLink working and now other people are going to depend on it. **What you'll have at the end:** a three-process deployment whose ports, state, and failure behaviour you can explain to someone else. **Time:** an afternoon, plus however long your change process takes. ## The shape Three facts about that picture are the whole design: ## Process 1: the CAS and Action Cache This is the process that holds bytes. Two stores, one for content and one for action results, each with its own eviction budget: ### Moving off local disk `filesystem` is the right terminal store for a single CAS node with a real disk. For a CAS you intend to replicate, put an object store underneath instead and keep the local disk as the fast tier: ## Process 2: the scheduler The scheduler holds the queue and matches actions to workers. It reaches the CAS over gRPC rather than owning a store: ## Process 3: the workers Workers are the fleet you scale. Note `servers: []`: a worker serves nothing and needs no inbound port at all. It dials out to the scheduler and the CAS. ## Which ports are public `worker_api` is how a process claims to be a worker and receives commands to run. It is on its own listener specifically so that it can be on its own network policy. Anything that can reach `:50061` can register as a worker, receive your actions, and return whatever outputs it likes, which then land in your Action Cache and get served to everyone. Keep it inside the cluster. ## What's durable and what's disposable The rule this table encodes: **exactly one component is stateful.** If you find yourself needing to protect a scheduler's disk or a worker's disk, something has drifted from this shape. ## Before you go live ### Raise the file-descriptor limit Every open blob is a file descriptor. The binary gates file opens on a semaphore sized to 80% of the limit it actually achieved, so a CAS at the default limit does not error under load; it waits. ### Turn on namespaces on Linux workers Bazel in particular can leave zombie processes behind on a worker. Two opt-in flags contain that: ### Put TLS on the client-facing listeners Setting `client_ca_file` makes the listener require a client certificate signed by that CA, and `client_crl_file` lets you revoke one. That is the whole authorization model on the listener: **any client with a valid, unrevoked certificate gets full access to that listener's services.** No per-identity allow-list exists and no mapping from certificate subject to permissions. ### Point telemetry somewhere The config file has no metrics section, and the binary has no Prometheus scrape endpoint in the binary. Telemetry is OTLP over gRPC, configured by environment: ### Opt in to cache metrics, if you want them Execution metrics are emitted unconditionally. **Cache metrics are not**: a store only reports hits, misses, and errors on its reads, writes, and deletes if you wrap it: ## What to watch Four signals cover most of what goes wrong. All four are recording rules in : ## What to tune Once the shape is right, [Tuning](/operate/tuning) is the table of levers: the signal you observe, the knob that moves it, and what it costs you. ## What to do when it breaks [Troubleshooting](/operate/troubleshooting) indexes symptoms to causes, and [Runbooks](/operate/runbooks) has the procedure for each of the four incidents this shape actually produces. ## FAQ They are lost from the scheduler's point of view. The default scheduler holds its queue in process memory and nothing reconstructs it on startup. Clients re-submit, so no build fails, but work in progress is repeated. Restart schedulers when the queue is shallow. # https://docs.nativelink.com/operate/tuning **Tuning**: The lever table: the signal you observe, the knob that moves it, the direction to turn it, and what turning it costs. **Who this is for:** you have the [production shape](/operate/production-config) running and a number you don't like. **What you'll have at the end:** the specific knob for that number, and an honest account of what moving it costs. ## Storage and the CAS path ### The `evict_page_cache` lever The `filesystem` store has an opt-in `evict_page_cache` flag (default `false`). When enabled, after every blob it writes or reads it asks the kernel to drop that file's pages from the page cache (`posix_fadvise(POSIX_FADV_DONTNEED)`). ## Workers Worker count itself is not a lever on this page; it is a [scaling](/operate) decision driven by `nativelink:queue_depth`, not by worker CPU. ## Scheduler ## Process-level ## What is not a lever Worth stating, because each of these gets reached for and none of them helps: # https://docs.nativelink.com/operate/deploy-docker-compose **Deploy with Docker Compose**: Bring up a complete CAS, scheduler and worker on one host with the compose files in the repository: what each service is, which ports it publishes, and the four things that make this a development setup rather than a production one. **Who this is for:** you want the whole system (cache *and* remote execution) running on one machine, either to try it or to give a small team something shared. **What you'll have at the end:** three containers serving a real CAS, scheduler and worker, a build pointed at them, and a clear picture of what separates this from production. **Time:** 20 minutes, most of it spent compiling. ## The configuration The first run builds the image from source with Bazel, which takes a while; see [the build is not fast](#the-build-is-not-fast) below. ### Point a build at it Cache and executor are **different endpoints**. That is not an accident of the example: the CAS process and the scheduler process are genuinely separate here, exactly as they are in the [production shape](/operate/production-config). ## What to change Only the fields carrying a decision. Everything else in the shipped files is already correct for a single host. ## The multi-worker variant `docker-compose-multi-worker.yml` runs one CAS, one scheduler and three workers sharing a `cas-data` volume: ## What makes this a development setup Four things, all of them deliberate in the example and all of them disqualifying for production. ### The executor runs privileged `worker.json5` sets `use_namespaces: true` and `use_mount_namespace: true`, which need `CLONE_NEWUSER`/`CLONE_NEWNS`, hence the flag. Note what this buys and what it does not: namespaces here are a hermeticity and process-reaping mechanism, not a security boundary. ### The TLS certificates are expired examples The 50071 listener loads `example-do-not-use-in-prod-rootca.crt` and `example-do-not-use-in-prod-key.pem` from the same directory. They are committed to the repository, self-signed, `CN=localhost`, and **expired in October 2024**. Every client must pass `--insecure` or equivalent, which is what the repository's own TLS integration test does. ### Nothing is authenticated Not a Compose limitation: open-source NativeLink has no inbound authentication on any service, in any deployment. `docker-compose.yml` does not publish the scheduler's port 50061 at all; the worker reaches it over the Compose network by service name. That is the only thing standing between an attacker on your network and the ability to register as a worker. Keep it that way. ### The build is not fast The `Dockerfile` compiles NativeLink from the repository root with Bazel, in a three-stage build, and the final image is a fresh Ubuntu with the binary copied in and `curl` installed. The Ubuntu base is digest-pinned and the bazelisk download is checksum-verified, so the build is reproducible, but it is a full from-source build every time the layer cache misses. ## Troubleshooting ## What's next # https://docs.nativelink.com/operate/deploy-bare-metal **Deploy on bare metal**: Run NativeLink as systemd services on hosts you own: writing the units, sizing the machines, placing the disks, and rolling an upgrade without dropping in-flight actions. **Who this is for:** you have VMs or physical machines and no interest in running a cluster orchestrator to get a build cache. **What you'll have at the end:** three systemd units, a disk layout that will not surprise you, host sizing you can defend, and a restart procedure. **Time:** an afternoon. ## The configuration Three units, one per process. They are near-identical; the differences are the config file and, for the worker, the security posture. ### The service account and directories Put the configuration files in `/etc/nativelink/{cas,scheduler,worker}.json5`. ### The CAS unit ### The scheduler unit Identical apart from the config file and the fact that it needs no writable state at all when the scheduler backend is the default in-memory one: ### The worker unit This one is different, and the differences matter. ## What to change ## Disk layout The CAS host is the one where placement matters. ## Host sizing No sizing numbers are published for NativeLink; measure your own workload before buying anything. What the code does fix: ## Restarts and upgrades The three processes have genuinely different restart characteristics, and treating them the same is how a routine upgrade turns into a stack of failed builds. ## Troubleshooting ## What's next # https://docs.nativelink.com/operate/observability **Observability**: NativeLink pushes OTLP and nothing else. Wire it to a collector, get the series into Prometheus under the names the shipped rules expect, and know which environment variables actually do something. **Who this is for:** you are running NativeLink for other people and need to see queue times, cache hit rates and worker behaviour on a dashboard. **What you'll have at the end:** a collector receiving OTLP from every NativeLink process, Prometheus holding the series under the names the shipped recording rules query, and a Grafana dashboard drawing them. **Time:** 30 minutes. ## NativeLink pushes; it is never scraped The binary has no `/metrics` endpoint and no configuration that creates one. NativeLink builds three OTLP exporters at startup (logs, traces and metrics) and pushes to whatever endpoint the environment names. Every Prometheus-shaped thing in this pipeline happens downstream of the process. ## The configuration Point every NativeLink process at a collector. Two variables carry the whole decision: ## Bring up the pipeline Then point NativeLink at the collector and restart it: ## What happens if you skip the collector The `namespace: nativelink` setting on the collector's Prometheus exporter is what produces every metric name you have ever seen in a NativeLink dashboard. Send the OTLP stream anywhere else (another collector without that setting, an OTLP-native backend, or the shipped collector's own `otlphttp/prometheus` pipeline into Prometheus's receiver) and the names change: ## Turn on cache metrics Every family except cache metrics is emitted with no configuration. Cache metrics exist only for stores you explicitly wrap: ## Which metrics exist The authority is the [metrics reference](/reference/metrics), which is generated from `nativelink-util/src/metrics.rs` and its call sites. It records every declared instrument, its type and unit, its histogram buckets, and (the part no hand-written table stays honest about) **whether anything in the binary ever emits it**. ## Queries worth having Cache hit rate, per cache type: ## The Grafana dashboard A reference dashboard lives at . The Compose stack provisions it automatically. Elsewhere: Dashboards → New → Import, paste the JSON, pick your Prometheus data source. It draws execution throughput, success rate, active actions by stage and stage transitions from `nativelink_execution_*`; it has no cache, worker or gRPC panels. ## Troubleshooting ## FAQ No. The binary has no scrape endpoint, at any configuration, and the binary cannot push to Prometheus's OTLP receiver either, because that receiver is HTTP-only and NativeLink's exporters are gRPC-only. Run a collector and scrape its Prometheus exporter; the shipped collector also forwards into Prometheus's OTLP receiver, with the renaming described in [what happens if you skip the collector](#wha… ## What's next # https://docs.nativelink.com/operate/security-hardening **Security hardening**: Open-source NativeLink has no inbound authentication on any service. What that means for your network, which ports must never be routable, how to configure mTLS, and why the worker sandbox is not a security boundary. **Who this is for:** you are about to put NativeLink somewhere more than one person can reach it. **What you'll have at the end:** a network layout where the dangerous ports are unreachable, mTLS on the ports that must be reachable, and an accurate picture of the trust you are extending to anyone who can submit an action. **Time:** 45 minutes, plus however long your certificate authority takes. ## Start here: there is no authentication Open-source NativeLink authenticates no inbound request, on any service, in any deployment. Not the CAS. Not the Action Cache. Not the Execution service. Not the worker API. Not the admin endpoint. No bearer-token check, no API key, no mTLS-identity-to-permission mapping; the code that would do it does not exist in the tree. ## The port inventory Every listener NativeLink opens, what it serves, and who may reach it. Ports are the ones the shipped examples use; nothing in the binary reserves them; [Production configuration](/operate/production-config) covers how the three processes divide these listeners between them. ## The worker API has no authentication Port 50061 is the one that turns a cache into a compromise. A process that can open a gRPC connection to it can call `ConnectWorker`, be added to the scheduler's pool, and start receiving actions, including the compiler invocations, the secrets in your action environments, and the outputs that get written back into the CAS under digests your developers will later trust. ## The admin endpoint has no authentication The admin REST API registers under `/admin` by default. Every shipped example that enables it puts it on the worker API listener; one (`nativelink-config/examples/worker_with_redis_scheduler.json5`) also enables it on the public listener, which is the thing not to copy. Its only route is: ## The committed certificates are not usable Two directories in the repository contain a certificate and a key: ## Configure TLS TLS is per-listener. Each server in the `servers` array carries its own optional `tls` block. Every field below is documented in [the configuration reference](/reference/nativelink-config): ### Outbound TLS Outbound gRPC connections (a `grpc` store's `endpoints`, and the worker's `worker_api_endpoint`) configure their client side separately with a `tls_config` block on the endpoint. Object-store backends do not use it; they carry their own settings (for example `root_certificates` on the `ontap` provider). ## The sandbox is not a security boundary Workers can be configured with `use_namespaces: true` and `use_mount_namespace: true`. Both are real, and neither makes an action safe to run on your behalf. ## What to do, in order From a developer workstation, `nc -vz 50061` fails to connect. If it succeeds, stop and fix the network before anything else. ## Troubleshooting ## What's next # https://docs.nativelink.com/operate/scaling-workers **Scaling workers**: How much work one worker should take, how many workers you need, which signal tells you, and how to let something else make the decision, including the parts NativeLink does not enforce for you. **Who this is for:** you have a worker pool serving real builds and the queue is either backing up or sitting idle. **What you'll have at the end:** a sized worker, a scaling signal you trust, and either a manual playbook or an autoscaler wired to it. **Time:** an afternoon for the manual version, longer if you're building the autoscaling path. ## Two axes, and only one of them is a NativeLink knob Capacity has a vertical axis (how many actions one worker runs at once) and a horizontal axis (how many workers exist). NativeLink has a field for the first. It has nothing at all for the second, by design: a worker is a process that dials the scheduler and announces itself, so "more workers" means "start more processes," and whatever starts processes for you is the thing that scales. ## Sizing one worker The knob is `max_inflight_tasks` on the worker's config. ## The ceilings you hit before you expect to A worker sized generously runs into limits that are not on the worker config, and they're worth knowing before you go hunting. ## The signal to scale on Not CPU. Worker CPU tells you whether a machine is busy; it cannot tell you whether there is work waiting that nobody is doing, and that's the only question an autoscaler needs answered. ## Scaling out
  • ## What the repository actually ships Being explicit, because the gap is larger than the docs elsewhere imply. ## Backpressure, when a worker needs to stop One mechanism lets a worker refuse work without being removed: `experimental_precondition_script`. It runs before every action, and a non-zero exit produces a `ResourceExhausted` error. The scheduler requeues the action without counting it as an attempt, and marks the worker `is_paused` **only if that worker still has other actions in flight**; the pause clears as soon as one of them completes. ## Heterogeneous pools One scheduler can route to pools of different shapes; the mechanism is platform properties, and it is covered in full on [platform properties](/remote-execution/platform-properties). The scaling point is narrower: **each pool is a separate scaling unit with its own queue-depth series**, because `nativelink:queue_depth` groups by instance and priority, not by which pool could serve the work. ## When it doesn't work Actions are queued that no worker matches. This is nearly always a platform property mismatch: an action declaring something no worker advertises, or a scheduler key configured as `priority` (which constrains nothing) when it should be `exact` or `minimum`. Adding workers will not help. Check the scheduler's worker-match logging, and see [platform properties](/remote-execution/platform-properties). # https://docs.nativelink.com/operate/scaling-cas **Scaling the CAS**: How to grow the content-addressable store: composing tiers, sizing eviction, sharding, and the specific reasons a second CAS replica is harder than a second worker. **Who this is for:** you run a CAS that real builds depend on, and it is running out of disk, running out of throughput, or both. **What you'll have at the end:** a store composition sized for your load, an eviction policy that doesn't thrash, and a clear-eyed view of what running more than one CAS process costs. **Time:** an afternoon to size and tune; longer if you go horizontal. ## Start here: a CAS process is not stateless Most guidance about scaling services assumes replicas are interchangeable. For NativeLink's CAS that assumption is wrong in five specific places, and every horizontal-scaling decision on this page follows from them. ## Scale vertically first, by composing The store graph is where most capacity problems are solved, and it is cheaper than adding processes because it changes nothing about consistency. [Compose stores](/how-to/stores/compose-stores) is the reference for the wrappers; what follows is which one relieves which pressure. ## Eviction is your capacity control, and its default thrashes Four fields, all on `EvictionPolicy`, all defaulting to `0`, and `0` means *disabled* in every case. A policy with no fields set is an unbounded store. ### Size the AC and the CAS together The Action Cache is just another store with its own `EvictionPolicy`; there is no separate mechanism. The shipped examples give it roughly a twentieth of the CAS budget, which is a reasonable starting ratio because AC entries are tiny. ## Sharding, and what changing it costs `shard` distributes keys across several backends by hash, with an optional integer `weight` per shard. The weights are turned into a cumulative table over the `u32` space at startup and the key is binary-searched into it. ## Going horizontal Everything above scales one process. If you have exhausted that, here is the order to add replicas in. ## The ceilings you hit before you expect to **File descriptors, at 80% of what you configured.** `global.max_open_files` defaults to 24576, and the process reserves a fifth of it for sockets and pipes, so the effective permit budget is about 19660. ## Metrics: nothing is on by default This is the finding most likely to catch you out while you're trying to measure any of the above. ## What the repository actually ships Being explicit, because the gap between the examples and a production CAS is wide. ## When it doesn't work That's the `evict_bytes: 0` thrash. With no low watermark, every write pushes the store over the limit and re-enters the eviction loop to remove the minimum. Set `evict_bytes` to a real fraction of `max_bytes`, enough that a burst of writes fits under the headroom it creates. # https://docs.nativelink.com/operate/autoscaling-reference **An autoscaling reference deployment**: The complete Kubernetes deployment that scales workers on queue depth without a human in the loop: every manifest, the reaction-time budget nobody publishes, and the five gaps in the shipped examples you have to close first. **Who this is for:** you have run [scaling workers](/operate/scaling-workers) and [scaling the CAS](/operate/scaling-cas) by hand, you trust the numbers, and you want the pool to size itself. **What you'll have at the end:** a worker deployment that grows and shrinks on queue depth, with probes, drain, and disruption budgets wired up. **Time:** a day to assemble, a week of watching before you trust it. ## What ships, and what this page supplies Being blunt about this up front, because it changes how you read everything below. ## The reaction-time budget The signal an autoscaler reads is not emitted by NativeLink. It is derived, several hops downstream, and each hop adds staleness. Here is the whole chain with its shipped defaults. ## Five gaps you close first The shipped Kubernetes manifests cannot support an HPA as they stand. Not because of anything subtle, but because of five specific absences. Each one is a short fix and all five are prerequisites. ## The health endpoint does less than you expect Before you write a readiness probe, understand what `/status` actually measures, because the answer is surprising in both directions. ## Build the deployment
  • ## What this deployment does not do Worth stating so the gaps are choices rather than surprises. ## When it doesn't work Work backwards along the chain. Query the recording rule in Prometheus directly; if it returns nothing, the problem is upstream of Kubernetes entirely and the adapter is fine. If the rule returns data but the adapter doesn't, the usual cause is the `resources.overrides` block: external metrics still need a namespace association, and a series with no `namespace` label cannot be matched to one. # https://docs.nativelink.com/operate/runbooks **Runbooks**: What to do when a NativeLink deployment is on fire: Redis failover, a worker OOM-killed, the disk full, and the queue backpressured. Each entry is a symptom, the signal that confirms it, the immediate action, and the change that stops it recurring. **Who this is for:** you are on call and something is broken right now. **What you'll have at the end:** for each of the four incidents NativeLink actually produces, a confirming signal, a stop-the-bleeding action, and the configuration change that prevents a repeat. **Time:** read once now, so you do not read it for the first time at 03:00. ## A worker is OOM-killed The most common production incident, and the one with the most misleading first symptom: your developers report a *build* failure, not an *infrastructure* failure. ### Confirm it Two distinct failures land here and they need different responses. Work out which one you have before doing anything. ### Stop the bleeding Drain the affected worker rather than killing it, so its in-flight actions finish instead of being destroyed: ### Make it stop happening NativeLink itself has **no memory configuration**. No `max_memory`, no per-action memory cap, nothing. Memory is bounded only indirectly, by store eviction policies and by whatever the container runtime enforces. Do not go looking for a config field; it does not exist. ## The disk fills up ### Confirm it Writes start failing with an OS-level message wrapped in a NativeLink err_tip: `Failed to write data into filesystem store`, `Failed to flush in filesystem store`, `Failed to sync in filesystem store`. ### The signal to look for first If your eviction snapshot says **"of unlimited"**, no size-based eviction is configured, and this incident was inevitable. `eviction_policy` is an `Option` on the filesystem store, and every field in it defaults to `0`, which means *never evict*. The config's own doc comment is blunt about it: failure to set this value causes items to never be removed. ### Stop the bleeding Set `max_bytes` and restart the process. Eviction is only ever invoked on map mutation (on insert and remove), so it runs as soon as traffic resumes. ### Size the volume for twice `max_bytes` The filesystem store keeps an executable-variant directory (`{content_path}.exec`) alongside the content path, and **that directory is invisible to `max_bytes`**. A variant is deleted when its primary entry is evicted, which bounds the total at roughly `2 * max_bytes` in the worst case (every blob also held as an executable). ### Two layout footguns `temp_path` must be on the **same block device** as `content_path`. If it is not, the atomic rename becomes a copy: slower, and it doubles peak space during every write. ## Redis fails over ### What actually happens Both Sentinel and Cluster modes are supported; the default is `standard`. On a Sentinel failover the demoted master answers `-READONLY`, which NativeLink classifies as retryable alongside dropped connections, refusals and IO errors. ### What breaks depends on your backend ### Stop the bleeding If you are on the `memory` backend, this is a cache-availability incident, not a scheduler incident. Builds degrade to local execution and cache misses. Fix Redis at your leisure. ### Configuration worth checking now `response_timeout_s` and `connection_timeout_s` are **deprecated**; `response_timeout_s` is ignored entirely. Both log a warning at startup. Setting both `connection_timeout_s` and `connection_timeout_ms` is a startup error. ## The queue backpressures and never recovers The subtlest incident of the four, because nothing errors and nothing restarts. Actions re-queue forever. ### The mechanism `ResourceExhausted` is NativeLink's backpressure code. When a worker returns it for an action, two things happen: the action goes back to `Queued` and (this is the important part) **its attempt counter is not incremented**. If the worker still has other actions in flight it is also marked `is_paused`, and that flag clears as soon as any of those actions completes. ### Confirm it then look at the scheduler's periodic pass: the action keeps appearing in `Oldest actions in state` as `Queued`, its age grows, and it never reaches a retry limit. The worker itself is quiet; the script's exit status is logged only at `TRACE` (`Preconditions script returned`), so `RUST_LOG=info` on the worker shows nothing. ### Stop the bleeding Fix or remove the precondition script and restart the worker. Nothing can clear `is_paused` from the outside; it is set and cleared by the scheduler in response to what the worker returns. ### Related limits that block rather than reject Several NativeLink limits queue instead of erroring, which is why "hung" is a more common symptom than "failed": ## Restarting the scheduler Not an incident so much as a procedure that behaves worse than you expect. ## What's next # https://docs.nativelink.com/operate/troubleshooting **Troubleshooting**: A symptom-to-cause-to-fix index for NativeLink, anchored on the error strings the system actually emits, including the ones that say nothing useful, and the failures that produce no error at all. **Who this is for:** something is wrong and you have a symptom but not a cause. **What you'll have at the end:** the cause, and either a fix or a pointer to the procedure that fixes it. **Time:** find your row, follow it. ## The index ## Actions queue forever and workers sit idle The single most important asymmetry in NativeLink: ### Diagnose it Raise the log level. Every useful diagnostic here is at `INFO`, and the shipped examples run at `warn`. ### Property types that constrain nothing A scheduler key configured as `priority` when it should be `exact` or `minimum` is the classic version of this bug, and it fails in the *opposite* direction: actions get scheduled onto workers that cannot run them. See [Scaling workers](/operate/scaling-workers) for the fleet-shaped version of this problem and [Platform properties](/remote-execution/platform-properties) for the full matching semantics. ## Actions fail with a CAS miss ### The string you will actually see Bazel reports `FAILED_PRECONDITION`. Not `NOT_FOUND`. That translation happens on the worker, which checks for a `NotFound` whose message contains `not found in either fast or slow store` and converts it before returning: ### The two causes **Workers do not share CAS storage.** The fix is exactly what the message says: every worker's `fast_slow` slow tier must resolve to the same CAS the clients upload to. In Compose, `docker-compose-multi-worker.yml` does that by pointing each worker's slow store at the `cas-server` over gRPC (the `cas-data` volume it also mounts into the workers is not what shares the data); in Kubernetes, a shared backing store rathe… ### A near-miss worth knowing Zero-byte outputs are not stored. Callers materialise them directly. If you see this in a stack trace it is a caller bug, not a cache problem. ## The timeout message that tells you nothing This means the client stopped waiting, and `client_action_timeout_s` (default 60 s) cleaned the operation up. It is a statement about the *client*, not about why the action never ran. ## Timeouts The timeout defaults that matter: `max_action_timeout_s` 20 minutes, `max_upload_timeout_s` 10 minutes, `max_cleanup_wait_s` 30 seconds, `worker_timeout_s` 5 seconds, `client_action_timeout_s` 60 seconds, `retain_completed_for_s` 60 seconds. ## Zero means default Setting a field to `0` to disable it usually does the opposite. ## Worker connection churn A worker that logs at a steady ~2 Hz is reconnecting: ## Limits that block instead of erroring If the symptom is "everything got slow and nothing errored", you are probably sitting on one of these: ## Startup failures These all abort the process before it serves anything. All of them are config wiring, and all of them name the offending key. ## Retries Retry exhaustion is **not** a distinct error. It is the last underlying error with a suffix appended: ## Where else to look [Scaling workers](/operate/scaling-workers) has a "when it doesn't work" section covering the fleet-shaped versions of several problems above: queue depth high while workers idle, adding workers not helping, scale-in causing failed actions, two workers on one host behaving erratically. It complements this page rather than duplicating it. ## What's next # https://docs.nativelink.com/explanations/architecture **Architecture**: The four roles, who actually writes to the action cache, and which parts of the system hold state you can lose. **Who this is for:** anyone deciding how to deploy NativeLink, or trying to predict which component's failure costs them what. **What you'll have at the end:** the four roles, the path an action takes on a miss and on a hit, and an accurate answer to "what do I have to back up". **Time:** fifteen minutes. ## The four roles ### CAS: content-addressable storage Every byte NativeLink touches lives in the CAS, keyed by a digest of its content. Source files, object files, binaries, stdout, all of it. Two identical files anywhere in your organisation collapse into one stored blob, because they hash to the same key. ### AC: action cache The action cache maps `action digest → ActionResult`. The action digest covers the command line, every input file's digest, the environment variables, and the platform properties, so an AC hit means *this exact computation has been performed before, and here is where its outputs are*. ### Scheduler The scheduler receives `Execute` RPCs, decides which queued action goes next, finds a worker whose platform properties satisfy the action's, and assigns the work. It also merges duplicate requests, so two developers building the same commit wait on one execution rather than two. ### Workers A worker fetches the action's inputs from the CAS, materialises them into a directory, runs the command, uploads the outputs, and reports back. ## One build, twice The concrete version. A developer runs `bazel build //app:server`, which expands to roughly two thousand actions. Take one of them: compiling `main.cc`. ## What holds state Deployment planning comes down to this table, and one widespread claim about it is wrong. ## Why Rust, why this shape Three properties drove the design, and all three are operational rather than aesthetic. ## Common questions The CAS, and only the CAS. The AC is a lookup table that refills itself. Workers hold nothing durable. The scheduler holds in-flight state that is worth nothing after a restart anyway; the clients will retry. # https://docs.nativelink.com/explanations/architecture-deep-dive **Architecture deep dive**: The crate graph, the startup sequence, and the path an action actually takes through the binary. **Who this is for:** anyone about to read, extend or debug the NativeLink source, and anyone who needs to predict what the binary does rather than what the protocol says it should. **What you'll have at the end:** the crate graph, the order things come up in at startup, and where an action's path forks. **Time:** fifteen minutes. ## One binary, five decisions `nativelink` is a single executable. Which roles it plays is decided entirely by the config file it is handed; there are no build features to toggle and no separate images per role. Everything below happens inside . ## The crate graph Twelve library crates plus the binary, in strict layers. The diagram shows the ten that give the program its shape; the other two are `nativelink-metric-macro-derive` (the proc macro behind `nativelink-metric`) and `nativelink-redis-tester` (a fake Redis used by the store crate and by tests). `nativelink-macro` is a test-only proc macro that no library crate depends on. ## Configuration is JSON5, and it is validated early Config parsing is JSON5, not JSON: comments and trailing commas are legal, which is why every example in these docs uses them. The entry point is `CasConfig::try_from_json5_file` in . ## Stores are built in two phases `store_factory` constructs each configured store, and then a separate `run_post_init()` pass runs across all of them. The second pass exists because a `ref_store` names another top-level store (possibly one declared later in the file), and a single-pass construction would make declaration order load-bearing. ## Services come up in a fixed order The registration order is: action cache, CAS, execution, `Operations`, Remote Asset fetch, Remote Asset push, ByteStream, capabilities, worker API, and the experimental Build Event Protocol service. Each is registered only if the config asks for it. ## Where an action's path forks The single most important structural fact about execution is that **a cache hit never reaches the scheduler**. Bazel calls `GetActionResult` before it calls `Execute`, so on a hit the execution service is never contacted at all. ### The pre-flight check is deliberately racy Before an `Execute` is accepted, the execution service does a shallow check that the action's inputs are present: the `Action` proto itself, its `command_digest` and its `input_root_digest`. It is documented in-source as deliberately shallow and deliberately racy: it does not walk the full input tree, and a blob can be evicted between the check passing and the worker asking for it. ## Two details that surprise people reading the wire **`GetTree` page tokens are `"{hash}-{size}"`.** They are constructed and parsed by splitting on `-`. They are not opaque, and they are not signed, though the protocol says clients should treat them as opaque, and clients that do will keep working if this ever changes. ## Common questions Because the roles have different scaling and failure characteristics, not because the code requires it. A CAS wants durable storage and steady memory; a worker wants CPU and is disposable; a scheduler wants to be restarted freely. Splitting them lets you size and autoscale each one separately. See [production configuration](/operate/production-config). # https://docs.nativelink.com/explanations/store-model **The store model**: What a store actually is, how composition behaves, and where a digest stops being a claim and becomes a fact. **Who this is for:** anyone composing stores in a config file and wanting to predict the result, and anyone writing a new store backend. **What you'll have at the end:** the trait, the key space, the composition rules, and the one place verification actually happens. **Time:** twenty minutes. ## Three names, one file `Store`, `StoreLike` and `StoreDriver` all live in , and the split between them matters when you read the source. ## The key space is not just digests `StoreKey` has two variants: `Digest(DigestInfo)` and `Str(Cow)`. Most of NativeLink deals in the first. The second exists because two subsystems need to store things that are not content-addressed at all: the Build Event Protocol service, and the scheduler's awaited-action database. ## The digest on the wire is a claim, not a measurement This is the single most important thing on this page. ### How the verify store actually works The implementation in ## Composition, and the barriers in it `inner_store` is how the system looks *through* a wrapper to find an optimisation opportunity, for example, to discover that the real backing store is a filesystem store and a hardlink is possible. ### `fast_slow` writes through, and deliberately re-uploads With the default `fast_direction` and `slow_direction`, writes go to both tiers concurrently and **both must succeed**. It has no write-behind and no eventual consistency window: an upload that returns OK is durable in the slow tier. ### `dedup` is a chunking index, and it has opinions A `dedup` store splits blobs with content-defined chunking (FastCDC), stores the chunks, and writes an index under the **original** key. Two large blobs that share most of their content share most of their chunks. ## Two things that are not what their filenames say and ## Where the exact truth lives Every store type, every field and every default is in the generated [store configuration reference](/reference/nativelink-config/store-overview). This page deliberately does not restate field names, because the reference is generated from the Rust types and this page is not. ## Common questions On the write path of anything shared between people, yes. The cost is one hash pass over uploaded bytes, concurrent with the write rather than serialised before it. The thing it buys is that a digest in your cache means what it says. On a purely local single-developer cache the calculation is different, because you are the only client and you are not defending against yourself. # https://docs.nativelink.com/explanations/scheduler-internals **Scheduler internals**: Action merging, the awaited-action database, how a worker is chosen, and what happens when an action fails. **Who this is for:** anyone tuning a scheduler, debugging an action that queued forever, or wondering why two identical builds did not share work. **What you'll have at the end:** the merge rule, the ordering rule, the matching rule, and the retry rule, each of which has at least one counter-intuitive edge. **Time:** twenty-five minutes. ## No `ActionScheduler` trait exists If you go looking for one (and older material about NativeLink will send you looking) you will not find it. The abstraction is split by *who is asking*, into three traits in : ## When are two actions the same action? Two clients asking for the same work should wait on one execution. The key that decides this is `ActionUniqueKey`, and it has exactly three fields: instance name, digest function, and the action digest. ## Ordering is one 64-bit integer `AwaitedActionSortKey` packs the entire queue ordering into a single `u64`: the upper 32 bits are the priority (shifted from `i32` into unsigned range so byte order matches numeric order), and the lower 32 bits are the insertion timestamp in whole seconds XORed with `u32::MAX`. ## Two state enums, and one that never appears `ActionStage` is the wire-facing state a client sees. `SortedAwaitedActionState` is the database index state the scheduler sorts on. They overlap but are not the same, and conflating them while reading the source is a reliable way to get lost. ## Matching is first-fit over an LRU The matching engine walks the queue in sort order, and for each action looks for the first worker that satisfies the action's platform properties. The source still carries an `O(n*m)` TODO in , but the walk is narrowed first: a `WorkerCapabilityIndex` (an inverted index over `Exact` and `Priority` properties) produces the candidate set, and only that set is scanned in LRU order, checking availability and `Minimum` va… ### Platform properties are typed, and `Minimum` is a resource pool Every property key a worker advertises has a declared type: ## Retries, and what "failed" means The attempt counter is incremented by two different events, with different rules. ### Two timeouts with very different defaults `max_action_executing_timeout_s` **defaults to `0`, which means disabled.** It is not the only guard against a hung action: the worker kills the process when the action's own timeout (or the worker's `max_action_timeout_s`, 20 minutes by default) elapses, and the scheduler re-queues an executing action whose worker stops sending updates for `worker_timeout_s`. ## Who writes the action cache entry? Not the scheduler. **The worker writes the action cache entry, and it does so before telling the scheduler that the action finished.** The ordering is what guarantees that a client which observes completion and then queries the cache finds the result there. ## Where the exact truth lives Every scheduler field and default is in the generated [scheduler configuration reference](/reference/nativelink-config#namedschedulerconfig). For the operational levers, see [tuning](/operate/tuning). ## Common questions Check the three fields in the merge key first: instance name, digest function, action digest. A different instance name is the usual answer, and the second most common is one client sending `skip_cache_lookup`, which opts that action out of merging entirely. # https://docs.nativelink.com/explanations/worker-execution **Worker execution**: How an action becomes a running process: input materialization, what isolation you actually get, and what the worker throws away. **Who this is for:** anyone running workers, debugging an action that behaves differently on a worker than locally, or deciding how much to trust the actions they run. **What you'll have at the end:** the lifecycle, the filesystem layout, the exact isolation boundary, and the list of things the worker deliberately discards. **Time:** twenty-five minutes. ## The connection is one stream A worker opens exactly one bidirectional gRPC stream to the scheduler (`ConnectWorker`, the sole RPC on NativeLink's own `WorkerApi` service), and everything flows over it. It has no polling loop and no second connection. ## The filesystem, and the hardlink that everything depends on On startup the worker **purges and recreates its `work_directory`**. Anything left there by a previous run is gone. This is intentional (a worker that inherits state is a worker whose actions are not hermetic), but it means the work directory must not be somewhere you also keep things. ## What isolation you actually get This section is deliberately blunt, because overclaiming here is dangerous. ## The environment is built from nothing The worker calls `env_clear()` first. Nothing is inherited. Variables are then applied in this order: ## Timeouts, signals, and the output you lose A requested timeout above `max_action_timeout_s` (20 minutes if unset) is **rejected with `InvalidArgument`, not clamped**. The client is told its request was unacceptable rather than silently receiving a shorter deadline than it asked for. That is the right choice, and one worth knowing before you set a low ceiling on a cluster whose clients ask for long ones. ## Uploading outputs, and what is dropped The ordering here is subtle and matters for anyone reasoning about consistency. When the command exits, the worker sends `ExecuteComplete` to the scheduler (the process is done, so a new action can be assigned) and *then* uploads outputs, deletes the action directory, and writes the action cache entry. ## Cleanup, and three things called eviction The entire action directory is deleted after every action, with a `Drop` backstop for the panicking case and a path-canonicalization check so that a malicious or malformed operation ID cannot direct the delete outside the work directory. ## Where the exact truth lives Every worker field and default is in the generated [worker configuration reference](/reference/nativelink-config#workerconfig). The execution code itself is ## Common questions It is almost certainly modifying an input file in place. Inputs are hardlinks to read-only CAS blobs. Copy the file into the output tree first, or declare it as an output rather than an input. # https://docs.nativelink.com/explanations/correctness-hermeticity **Correctness and hermeticity**: What content addressing guarantees, what it doesn't, and which of the gaps are yours to close. **Who this is for:** anyone who needs to know how much to trust a cache hit, and anyone chasing a build that is correct on one machine and wrong on another. **What you'll have at the end:** an honest inventory of the guarantees: the ones you get for free, the ones you have to configure, and the ones nobody is offering. **Time:** twenty minutes. ## What you get for free **Collision resistance.** SHA-256 and BLAKE3 are the supported digest functions. Two different blobs will not share a key. ## What you do not get by default Five gaps, in rough order of how often they bite. ### The digest is a claim until you verify it The server does not hash plain uploaded bytes. The client declares the digest and the server files the bytes under it. (Zstd-compressed ByteStream uploads are the exception: the decoder checks size and hash as it decodes.) For everything else the only thing that checks is a `verify` store, and both `verify_hash` and `verify_size` default to `false`. ### Verification is write-side only Even with a `verify` store, nothing is re-checked on read. Blobs that entered before you added the wrapper, or through a path that bypassed it, are served without complaint. ### Hermeticity is the client's job The action digest covers exactly what the client put into the `Action` message: the command, the input tree, the platform properties, the environment variables it declared. Anything the action reads that is not in there is invisible to the cache, and a cache hit will confidently return a result computed with a different value of it. ### Actions can run more than once The default `max_job_retries` of 3 is checked strictly, so an action may execute **four** times. A worker disconnect always costs an attempt; a `ResourceExhausted` never does. ### Some output metadata is discarded `node_properties` is never populated, so output file modes and mtimes do not survive a round trip. Absolute output symlinks are dereferenced and uploaded as content; only relative ones stay symlinks. A declared output that the action did not produce is not an error. ## Caching semantics that surprise people Three, all of which have bitten someone: ## A checklist for a cache you can trust
  • **Put `verify_hash: true` in the write path.** Everything below is downstream of this being true.
  • **Pin the toolchain at the client**, so the compiler's identity is inside the action digest rather than beside it.
  • **Make undeclared inputs fail rather than succeed.** Strict action environment, strict include checking, no reliance on ambient `PATH`. ## Common questions Build the same target on two machines that differ, and compare outputs byte for byte; anything that differs is reading something undeclared. At cluster scale the symptom is subtler: cache misses where you expected hits, because a leaked input changed and moved the action digest with it. A rising miss rate on unchanged code is the signal. # https://docs.nativelink.com/explanations/lre **Local Remote Execution**: What LRE actually is: a way to make your local toolchain and your remote workers the same toolchain, so both sides share one cache. **Who this is for:** anyone deciding whether LRE is worth adopting, and anyone who has read about it and formed the wrong model of what it does. **What you'll have at the end:** the actual mechanism, the guarantee it makes and its exact qualifiers, and what it does not cover. **Time:** fifteen minutes. ## The mechanism, precisely The trick is absolute paths. ## The guarantee, with its qualifiers intact The source states the promise carefully, and the qualifiers are load-bearing: ## What is pinned, and what is not This is where careful reading pays off. ## Coverage Windows is not supported. On macOS the C++ path has no local toolchain at all (the flake module only emits the C++ platform and toolchain flags on Linux), so an `lre-cc` action can only run on a Linux worker carrying the image; you get hermeticity there, not locality. ## Why Nix, specifically Because the design needs every tool the build invokes to be identified by a hash of everything that produced it, and to live at a path that encodes that hash. That is precisely what a Nix store path is. ## When it is worth it ## Where the exact truth lives The flake-side and Bazel-side wiring, the regeneration commands, and the downstream-project setup are in . For actually running it (the commands, and what is local versus remote once you leave `x86_64-linux`) see [toolchains and hermeticity](/remote-execution/toolchains-and-hermeticity), which is in the Remote execution section. ## Common questions No, and it does not try to. LRE is about toolchain identity, not about where work runs. Fan-out across hundreds of cores still wants remote workers; the point is that the workers and your laptop now agree on what a compiler is, so both populate one cache. # https://docs.nativelink.com/explanations/history **History and licensing**: What NativeLink was built to replace, the three properties that drove the rewrite, and exactly how the source is licensed. **Who this is for:** anyone evaluating NativeLink against an existing build farm, and anyone who needs to answer a licensing question precisely. **What you'll have at the end:** the design pressures that produced the current system, and a file-by-file account of what license applies. **Time:** ten minutes. ## What the existing options traded away Two of those three problems are runtime problems rather than design problems. That observation is the whole argument for the rewrite: the protocol was fine, the storage model was fine, and what was expensive was the substrate. ## The three properties that drove the design **Predictable latency.** In a build system, the p99 is the number developers experience: a single stalled action blocks everything downstream of it in the graph. Removing the garbage collector removes a class of latency spike that is very hard to tune away rather than eliminate. ## Licensing, precisely This question gets answered vaguely often enough to be worth stating exactly. ## Versioning NativeLink versions look like SemVer and are not. ## Why these docs were rewritten The previous documentation accumulated during the period when the project was changing shape weekly. It described commands that no longer existed, components that had been renamed, and performance numbers with no source. ## Common questions It is source-available today and open source on a timer. The FSL 1.1 grant permits everything except offering NativeLink itself as a competing commercial service, and each release converts to Apache 2.0 two years after publication. If your concern is "can I read and modify the code that runs my builds", the answer is yes, without qualification. # https://docs.nativelink.com/reference/nativelink-config **Configuration reference**: Every knob in the NativeLink JSON5 configuration — types, defaults, and links to source, autogenerated from the Rust config crate. _Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/nativelink-config._ # https://docs.nativelink.com/reference/nativelink-config/store-overview **Store overview**: The mental model behind every store in the configuration reference: what holds data, what wraps another store, and how they compose. The [full reference](/reference/nativelink-config) lists every field on every store. This page is the map you need before that list is useful: what a store actually is, the two families every store falls into, and the compositions people actually run in production. ## What a store is A store is a named, pluggable backend for CAS or AC data. You declare each one once in the top-level `stores` array, then reference it by name from `servers`, `workers`, or from inside another store: ## Two families Every store type is either **terminal** (it actually holds or fetches bytes) or a **wrapper** (it adds behavior in front of another store, which it embeds as a nested `StoreSpec`). Wrappers nest arbitrarily deep: a `verify` can wrap a `compression`, which wraps a `fast_slow`, which wraps two more terminal stores: ### Terminal stores: where bytes actually live Every implementation in this table lives in `nativelink-store/src`. ### Wrapper stores: behavior layered on a backend The nested-`StoreSpec` shape each wrapper takes (one backend, two named backends, or N) is declared alongside the terminal specs. ## Choosing a terminal backend ## Common compositions **Single-node dev cache**: no wrapping at all. See [Configuration → Stores](/configuration/stores). ## FAQ No; it never checks that an object in `fast` also made it to `slow`. If an artifact must survive a fast-tier wipe, make sure the write path covers both tiers deliberately. ## What's next - [Configuration reference](/reference/nativelink-config): every field, default, and JSON5 example for every store type above. - [Configuration](/configuration): how stores fit alongside servers, schedulers, and workers. - [Production configuration](/operate/production-config): the store composition a real cluster uses end to end. # https://docs.nativelink.com/reference/protocol-api **Protocol and API surface**: Every gRPC service NativeLink serves, which RPCs are implemented, and the values it advertises. **Who this is for:** anyone writing or debugging a client, or checking whether a given REAPI feature is available. **What you'll have at the end:** the complete service list with implementation status. **Time:** use as a lookup. ## Services Each is registered only if the configuration asks for it. Registration order at startup is fixed: action cache, CAS, execution, `Operations`, fetch, push, ByteStream, capabilities, worker API, BEP. ## `WorkerApi`: the only NativeLink-defined service Defined in . It has **exactly one RPC**: ## Partially implemented RPCs Three `Operations` RPCs are registered and return `unimplemented`: ## What `Capabilities` advertises Two of these need a caveat. ## Wire-level details worth knowing **ByteStream resource names are parsed right to left.** A resource name is `{instance_name}/blobs/{hash}/{size}`, and the instance name may itself contain slashes, so left-to-right parsing cannot find the boundary, while right-to-left can, because the trailing components are fixed. See . ## Related - [Architecture deep dive](/explanations/architecture-deep-dive): where each service sits in the process - [Scheduler internals](/explanations/scheduler-internals): what happens between `Execute` and a worker - [Worker execution](/explanations/worker-execution): the `WorkerApi` stream from the worker's side - [Configuration reference](/reference/nativelink-config): which services a given config enables # https://docs.nativelink.com/reference/metrics **Metrics reference**: Every OpenTelemetry instrument NativeLink declares, with its type, unit, attributes, Prometheus series, and whether the binary actually emits it. Autogenerated from the Rust source and its call sites. _Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/metrics._ # https://docs.nativelink.com/reference/cli-and-env **CLI and environment**: Every command-line argument the nativelink binary accepts and every environment variable it reads: the telemetry variables, the ones the configuration expands, and the ones it passes through to actions. **Who this is for:** you are writing a systemd unit, a Dockerfile `CMD`, or a Kubernetes `env:` block and need to know what the process actually reads. **What you'll have at the end:** the complete argument and variable surface, with the sharp edges marked. ## The command line That is the whole thing. `config_file` is a required positional argument, the path to a JSON5 configuration file, and there are no other arguments. The binary is a [clap](https://docs.rs/clap) parser with `version`, `about` and `author` derived, so `--help` and `--version` work and nothing else does. ## Telemetry variables These are read once, at startup, by `nativelink-util`'s tracing initialiser. They are the only variables that change the binary's own behaviour. ## Variables the configuration file expands Many configuration fields are deserialized through a shellexpand wrapper rather than directly, so their string values undergo environment substitution before the value is used. ## Variables the worker passes to actions A worker does not forward its own environment to the commands it runs: the child is spawned with `env_clear()`. An action sees exactly two things: the `environment_variables` its own `Command` message carries, and whatever the worker's `additional_environment` map (on the `local` worker configuration) names explicitly. ## What NativeLink does not read Stated explicitly, because each of these is a reasonable guess that is wrong: ## Reading the source If anything here disagrees with the binary, the source wins: ## Where to go next # https://docs.nativelink.com/reference/oss-and-enterprise **Open source and Enterprise**: What the open-source distribution actually contains, which two modules are licensed differently, why nothing in the binary enforces that, and what the paid tiers are for. **Who this is for:** anyone who has to answer "are we allowed to run this?", and anyone deciding between self-hosting and NativeLink Enterprise. **What you'll have at the end:** an accurate picture of what is licensed how, what activates it, and what the open-source build does and does not restrict. **Time:** ten minutes. ## The short version The whole of NativeLink is in the public repository. No separate "enterprise build", no stripped-down community edition, and no compile-time feature flag that divides them. What differs between the free and paid product is the *licence you hold*, not the *code you run*. ## Two licences, one repository The repository root carries `LICENSE`, the Functional Source License 1.1 with an Apache 2.0 future grant. Its restriction is narrower than most people expect on first reading: it permits everything except a **Competing Use**, which the licence defines as making the software available to others in a commercial product or service that substitutes for NativeLink, substitutes for something Trace Machina already offers us… ## The Business Source Licensed modules Five files in the tree carry a Business Source License 1.1 header instead of the repository licence. They form two modules: ## Nothing in the binary enforces any of this No licence key field exists in the configuration schema. No startup validation reads one. No runtime gate wraps either module. Searching the whole tree for enforcement turns up exactly five hits: the five header comments listed above, and nothing else. Both modules are compiled unconditionally and both run by default. ### Metrics are always on No configuration turns instrumentation off. The meter provider is constructed unconditionally during startup and installed globally, and the scheduler and store layers call into the licensed instruments on their normal paths: `simple_scheduler_state_manager.rs` records completed executions, `memory_awaited_action_db.rs` records stage transitions, and `cache_metrics_store.rs` records every cache operation. ### Persistent workers activate from the action, not from your config No `persistent_worker` key exists anywhere in the configuration schema; searching `nativelink-config` for it returns nothing at all. The module is reached entirely from the action's own platform properties: ## What the open-source build does not limit Nothing in the code artificially constrains scale. Specifically, there is no cap on cache size, stored object count, action throughput, connected worker count, or number of people using the cluster; no trial period, expiry date, or time-limited mode; and no registration, activation, or account required to start the binary. ## What Enterprise adds The honest answer, from inside this repository, is that the paid tier is not a different codebase, so this documentation cannot enumerate its feature list by reading the source. What can be said accurately: ## Telling what you are running The binary is a [clap](https://docs.rs/clap) parser with `version` derived from the crate, so: ## FAQ Running it as your own build infrastructure is internal use, which the Functional Source License permits regardless of whether your company is commercial. The separate question is the two Business Source modules: metrics in a shared or production setting needs a licence, and metrics cannot be switched off. # https://docs.nativelink.com/reference/glossary **Glossary**: The vocabulary you need to read the rest of the docs. **Who this is for:** anyone reading the rest of these docs who hit a word that was used as if it needed no explanation. **What you'll have at the end:** the one sentence you needed, and a link to the page that goes deeper. **Time:** use as a lookup. ## Action A single unit of build work. Encodes a command line, input file digests, the platform requirements, and the expected outputs. Hashing an action produces a stable identifier; identical actions have identical hashes. ## Action Cache (AC) A keyed store: `hash(Action) → ActionResult`. Cache hits skip the work entirely. ## ActionResult What an action produces. Output file digests, exit code, captured stdout/stderr, timing metadata. ## Awaited action The scheduler's record of one action between `Execute` and completion: its current [execution stage](#execution-stage), its priority, and every client waiting on it. Two identical actions submitted at the same time join the same awaited action instead of running twice. ## ByteStream API The streaming transport for blobs too large to move in a single `BatchUpdateBlobs` or `BatchReadBlobs` call, with the [digest](#digest) encoded in the resource name. Where the batch calls stop and this one starts is a server limit the [Capabilities API](#capabilities-api) advertises. ## Capabilities API The RE-API service a client calls first, to learn which [digest functions](#digest-function) the server accepts and what its size limits are rather than assuming. See [protocol and API surface](/reference/protocol-api). ## CAS: Content-Addressable Storage Stores every blob (source file, intermediate output, final binary) under the digest of its contents. The same byte sequence stored from anywhere collapses to one entry. ## Completeness-checking store An [Action Cache](#action-cache-ac) wrapper that checks every output [digest](#digest) still exists in the CAS before serving a hit. Without it a CAS eviction turns old cache entries into hits that reference blobs nobody can download, and the build fails while fetching outputs instead of quietly missing the cache. Valid only on AC stores. ## Dedup store A store that splits blobs into content-defined chunks with FastCDC () and stores each chunk once, so two versions of a large file that differ in the middle share everything else. The cost is an index lookup per read. See [the store model](/explanations/store-model). ## Digest A content hash plus a size. Used everywhere instead of file paths. ## Digest function The hash a client and server agree on for [digests](#digest): SHA-256 or BLAKE3, negotiated per request rather than fixed at build time (). Two clients using different functions produce different keys for the same bytes, so they do not share cache entries. ## Eviction policy The limits a store evicts against: `max_bytes`, `max_seconds`, `max_count`, and the `evict_bytes` low watermark that stops eviction thrashing right at the limit. Each defaults to `0`, which means *never evict on that dimension*: a store configured without a policy grows until the disk does. ## Execution stage Where an action is in its life: `CacheCheck`, `Queued`, `Executing`, then `Completed` or `CompletedFromCache` (). An action stuck in `Queued` has no free [worker](#worker) whose [platform properties](#platform-properties) match what it asked for. `CacheCheck` is declared but the scheduler never reports it, because the cache lookup happens in a wrapper in front of the scheduler. ## Existence cache store A store that remembers which digests it has already confirmed exist, so repeated [`FindMissingBlobs`](#findmissingblobs) calls stop re-querying the backend. It answers the existence question only; reads pass through. ## `fast_slow` store A two-tier store. Reads try the fast backend, fall back to the slow one, and copy what they find into the fast tier on the way back; writes mirror to both. Its one sharp edge: it never checks the slow tier for something the fast tier already has, so a blob that reached the fast tier without reaching the slow one stays invisible to everything reading the slow tier directly. ## FindMissingBlobs The CAS call that makes uploads incremental: the client offers a list of [digests](#digest), the server returns the ones it does not have, and the client uploads only those. On a warm cache this and the [AC](#action-cache-ac) lookup are most of what a client does. ## Hermetic build A build whose outputs depend only on its declared inputs. Same inputs → same outputs, on any machine, any time. What NativeLink does and does not guarantee here is in [correctness and hermeticity](/explanations/correctness-hermeticity). ## Input root The complete file tree an action is allowed to see, encoded as a Merkle tree of `Directory` messages that reference their files and each other by [digest](#digest). Its root digest is part of the [action](#action) hash, so changing one input file changes the cache key. ## Instance name A namespace inside a NativeLink cluster. Each service entry maps an instance name to a store or scheduler, so one cluster can serve multiple isolated environments, and the scheduler keys in-flight actions by instance name as well as digest. Two instance names that point at the same store share its contents. ## Isolation Keeping an action from seeing state outside its declared inputs. On Linux, workers can isolate actions with kernel namespaces via the `use_namespaces` and `use_mount_namespace` worker options, which also reap zombie processes and improve hermeticity. NativeLink doesn't currently integrate external sandboxing tools like `bwrap`, `landlock`, or `sandbox-exec`. ## JSON5 The dialect NativeLink's configuration file is written in: JSON plus comments, trailing commas, and unquoted keys. Every example in [the configuration reference](/reference/nativelink-config) is JSON5. ## LRE: Local Remote Execution Building your toolchain with Nix so that your local build and your remote workers invoke the *same* toolchain binaries, at the same `/nix/store` paths, which makes their action digests match and lets both share one cache. No worker runs on your laptop. See [LRE](/explanations/lre). ## Operation The handle a client holds on a running action, from `google.longrunning.Operation`. `Execute` returns one immediately and the client watches it for [stage](#execution-stage) changes; `WaitExecution` re-attaches to it after a disconnect, which is what stops a dropped connection from re-running the work. ## Origin events An experimental stream of the requests and responses a server handled, published to a store for an external consumer to read (). Off unless `experimental_origin_events` is configured. ## Platform properties Key/value tags attached to an action ("this action needs Linux, x86_64, a GPU") and to a worker ("this worker has Linux, x86_64, a GPU"). The scheduler matches them. ## Priority An integer on `ExecuteRequest`'s `execution_policy`; NativeLink dequeues the highest value first when workers are scarce (the RE-API leaves the direction to the server, and its suggested default is the reverse). It orders the queue and nothing more; an action already executing is not interrupted for a higher-priority one that arrives after it. ## Remote Execution API (RE-API) The standard gRPC protocol every supported build system speaks. [Spec](https://github.com/bazelbuild/remote-apis). ## Scheduler The dispatcher. Receives `Execute` calls, picks workers, tracks in-flight actions. ## Shard store A store that routes each key to one of several backends by digest hash, with a `weight` per backend. It spreads capacity; it is not replication, so losing one backend loses that backend's share of the data rather than none of it. ## Store NativeLink's unit of storage configuration: a named thing that gets and puts bytes under a key. Stores compose, because a store's backend is another store, so the CAS a server serves is usually a stack of wrappers over one real backend. See [the store model](/explanations/store-model). ## Toolchain The bundle of binaries an action needs to run: compiler, linker, standard library, etc. See [how toolchains are provided](/remote-execution/toolchains-and-hermeticity#faq). ## Verify store A wrapper that, depending on `verify_size` and `verify_hash`, checks a blob's size and re-hashes it on write, failing the upload when the bytes do not match the [digest](#digest) they were offered under. It turns silent corruption into a write error. Both checks belong on CAS stores and neither belongs on an [AC](#action-cache-ac) store. ## Worker The process that runs an action. Fetches inputs from CAS, runs the command, uploads outputs to CAS. ## Worker API NativeLink's own gRPC service, not part of the RE-API, over which a worker registers with a scheduler and receives work (). One bidirectional `ConnectWorker` stream carries the whole conversation, which is what keeps a worker talking to the same scheduler instance even behind a load balancer. # https://docs.nativelink.com/reference/changelog **Changelog**: Notable changes per release. Latest first. _Generated reference page, not inlined. Fetch it from https://docs.nativelink.com/reference/changelog._ # https://docs.nativelink.com/contribute/guidelines **Contribution guidelines**: What NativeLink accepts, the git setup it requires, and the commit and review conventions a PR is held to. **Who this is for:** anyone about to open their first pull request. **What you'll have at the end:** the git configuration the project requires, the commit conventions, and an accurate picture of how review works. **Time:** thirty minutes, most of it one-time setup. ## What gets accepted Three things reliably do not land: breaking changes to the Remote Execution API surface, which is an upstream spec the project tracks rather than owns; vendor-specific code paths only one deployment can use; and refactoring proposed on its own merits rather than as part of a change that needed it. ## Git setup, once
  • **Create distinct authentication and signing keys** in your [GitHub key settings](https://github.com/settings/keys). Two keys, not one reused.
  • **Fork the repository** and clone your fork, not upstream: ## The pull request loop
  • **Sync your fork**, then branch: ## Before you push `.rustfmt.toml` uses `group_imports` and `imports_granularity`, both nightly-only. Stable `cargo fmt` ignores them silently and produces output CI rejects. Use the Bazel target. ## What reviewers look for **One change per pull request.** Small ones land; large ones get split. ## Licensing Your contribution is licensed under the license that applies to the file or module you change. Most of the repository is `FSL-1.1-Apache-2.0`; a handful of files are Apache-2.0; and two areas, `nativelink-util/src/metrics.rs` and everything under `nativelink-worker/src/persistent_worker/`, are **Business Source License**. Copy the header from the file next to the one you are creating rather than from memory. ## Common questions [Cargo](/contribute/cargo) for small localized changes, the fastest way in. [Bazel](/contribute/bazel) when you want exactly what CI runs. [Nix](/contribute/nix) for the pinned shell, required for LRE work and for the generated `bazelrc` fragments. # https://docs.nativelink.com/contribute/bazel **Develop with Bazel**: Build, test, and run NativeLink with Bazel, the same dev loop CI uses. The Bazel-based dev loop is the one our CI runs. Use this if you want the most reproducible setup or if your editor integration prefers Bazel. ## Prerequisites - Bazelisk, which reads `.bazelversion` and fetches the right Bazel for you. CI runs 9.1.1 and also checks 8.7.0. - A C++ toolchain (clang or gcc) for native dependencies. - On macOS: XCode command-line tools. - On Linux, inside the Nix shell, the LRE toolchains link with `mold`; outside it Bazel uses whatever the host toolchain provides. ## First build The first `bazel test` will take 10-20 minutes as it builds the toolchain and dependencies. Subsequent builds finish in seconds. ## Common commands See [the testing guide](/contribute/testing-guide) for what `unit_test` and `integration` actually mean in this repository; they do not mean what the names suggest. ## Editor integration For `rust-analyzer`: ## Common pitfalls - **LRE flags are missing**: `.bazelrc` pulls in `lre.bazelrc`, `nativelink.bazelrc`, `nixos.bazelrc` and `darwin.bazelrc` with `try-import`, and the Nix dev shell is what generates them. Without `nix develop` they do not exist, and the `try-` means Bazel says nothing about it. See [Develop with Nix](/contribute/nix). - **Slow first build with no cache**: expected. ## FAQ It bootstraps the toolchain and builds every dependency from source. Subsequent builds are seconds, and pointing `--remote_cache` at a NativeLink cluster (above) makes even a fresh clone fast. # https://docs.nativelink.com/contribute/cargo **Developing with Cargo**: Pure Cargo workflow for NativeLink contributors who'd rather not run Bazel. Every NativeLink crate compiles cleanly with plain Cargo. Use this workflow if you want a smaller dev shell, faster incremental builds during iteration, or just don't have Bazel installed. ## Prerequisites - A recent stable Rust toolchain. No `rust-toolchain.toml` is checked in; the minimum is `rust-version` in the root `Cargo.toml`, currently 1.97.1, on edition 2024. - On Linux: `mold` is optional and has to be configured by hand; see [faster incremental builds](#faster-incremental-builds). ## First build First build is a few minutes (dependency compilation); subsequent incremental builds are typically a few seconds. ## Common commands `.rustfmt.toml` uses `group_imports` and `imports_granularity`, both nightly-only. Stable `cargo fmt` silently ignores them and produces output CI rejects. Run `bazel run --config=rustfmt @rules_rust//:rustfmt` instead. ## Faster incremental builds A few flags that help: ## Caveats - Three directories are excluded from the workspace and build on their own: `nativelink-config/generate-stores-config`, `nativelink-test/fuzz`, and `tools/generate-bazel-rc`. The LRE flake outputs are Nix, not Cargo. - Regenerating protos goes through Bazel (`bazel run nativelink-proto:update_protos`), not Cargo. - CI runs Bazel, and `bazel test` attaches rustfmt and clippy aspects that a Cargo run does not. ## FAQ Not to build or test; the whole workspace builds with plain Cargo. You need Bazel for a pre-submit `bazel test //...`, for regenerating protos, and for formatting, since that's what CI runs. # https://docs.nativelink.com/contribute/nix **Develop with Nix**: The dev shell that generates the bazelrc files the rest of the build assumes, and what is actually inside it. **Who this is for:** anyone who wants the toolchain CI uses, and anyone working on [LRE](/explanations/lre), which needs it. **What you'll have at the end:** the shell, the five generated `bazelrc` files it produces, and an accurate list of what it does and does not contain. **Time:** twenty minutes, mostly download. ## Install Nix The [next-gen Nix installer](https://github.com/NixOS/experimental-nix-installer) is the easiest path: ## Enter the dev shell First run downloads the toolchain, which takes a few minutes. After that, entering is close to instant; everything is content-addressed in `/nix/store`. ## What entering the shell generates This is the part that matters and the part that is invisible. The shell hook writes five things into your working tree: ## What is in the shell The package list is in . Broadly: ## Running things inside the shell Everything from the [Bazel](/contribute/bazel) and [Cargo](/contribute/cargo) workflows works unchanged. The shell guarantees the versions, and supplies the `bazelrc` fragments those workflows assume. ## LRE and worker images The flake builds the LRE worker images rather than exposing an LRE shell. The ones you are likely to want: ## When Nix isn't worth it For a typo fix, a doc tweak or a single unit test, the [Cargo workflow](/contribute/cargo) is faster to get into and nothing will go wrong. Reach for Nix when you are touching the build graph, working on LRE, chasing a difference between your machine and CI, or working on the docs, since `vale` and `bun` both come from here. ## Common questions A package manager that builds every package in isolation and addresses the result by a hash of all its inputs, which is why two machines running `nix develop` on this repo get bit-identical toolchains. # https://docs.nativelink.com/contribute/repo-crate-map **Repository and crate map**: What every crate and top-level directory in the NativeLink repository is for, and which one your change belongs in. **Who this is for:** anyone opening the NativeLink repository for the first time and trying to work out where their change goes. **What you'll have at the end:** the crate layering, the size of each piece, the directories that are not crates, and the half-dozen places where the first guess is wrong. **Time:** fifteen minutes. ## The workspace has no members list The root ## The crates Source sizes exclude each crate's `tests/` directory; the test column counts files directly under `tests/`. ## The layering Every internal dependency points down this stack. It has no cycles, and the shape is worth memorising because it tells you what a change can reach. ## Where the first guess is wrong Six of these, and each one has cost somebody an afternoon. ## Directories that are not crates They are not alternative spellings of each other. `deploy/` is the overlays, `deployment-examples/` is copyable manifests, and `kubernetes/` is the building blocks the overlays compose. ## Conventions you will notice immediately Every manifest except the nested macro-derive crate's starts with `#:schema tools/cargo-with-detailed-deps.json`, which is what gives you completion and validation in a TOML-aware editor. ## Common questions Two, plus a Bazel file: the spec goes in `nativelink-config`, the implementation and its factory arm go in `nativelink-store`, and the new file has to be added to `nativelink-store/BUILD.bazel`. The step-by-step is in [how to extend](/contribute/how-to-extend#adding-a-store). # https://docs.nativelink.com/contribute/codebase-internals **Codebase internals**: The traits, the error model, the async rules and the two metric systems: the conventions a reviewer will assume you already know. **Who this is for:** anyone about to write NativeLink code rather than read it. **What you'll have at the end:** the four things that are enforced rather than suggested: the store trait split, the error model, the spawning rules, and which metric system to use. **Time:** thirty minutes. ## The three store traits, and which one you implement Three of them exist, and they are not interchangeable. ## The scheduler traits are split by caller No `ActionScheduler` trait exists. The state layer is split into `ClientStateManager`, `WorkerStateManager` and `MatchingEngineStateManager` in , plus `WorkerScheduler`, `KnownPlatformPropertyProvider` and `AwaitedActionDb`. The split by *caller* is what allows the in-memory and Redis-backed implementations to be swapped wholesale. ## The error model `Code` **is** `tonic::Code`, re-exported from . No separate internal code enum and no mapping layer exist, which is why an error raised deep in a store arrives at a client with a sensible gRPC status without anyone writing a conversion. ## Async: `tokio::spawn` is banned The Tokio runtime is built by hand in `main()`, not with `#[tokio::main]`: the builder installs an `on_thread_start` hook (macOS QoS), and the process signal handlers and the shutdown broadcast are spawned onto it before `inner_main` runs. Each of those calls carries an `#[expect(clippy::disallowed_methods, reason = …)]`, because the same lint that bans `tokio::spawn` everywhere else bans the runtime builder too. ## Two metric systems They are not redundant, and picking the wrong one is a common review comment. ## Lints, formatting and the small stuff `clippy::all`, `clippy::nursery` and `clippy::pedantic` are all denied at the workspace level. `std_instead_of_core` is denied too, which is why you will see `use core::…` throughout for anything that does not need `std`. ## Common questions Because a bare spawn has no name, no tracing span, no OpenTelemetry context and no cancellation story. The three macros in `nativelink-util::task` supply all four, and the clippy config makes the alternative a build failure rather than a review comment. # https://docs.nativelink.com/contribute/how-to-extend **How to extend NativeLink**: Complete file-by-file recipes for adding a store, a config field, a metric, or a gRPC service. **Who this is for:** anyone adding a new component rather than changing an existing one. **What you'll have at the end:** four recipes, each listing every file a working change touches, including the ones that compile fine when you forget them and fail in CI. **Time:** use as a checklist. ## Adding a store Model: `nativelink-store/src/noop_store.rs`, the smallest complete `StoreDriver` in the tree. ## Adding a config field Config fields are documentation. The doc comment becomes the description in the generated reference, and the ```` ```json ```` fences in `stores.rs` are also harvested into the examples file, which is why they have to be valid JSON rather than illustrative fragments. ## Adding a metric Decide which system first; see [two metric systems](/contribute/codebase-internals#two-metric-systems). ## Adding a gRPC service Model: `nativelink-service/src/fetch_server.rs`: 165 lines, and it has every convention in it. ## Common questions Almost certainly the `srcs` list in `nativelink-store/BUILD.bazel`. Cargo globs; Bazel enumerates. Add the file, keep the list alphabetical, and run `bazel build //nativelink-store:nativelink-store` before pushing. # https://docs.nativelink.com/contribute/testing-guide **Testing guide**: How NativeLink's tests are organised, what `#[nativelink_test]` gives you, and which local environment problems cause which confusing failure. **Who this is for:** anyone writing a test, or trying to work out why a test that passes in CI fails on their laptop. **What you'll have at the end:** the commands, the conventions, the fixtures that already exist, and a diagnosis list for the local failures that are not your code's fault. **Time:** twenty minutes. ## The commands In this repository, `unit_test` means *tests written inside a source file* and `integration` means *tests in the crate's `tests/` directory*. Neither implies anything about scope or about talking to a network. The genuinely end-to-end suites are the shell scripts in `integration_tests/`, which are separate again. ## Where tests live The convention is a separate `tests/` directory per crate, not inline modules; only nine `mod tests` blocks exist in the whole tree. ## `#[nativelink_test]` Around 780 tests use it. It is defined in ## Fixtures that already exist Reach for these before writing your own. ## The end-to-end suites It refuses to run as root, requires Bazel, Docker and `sudo`, and waits for ports 50051, 50052 and 50071 before starting. It drives `simple_cache_test.sh`, `simple_remote_execution_test.sh`, `simple_tls_test.sh` and `chunking_cache_test.sh`. You can pass a pattern to run one of them. ## Coverage The `result` symlink contains an HTML report. **No coverage threshold gate**; no PR is blocked on a percentage. Coverage is a tool for finding untested paths, not a target to hit. ## When it fails locally but not in CI Most of these are environment, not code. ## Common questions Run the crate you touched with Cargo while iterating, then `bazel test //...` once before pushing. CI runs Bazel, and the aspects attached to `bazel test` catch formatting and clippy problems that a Cargo run does not. # https://docs.nativelink.com/contribute/release-versioning **Releases and versioning**: How a NativeLink version becomes a signed tag, what the tag triggers, and what compatibility the project does and does not promise. **Who this is for:** maintainers cutting a release, and anyone who needs to know what a version number means before pinning one. **What you'll have at the end:** the fourteen files a version lives in, the release sequence, what each workflow produces, and the compatibility policy, which is mostly the absence of one. **Time:** twenty minutes. ## The version lives in fourteen files No single source of truth for the version exists, because the root `Cargo.toml` declares a `[package]` version rather than a `[workspace.package]` version, so nothing inherits it. ## Cutting a release The authoritative sequence is the numbered list under "Creating releases" in . The shape of it: ## What the tag triggers `release.yaml` builds three targets (`x86_64-unknown-linux-musl`, `aarch64-unknown-linux-musl` and `aarch64-apple-darwin`), signs each with keyless Sigstore cosign, and attaches SLSA Build Level 3 provenance. ## What the version number promises Less than you would assume, and it is better to say so than to imply otherwise. ## Regenerating the configuration reference The reference pages under `reference/nativelink-config/` are generated from the Rust config crate and must never be hand-edited. The chain is: `cargo run --bin build-schema --features dev-schema` produces a JSON schema, `web/apps/docs/scripts/gen-config-reference.mjs` and `scripts/lib/schema-to-mdx.mjs` turn it into MDX, and the whole thing is invoked as: ## Common questions It is not guaranteed to. No SemVer policy is stated, and `compatibility_level = 0` in `MODULE.bazel` explicitly asserts none. Read the release notes, which do call out breaking changes and include migration instructions. # https://docs.nativelink.com/contribute/docs **Working on documentation**: How these docs are built: the four page archetypes, the components, the anchor and snippet lints, and what a docs PR is reviewed against. **Who this is for:** anyone writing or editing a page on this site. **What you'll have at the end:** the local dev loop, the archetype your page has to satisfy, the components you can use without importing them, and the lints that will fail your PR if you skip them. **Time:** thirty minutes, once. ## The local loop Edit any `.mdx` file under `apps/docs/content/docs/` and the dev server picks it up on save. To preview the marketing site and the docs together (the marketing app redirects `/docs` to the docs app in dev), run `bun dev` and use `http://localhost:3000/docs`. ## Pick the archetype first Every page is exactly one of four kinds, and the kind determines the shape. Mixing two is the most common structural review comment, because a tutorial that pauses to explain tradeoffs stops being a guaranteed happy path, and an explanation that turns into a procedure stops being readable out of order. ## The opening block every page has Directly under the archetype comment, before anything else: ## Frontmatter and navigation Two required fields, one optional: ## Components All of these are provided globally. **Never write an import in an `.mdx` file**; an import statement is the usual reason a page compiles locally and fails in the MDX check. # https://docs.nativelink.com/ **For agents**: How an AI agent should read these docs end to end, where every entry point is, how to cite and verify a claim, and what the docs guarantee to a machine reader. **Who this is for:** an AI agent (Claude Code, Cursor, Copilot, a homegrown tool) that has been pointed at NativeLink, and the person pointing it. **What you'll have at the end:** the entry points, a procedure for reading the whole corpus in order, the conventions that make pages citable and verifiable, and a task index into the rest of the docs. **Time:** five minutes to read, one fetch to load everything. ## Read this first if you are an agent 1. Fetch **[`/llms-full.txt`](/llms-full.txt)**. It is the entire documentation corpus, every page body verbatim, in reading order, in one plain-text file (under 1 MB). One fetch gives you everything a human would find by clicking through the sidebar. 2. ## The entry points **`AGENTS.md` at the repo root** is for an agent working *on the code*. It maps every `nativelink-*` crate to what it owns, says where config, store, scheduler and worker logic actually live, gives the build and test commands, and carries a table of "you changed X, so this doc has to follow" pairs that are often missed. ## Skills for coding agents The repository's `.claude/skills/` directory carries skills in the `SKILL.md` format that Claude Code, Cursor, Copilot, Codex and other agents read. Claude Code loads them automatically inside the repository; elsewhere, copy or symlink a skill into the agent's skills directory. ## Contributing as an agent Agents are welcome contributors when a person stands behind the change. The rules are in [`CONTRIBUTING.md`](https://github.com/TraceMachina/nativelink/blob/main/CONTRIBUTING.md#ai-assisted-contributions) and are short: the person submitting the change must understand it well enough to answer for any line; the pull request template's "AI assistance" section must say which tools helped and how much ("None" is a comple… ## How the corpus is ordered The sidebar, `llms.txt` and `llms-full.txt` all use the same order, defined once in the `meta.json` files under `content/docs/`: