Toolchains and hermeticity

Who this is for: anyone whose actions execute remotely but fail on the worker for reasons that never happen locally. What you'll have at the end: a Nix-pinned toolchain that produces the same action digests on your machine and on a worker. Time: thirty minutes for the walkthrough.

The problem

An action is a closed description of some work: input digests, a command line, an environment, and platform properties. If the compiler your command invokes isn't in that description, the action is a lie: it works on your machine because your machine happens to have the compiler, and it fails on a worker that doesn't.

This is not a NativeLink behaviour. It is what remote execution is: the protocol will faithfully reproduce whatever you described, and no more. Caching lets you get away with an under-described action because the only machine that ever ran it was yours. Execution doesn't.

Two ways out exist.

1. Make the environment part of the image. Bake the toolchain into the worker's container, and route actions to it with the container-image property. Minimal, and it's what most fleets do first. Containers and images covers this path.
2. Make the toolchain part of the action. Pin every tool at a content-addressed path so the toolchain itself is an input the CAS knows about. This is Local Remote Execution, and it's the rest of this page.

The second is strictly stronger: it gets you identical action digests locally and remotely, which means a local build and a remote build share cache entries instead of each maintaining their own. What LRE is and why is the conceptual write-up; this page is the mechanism.

What ships in @local-remote-execution

Three toolchain families, and they do not behave the same way once you leave x86_64-linux.

README.md flake-module.nix

The flake module always emits the Rust platform and toolchain flags, and adds the C++ ones only on Linux, where the generated config is x86_64 and carries a container-image property. On an x86_64-linux host that means C++ actions can run locally out of the same /nix/store paths the worker image has, or remotely on that image, with the same digest either way. On macOS there is no local C++ or Java toolchain at all, so those actions can only execute on a remote worker running the matching image, and even a Nix-pinned toolchain costs a network round-trip: LRE gives you hermeticity, not offline-ness.

Set it up

1. Install Nix with flakes enabled

   The next-gen installer is the shortest path. If you already have Nix, make sure experimental-features = nix-command flakes is in your nix.conf.

   The pinning is the guarantee here: every tool lives at a content-addressed /nix/store path, and that path is what makes action hashes match across machines. No non-Nix version of this mechanism exists.

2. Pull the template

   mkdir my-lre-test && cd my-lre-test
   nix flake init -t github:TraceMachina/nativelink#bazel

   This writes a flake.nix importing nativelink.flakeModules.lre and pinning lre = { inherit (pkgs.lre.lre-cc.meta) Env; }, plus a hello-world.cpp example, its BUILD.bazel, and platforms/BUILD.bazel.

   bazel is the only template the flake ships; Examples and templates has the full list and what to do if you want a different starting point.

3. Initialise git before you enter the shell

   git init && git add -A

   This step is not optional and it fails quietly.

   Skip this and lre.bazelrc is silently never written

   The flake module's install script checks for a .git directory before generating anything. In a plain non-git folder, nix develop prints this and carries on without writing lre.bazelrc (no error, no toolchain):

   WARNING: lre: .git not found; skipping installation.

4. Enter the dev shell

   nix develop

   This fetches the Nix-pinned toolchain and writes lre.bazelrc (a symlink into the Nix store). On aarch64-darwin the generated file looks like this, abbreviated; your store paths will differ:

   # These flags are dynamically generated by the lre flake module.
   #
   # PATH=/nix/store/...-binutils-wrapper-.../bin:/nix/store/...-customClang/bin:...
   # CC=/nix/store/...-customClang/bin/customClang
   
   # Bazel-side configuration for LRE.
   build --define=EXECUTOR=remote
   build --extra_execution_platforms=@local-remote-execution//rust/platforms:aarch64-apple-darwin,@local-remote-execution//rust/platforms:aarch64-apple-darwin
   build --extra_toolchains=@local-remote-execution//rust:rust-aarch64-darwin
   build --extra_toolchains=@local-remote-execution//rust:rustfmt-aarch64-darwin
   build --platforms=@local-remote-execution//rust/platforms:aarch64-apple-darwin

   Note that it registered only the Rust platform and toolchains even though the flake asked for the lre-cc Env: the module emits the Rust lines on every system and adds the generated-cc platform and cc-toolchain lines only on Linux. Env only decides which store paths are listed (and therefore fetched) in the commented header. The template's own .bazelrc separately adds --extra_execution_platforms=@//platforms:lre-cc for the C++ example, which is the container-based platform from the table above.

5. Point user.bazelrc at your cluster

   nix flake init writes a user.bazelrc with the literal string TODO as every value. That is what lands on disk, not a placeholder left in this page:

   build --remote_cache=grpcs://TODO
   build --bes_backend=grpcs://TODO
   build --remote_timeout=600
   build --remote_executor=grpcs://TODO

   Filled in against a plain local cluster: no TLS, so grpc://, and no BES endpoint to point at:

   build --remote_cache=grpc://127.0.0.1:50051
   build --remote_timeout=600
   build --remote_executor=grpc://127.0.0.1:50051

   A hosted or self-managed cluster is the same shape with grpcs:// and real hostnames, plus a bes_backend line if you want build results streamed.

   If you build before editing this file, Bazel fails on the literal hostname TODO:

   [8 / 10] Compiling src/hello-world.cpp; 0s remote, remote-cache
   ERROR: BUILD.bazel:3:10: Compiling src/hello-world.cpp failed:
     Failed to query remote execution capabilities: UNAVAILABLE: Unable to resolve host TODO

   Everything before that DNS failure worked: the Nix toolchain, the platform wiring, and the lre-cc execution platform selection. 0s remote, remote-cache means Bazel had already committed to executing remotely.

6. Build

   bazel build hello-world

   With a real endpoint and a worker running the nativelink-worker-lre-cc image, this compiles src/hello-world.cpp on that worker and returns the binary through the same CAS your local Bazel reads from. Run it again and it's a cache hit.

Adding LRE to a project you already have

The template bootstraps a new workspace. For an existing one you need the flake-parts module wiring, the bazel_dep override, and a try-import of the generated lre.bazelrc.

local-remote-execution/README.md

has that path, and also covers verifying the setup against a Kubernetes cluster.

Troubleshooting

FAQ

NextContainers and images

The other path to a reproducible worker environment, and what the container-image property actually does.

SidewaysWhat LRE is and why

The conceptual write-up behind the mechanism on this page.