Containers and images

Who this is for: anyone whose actions need a specific environment, and anyone running more than one kind of worker image. What you'll have at the end: worker images that carry your toolchain, and a routing scheme that sends each action to the right one. Time: twenty minutes.

What container-image is not

Start here, because the name is misleading and the misunderstanding is expensive.

NativeLink does not pull an image and launch a container for each action. No code sits behind the container-image key. Search the source for it and you will find it in example configs, in the LRE platform definitions under local-remote-execution/, and in CI wiring; nothing in the scheduler or worker source reads it. It is an ordinary string platform property with an evocative name.

What it actually expresses is: this worker process is running inside this image. The worker asserts a value, the build requests a value, and the scheduler matches them like any other exact or priority property. The environment is real; it's just established by how you started the worker, not by anything the scheduler does per action.

That failure is the silent-queue failure from Platform properties, and it is by far the most common way a container-image scheme goes wrong.

The actual model

One image per worker pool. Every action a given worker runs shares that worker's environment. Heterogeneity comes from running several pools and routing between them, not from varying the image per action.

This has a consequence worth planning around: image churn is a fleet operation. Changing a toolchain means building a new image, rolling a pool onto it, and changing what builds request, in an order that doesn't strand actions in the queue. There's a safe sequence below.

Build a worker image

A NativeLink worker image is any image that contains your toolchain and can run the nativelink binary with a worker config. The repo's Nix helper, createWorker, turns a toolchain image into a nativelink-worker-* image:

nativelink-worker-lre-cc = createWorker pkgs.lre.lre-cc.image;
nativelink-worker-lre-java = createWorker lre-java;
nativelink-worker-lre-rs = createWorker pkgs.lre.lre-rs.image;
nativelink-worker-siso-chromium = createWorker siso-chromium;
nativelink-worker-toolchain-drake = createWorker toolchain-drake;
nativelink-worker-toolchain-buck2 = createWorker toolchain-buck2;

What it adds on top of the base image is small: a nativelink user (uid 1000) and home directory, a world-writable /tmp, /usr/bin/env, and coreutils, bash and gnused on /bin. It keeps the base image's tag so the worker image and the toolchain it wraps stay visibly paired. It does not bake the nativelink binary in, whatever its header comment says: the Kubernetes examples inject the binary at pod start with the nativelink-worker-init init container, which copies it into a shared volume, and the worker container then runs /shared/nativelink /worker.json5. That keeps the toolchain image and the NativeLink version independently upgradable.

Push one to a registry the way the repo's own tooling does:

nix run .#nativelink-worker-lre-cc.copyTo docker://localhost:5001/nativelink-worker-lre-cc:latest

You do not have to use Nix for this. Any image that contains your toolchain and runs nativelink with a config is a valid worker image: a Dockerfile that starts FROM your-toolchain-base, copies in the binary, and sets the entrypoint works identically. Nix is how this repo gets reproducible images; it isn't a requirement of the worker protocol.

Wire up the routing

1. Declare the key on the scheduler

   schedulers: [{
     name: "MAIN_SCHEDULER",
     simple: {
       supported_platform_properties: {
         cpu_count: "minimum",
         OSFamily: "priority",
         "container-image": "exact",
       },
     },
   }]

   Use exact once you have more than one image. The examples ship priority, which matches any value: fine for a single-pool tutorial, useless as a routing mechanism.

2. Advertise it on each worker

   workers: [{
     local: {
       worker_api_endpoint: { uri: "grpc://scheduler.internal:50061" },
       cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
       upload_action_result: { ac_store: "AC_MAIN_STORE" },
       work_directory: "/tmp/nativelink/work",
       platform_properties: {
         cpu_count: { query_cmd: "nproc" },
         OSFamily: { values: [""] },
         "container-image": { values: ["ghcr.io/example/toolchain:2026-07-01"] },
       },
     },
   }]

   Pin a tag you can change deliberately. :latest makes the advertised string stable while the actual contents drift, which defeats the entire point: the scheduler will keep matching, and your builds will silently change behaviour.

   Better still, template it from the deployment so the value can't disagree with the image that's actually running:

   "container-image": { values: ["${NL_WORKER_IMAGE}"] },

3. Request it from the build

   Globally, if your whole build wants one environment:

   bazel build \
     --remote_executor=grpc://scheduler.internal:50052 \
     --remote_default_exec_properties=container-image=ghcr.io/example/toolchain:2026-07-01 \
     //...

   Or per target, when only some actions need it:

   cc_test(
       name = "integration_test",
       exec_properties = {
           "container-image": "ghcr.io/example/toolchain:2026-07-01",
       },
       ...
   )

Rolling an image without stranding actions

Order matters, because the two sides are matched on a literal string.

1. Bring up the new pool first, advertising the new image string, while the old pool keeps running and advertising the old one. Both are now satisfiable.
2. Switch what builds request. New builds land on the new pool; in-flight ones finish on the old.
3. Drain and remove the old pool once nothing requests it.

Doing this in the other order (changing the request before the pool exists, or removing the pool before builds stop asking for it) produces a queue that never clears and no error anywhere.

Troubleshooting

FAQ