Deploy with Docker Compose

Who this is for: you want the whole system (cache and remote execution) running on one machine, either to try it or to give a small team something shared. What you'll have at the end: three containers serving a real CAS, scheduler and worker, a build pointed at them, and a clear picture of what separates this from production. Time: 20 minutes, most of it spent compiling.

Of the deployment shapes NativeLink ships, Compose is the one that runs end-to-end out of the box. The Kubernetes building blocks need image tags and a Gateway you supply; the Compose files need nothing but Docker.

The configuration

git clone https://github.com/TraceMachina/nativelink
cd nativelink/deployment-examples/docker-compose
docker compose up -d

The first run builds the image from source with Bazel, which takes a while; see the build is not fast below.

Three services come up:

They talk to each other by service name. The worker config reaches the scheduler at grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061 and the CAS at grpc://${CAS_ENDPOINT:-127.0.0.1}:50051; Compose sets both variables, and NativeLink expands them at config-parse time; see CLI and environment.

Point a build at it

bazel build //... \
  --remote_cache=grpc://127.0.0.1:50051 \
  --remote_executor=grpc://127.0.0.1:50052 \
  --remote_default_exec_properties=cpu_count=2

Cache and executor are different endpoints. That is not an accident of the example: the CAS process and the scheduler process are genuinely separate here, exactly as they are in the production shape.

cpu_count is what the shipped README passes. The scheduler declares it as a minimum platform property and the worker advertises it from nproc, so an action is only matched to a worker with at least that many cores; ask for more than the worker has and the action queues silently forever. Omitting the property is fine: an action with no platform properties matches any worker. See platform properties.

What to change

Only the fields carrying a decision. Everything else in the shipped files is already correct for a single host.

The multi-worker variant

docker-compose-multi-worker.yml runs one CAS, one scheduler and three workers sharing a cas-data volume:

docker compose -f docker-compose-multi-worker.yml build
docker compose -f docker-compose-multi-worker.yml up -d

The shared volume is the entire point, and the README says so more loudly than anything else in it: all workers must share the same CAS storage path. When they do not, actions fail with

Object <digest> not found in either fast or slow store. If using multiple
workers, ensure all workers share the same CAS storage path.

which the worker reports to the scheduler as FAILED_PRECONDITION with that message intact, so that is the text your developers will see. Troubleshooting has the full path.

What makes this a development setup

Four things, all of them deliberate in the example and all of them disqualifying for production.

The executor runs privileged

nativelink_executor:
  privileged: true # So we can do use_namespaces

worker.json5 sets use_namespaces: true and use_mount_namespace: true, which need CLONE_NEWUSER/CLONE_NEWNS, hence the flag. Note what this buys and what it does not: namespaces here are a hermeticity and process-reaping mechanism, not a security boundary. Actions still run as the worker's own uid through an identity uid map, still see the host filesystem, and still have network access to every NativeLink port. A privileged container running arbitrary build actions is a lateral-movement primitive. Security hardening covers what the sandbox actually does.

The TLS certificates are expired examples

The 50071 listener loads example-do-not-use-in-prod-rootca.crt and example-do-not-use-in-prod-key.pem from the same directory. They are committed to the repository, self-signed, CN=localhost, and expired in October 2024. Every client must pass --insecure or equivalent, which is what the repository's own TLS integration test does. No certificate-generation script exists anywhere in the tree; you supply real material yourself.

Nothing is authenticated

Not a Compose limitation: open-source NativeLink has no inbound authentication on any service, in any deployment. docker-compose.yml does not publish the scheduler's port 50061 at all; the worker reaches it over the Compose network by service name. That is the only thing standing between an attacker on your network and the ability to register as a worker. Keep it that way.

The build is not fast

The Dockerfile compiles NativeLink from the repository root with Bazel, in a three-stage build, and the final image is a fresh Ubuntu with the binary copied in and curl installed. The Ubuntu base is digest-pinned and the bazelisk download is checksum-verified, so the build is reproducible, but it is a full from-source build every time the layer cache misses. For repeated use, build once and push the image to a registry your hosts can pull from.

Troubleshooting

What's next