Deploy on bare metal

Run NativeLink as systemd services on hosts you own: writing the units, sizing the machines, placing the disks, and rolling an upgrade without dropping in-flight actions.

Who this is for: you have VMs or physical machines and no interest in running a cluster orchestrator to get a build cache. What you'll have at the end: three systemd units, a disk layout that will not surprise you, host sizing you can defend, and a restart procedure. Time: an afternoon.

Before you start

A config you wrote (see Your first full config), and hosts you can install a systemd unit on.

No systemd unit exists in the NativeLink repository. That is the honest starting point for this page: unlike Compose and Kubernetes, where you can point at shipped files, here you write everything. The good news is that there is very little to write: the binary takes one positional argument and reads nothing else from its command line.

If what you actually want is a cache for a handful of people rather than a production deployment, Shared cache is the shorter road and ends somewhere useful.

The configuration

Three units, one per process. They are near-identical; the differences are the config file and, for the worker, the security posture.

The service account and directories

sudo useradd --system --home-dir /var/lib/nativelink --shell /usr/sbin/nologin nativelink
sudo install -d -o nativelink -g nativelink /var/lib/nativelink
sudo install -d -o nativelink -g nativelink /etc/nativelink
sudo install -m 0755 nativelink /usr/local/bin/nativelink

Put the configuration files in /etc/nativelink/{cas,scheduler,worker}.json5.

The CAS unit

# /etc/systemd/system/nativelink-cas.service
[Unit]
Description=NativeLink CAS and Action Cache
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/cas.json5
Restart=always
RestartSec=2s

# The binary raises its own nofile limit at startup, but only up to the hard
# limit systemd gives it. See "File descriptors" below.
LimitNOFILE=65536

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

# The CAS touches only its own data directory.
StateDirectory=nativelink
ReadWritePaths=/var/lib/nativelink
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The scheduler unit

Identical apart from the config file and the fact that it needs no writable state at all when the scheduler backend is the default in-memory one:

# /etc/systemd/system/nativelink-scheduler.service
[Unit]
Description=NativeLink scheduler
After=network-online.target nativelink-cas.service
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/scheduler.json5
Restart=always
RestartSec=2s
LimitNOFILE=65536

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
NoNewPrivileges=true

[Install]
WantedBy=multi-user.target

The worker unit

This one is different, and the differences matter.

# /etc/systemd/system/nativelink-worker.service
[Unit]
Description=NativeLink worker
After=network-online.target
Wants=network-online.target

[Service]
Type=exec
User=nativelink
Group=nativelink
ExecStart=/usr/local/bin/nativelink /etc/nativelink/worker.json5
Restart=always
RestartSec=2s

# Workers open far more files than the other two processes: every input in
# every action tree, plus the CAS entries backing them.
LimitNOFILE=131072

Environment=RUST_LOG=info
Environment=NL_LOG=json
Environment=OTEL_EXPORTER_OTLP_ENDPOINT=http://127.0.0.1:4317

# Actions run as this user, in this unit's cgroup. Everything the unit can
# reach, an action can reach.
ReadWritePaths=/var/lib/nativelink
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true

# Bound the blast radius of an action that allocates without limit.
MemoryMax=48G
MemoryAccounting=true

# On SIGTERM the worker stops taking new actions and waits for in-flight
# ones to finish. Set this longer than your longest action (the worker's
# max_action_timeout_s defaults to 20 minutes), or systemd SIGKILLs it
# mid-action.
KillMode=mixed
TimeoutStopSec=25min

[Install]
WantedBy=multi-user.target

NoNewPrivileges is absent from the worker unit on purpose

If your worker configuration sets use_namespaces: true, the process calls unshare(CLONE_NEWUSER | CLONE_NEWPID | ...), which NoNewPrivileges=true can block depending on kernel and distribution policy. If you are not using namespaces, add NoNewPrivileges=true back; it costs nothing.

Either way, understand what you are relying on: NativeLink's namespace support is a hermeticity and process-reaping mechanism, not a sandbox. Actions run as the worker's own uid, see the whole filesystem the unit can see, and can reach every NativeLink port on the network. Security hardening is the full picture, and it is the reason MemoryMax and ProtectSystem are in this unit rather than left to chance.

Then:

sudo systemctl daemon-reload
sudo systemctl enable --now nativelink-cas nativelink-scheduler nativelink-worker

What to change

Setting	Where	Why you'd change it
`LimitNOFILE`	unit	NativeLink raises its own soft limit at startup toward `max_open_files` (default 24,576) but cannot exceed the hard limit systemd grants. Set the unit's limit above the config value, or the config value is capped (the startup log line `set_open_file_limit() assigns new open file limit N` shows what it got).
`max_open_files`	`global` in the config	The binary reserves 80% of the achieved limit as a concurrency semaphore. Exhausting it does not error; it blocks.
`MemoryMax`	worker unit	NativeLink's configuration has no memory limit anywhere. The cgroup is the only bound that exists.
`RestartSec`	worker unit	The worker's own scheduler-reconnect retry is a hard-coded 0.5 s with no backoff. A restart loop plus that retry is a lot of connection attempts; keep `RestartSec` at seconds, not milliseconds.
`TimeoutStopSec`	worker unit	On `SIGTERM` the worker rejects new actions, waits for in-flight ones to finish, then tells the scheduler it is going away and exits. If this timer fires first, systemd `SIGKILL`s it and every in-flight action fails and re-queues.
`NL_LOG`	environment	`json` if anything is ingesting these logs. Any unrecognised value silently falls back to `pretty`; see CLI and environment.

Disk layout

The CAS host is the one where placement matters.

Path	Requirement
`content_path`	The bulk of the data. On the CAS host, provision `max_bytes` plus headroom. On a worker host, provision twice the fast tier's `max_bytes`: the worker creates an executable-variant directory next to `content_path` that is invisible to the eviction accounting, so that store can reach roughly `2 × max_bytes` in the worst case.
`temp_path`	Must be on the same block device as `content_path`. Otherwise the atomic rename that completes every write degrades to a copy: slower, and it doubles peak space during writes. Everything here is deleted on every startup.
Worker `work_directory`	Scratch. Purged at worker startup, which means two workers must never share one. A second worker starting will delete the first worker's in-flight action trees.

Eviction never looks at free space

NativeLink evicts on the size it has accounted for, not on the size the filesystem reports, and it evicts only when the store is mutated, never in response to a failed write. An eviction_policy is optional and its max_bytes defaults to zero, which means never evict. Leaving it unset on a filesystem store is the single most common way to fill a disk. Runbooks has the recovery.

Host sizing

No sizing numbers are published for NativeLink; measure your own workload before buying anything. What the code does fix:

Resource	Starting point	Notes
CAS storage	The `max_bytes` you set, plus headroom	It is a cache of the working set, not an archive; the eviction policy is the only bound.
Worker vCPU	1 per concurrent action	Set `max_inflight_tasks` to match; it defaults to 0, meaning unlimited.
Worker memory	The sum of your largest concurrent actions	The configuration has no per-action memory limit; `MemoryMax` on the unit is the only bound.
Scheduler	Small	It holds queue state in memory and owns no storage.

CAS storage and I/O dominate the bill; the scheduler is the cheapest process in the system. If you are choosing where to spend, spend on CAS I/O.

Restarts and upgrades

The three processes have genuinely different restart characteristics, and treating them the same is how a routine upgrade turns into a stack of failed builds.

The scheduler is the disruptive one. When it goes away, every worker notices within seconds and reconnects at a flat 0.5-second interval with no backoff and no jitter; a fleet of 200 workers produces up to 400 connection attempts per second at the socket for the whole outage. Worse, each worker's reconnect path calls kill_all() first, destroying every action it was running. Restart the scheduler when the queue is quiet, and restart it once.

Workers drain themselves on SIGTERM: the process rejects new actions, waits for the ones it is running to finish, sends the scheduler a GoingAway, and exits. systemctl stop is therefore safe as long as TimeoutStopSec outlasts your longest action. To take a worker out of rotation without stopping it, and if your scheduler configuration enables the admin service on the worker API listener:

curl -X POST \
  "http://127.0.0.1:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/${WORKER_ID}/1"

The path segment after scheduler/ is the scheduler's name from the config, not a REAPI instance name. The worker finishes what it has and takes nothing new. What loses work is a SIGKILL (the stop timer firing, or the OOM killer): every in-flight action fails and re-queues, and in this tree a disconnect consumes a retry attempt; enough of them in a row and the action fails outright with a message naming OOM-kill or eviction as the likely cause. Note that the admin endpoint is unauthenticated; Security hardening explains why it must not be routable.

The CAS is the safe one. It holds no scheduling state; a restart is a window of UNAVAILABLE responses and nothing more. Its temp_path is pruned on the way back up, so a crash mid-write reclaims itself.

CONNECTION_RETRY_DELAY_S GoingAwayRequest UpdateWithDisconnect set_open_file_limit set_drain_worker

You did it right if

systemctl is-active nativelink-cas nativelink-scheduler nativelink-worker prints active three times.

journalctl -u nativelink-worker | grep 'Worker registered with scheduler' finds a line. If instead you find Could not connect to endpoint repeating about twice a second, the worker cannot reach the scheduler's worker API port.

journalctl -u nativelink-cas | grep 'open file limit' shows the limit the process actually achieved, not the one you asked for. A warn about the limit being below the recommended value means LimitNOFILE is too low.

A build run twice against the cache reports hits on the second run.

Troubleshooting

Symptom	Cause	Fix
Unit restarts in a loop, journal says `unknown field`	Config is parsed with `deny_unknown_fields`; one unrecognised key is fatal	Check the key in the configuration reference
Process exits during startup, no error about ports	`NL_OTEL_ENDPOINT` without a scheme or explicit port panics before binding	Use `http://host:4317`; see CLI and environment
Everything slows down under load, nothing errors	File-descriptor semaphore exhaustion blocks rather than failing	Raise `LimitNOFILE` and `max_open_files` together
Two workers on one host behave erratically	They share a `work_directory`, which is purged at startup	Give each worker its own
Disk full, then every request fails and keeps retrying	ENOSPC surfaces as a generic error code that clients and upstream stores retry, and eviction is not triggered by write failures	Runbooks
Actions fail with exit code 9 and no error message	The kernel OOM-killer took the action's child process	Runbooks
A rolling restart produces a burst of failed builds	Workers were `SIGKILL`ed before their in-flight actions finished (`TimeoutStopSec` too short)	Raise `TimeoutStopSec`, or drain via the admin endpoint before stopping

What's next

NextRunbooks

The four incidents that actually page you, each with the log line that identifies it and the sequence that resolves it.

SidewaysHarden it

Port exposure, TLS, and why the worker API must never be routable.

SidewaysAdd more workers

Sizing one worker, then deciding how many.

On this page