Runbooks

Who this is for: you are on call and something is broken right now. What you'll have at the end: for each of the four incidents NativeLink actually produces, a confirming signal, a stop-the-bleeding action, and the configuration change that prevents a repeat. Time: read once now, so you do not read it for the first time at 03:00.

These are incident procedures. They deliberately do not explain how NativeLink works (Architecture does that) and they do not tune anything. When you have stopped the bleeding and want to make the system faster rather than merely alive, go to Tuning.

If you are not yet sure which incident you are having, start at Troubleshooting, which indexes symptoms to causes. This page assumes you already know.

A worker is OOM-killed

The most common production incident, and the one with the most misleading first symptom: your developers report a build failure, not an infrastructure failure.

Confirm it

Two distinct failures land here and they need different responses. Work out which one you have before doing anything.

The action's child process was killed. The worker survived. The action reports exit code 9, and the worker logs:

Runner subprocess terminated by signal (no exit code); likely OOMKilled or
externally killed. If this repeats for the same action, raise
`workers.specs[*].resources.limits.memory` or shrink the action's concurrency.

That exit code is a fixed sentinel: every signal collapses to 9, so you cannot tell SIGKILL from SIGTERM from the code alone. The warn! line above is the only thing that tells you it was a signal at all. From Bazel's side this looks like an ordinary action failure with a non-zero exit status, so Bazel will not treat it as an infrastructure error and will not retry it. On the scheduler it lands in nativelink_execution_completed_count_total{execution_result="failure"} like any other non-zero exit; there is no exported metric that singles out signal deaths, so the log line is what you alert on.

The worker process itself was killed. The scheduler notices after worker_timeout_s (default 5 seconds, checked once a second), evicts it, and re-queues its in-flight actions. In this tree a disconnect counts as an attempt, so after max_job_retries the client finally sees:

Worker disconnected repeatedly while executing this action (4 > 3 attempts);
the runner likely OOMKilled or the pod was evicted. for operation_id: ...,
maybe_worker_id: ...

That string is the single best diagnostic for a memory-pressured worker fleet. If you see it, you are in this case, not the first one.

Stop the bleeding

Drain the affected worker rather than killing it, so its in-flight actions finish instead of being destroyed:

curl -X POST \
  "http://127.0.0.1:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/${WORKER_ID}/1"

The path segment after scheduler/ is the scheduler's name from the schedulers list, not a REAPI instance name. The endpoint only exists on a listener with admin: {} in its services (the shipped Kubernetes config has none); put it on the worker-API listener. It has no authentication; reach it over a private path only. See Security hardening.

Undrain with a trailing 0 when the host is healthy again. A drained worker appears in the scheduler's matching diagnostics as:

Worker <id> cannot accept work: is_paused=false, is_draining=true, inflight=0/0

Make it stop happening

NativeLink itself has no memory configuration. No max_memory, no per-action memory cap, nothing. Memory is bounded only indirectly, by store eviction policies and by whatever the container runtime enforces. Do not go looking for a config field; it does not exist.

The disk fills up

Confirm it

Writes start failing with an OS-level message wrapped in a NativeLink err_tip: Failed to write data into filesystem store, Failed to flush in filesystem store, Failed to sync in filesystem store.

The gRPC code will be Unknown, not anything storage-shaped. ENOSPC arrives as ErrorKind::StorageFull, which NativeLink's io::ErrorKind to gRPC-code map does not name, so it falls to the catch-all arm. No ENOSPC-specific handling exists anywhere in the tree.

That has an unpleasant consequence: Unknown is retryable. A full disk gets retried through the entire retry budget before it fails, so the first thing you notice is usually latency, not errors.

The signal to look for first

Bytes: 41231872418 of unlimited;

If your eviction snapshot says "of unlimited", no size-based eviction is configured, and this incident was inevitable. eviction_policy is an Option on the filesystem store, and every field in it defaults to 0, which means never evict. The config's own doc comment is blunt about it: failure to set this value causes items to never be removed.

Stop the bleeding

Set max_bytes and restart the process. Eviction is only ever invoked on map mutation (on insert and remove), so it runs as soon as traffic resumes.

{/* lint-snippets: ignore */}
{
  "filesystem": {
    "content_path": "/var/lib/nativelink/content",
    "temp_path": "/var/lib/nativelink/temp",
    "eviction_policy": {
      "max_bytes": 50000000000,
      "evict_bytes": 5000000000
    }
  }
}

evict_bytes creates a low watermark: once max_bytes is reached, eviction continues down to max_bytes - evict_bytes rather than evicting one entry per insert forever. Set it to roughly 10% of max_bytes.

Size the volume for twice max_bytes

The filesystem store keeps an executable-variant directory ({content_path}.exec) alongside the content path, and that directory is invisible to max_bytes. A variant is deleted when its primary entry is evicted, which bounds the total at roughly 2 * max_bytes in the worst case (every blob also held as an executable). The variants are only ever created on a worker, for executable inputs hardlinked out of its fast store, so this applies to worker fast tiers rather than to a CAS server's store.

Provision accordingly: if you want a 50 GB worker cache, give it a 100 GB volume.

Two layout footguns

temp_path must be on the same block device as content_path. If it is not, the atomic rename becomes a copy: slower, and it doubles peak space during every write.

Everything in temp_path is deleted at startup, and the content path is rescanned and re-accounted. A crash mid-write leaves a partial temp file that is reclaimed on the next boot, but only on the next boot. A process that crashes repeatedly without a clean restart accumulates.

Redis fails over

What actually happens

Both Sentinel and Cluster modes are supported; the default is standard. On a Sentinel failover the demoted master answers -READONLY, which NativeLink classifies as retryable alongside dropped connections, refusals and IO errors.

Each retryable failure reconnects and sleeps 100 ms, up to 5 attempts. Reconnects are serialised behind a dedicated mutex so a failover storm does not have every in-flight request stampeding the new master.

Net behaviour: about 500 ms of retrying, then the operation fails with the underlying Redis error. The process does not exit.

What breaks depends on your backend

Stop the bleeding

If you are on the memory backend, this is a cache-availability incident, not a scheduler incident. Builds degrade to local execution and cache misses. Fix Redis at your leisure.

If you are on the redis backend, builds hang. Failing over Redis is the fix; there is no NativeLink-side mitigation.

Configuration worth checking now

response_timeout_s and connection_timeout_s are deprecated; response_timeout_s is ignored entirely. Both log a warning at startup. Setting both connection_timeout_s and connection_timeout_ms is a startup error.

The queue backpressures and never recovers

The subtlest incident of the four, because nothing errors and nothing restarts. Actions re-queue forever.

The mechanism

ResourceExhausted is NativeLink's backpressure code. When a worker returns it for an action, two things happen: the action goes back to Queued and (this is the important part) its attempt counter is not incremented. If the worker still has other actions in flight it is also marked is_paused, and that flag clears as soon as any of those actions completes.

That is deliberate. Backpressure is not the action's fault, so it should not burn a retry. But it means a worker that returns ResourceExhausted permanently produces an invisible infinite loop: the scheduler hands it the action, it rejects it, the action requeues without counting, and the next matching pass hands it over again. max_job_retries does not protect you, because no attempt is ever counted, and an idle worker is never paused, so nothing takes it out of the candidate set.

The usual cause is experimental_precondition_script, the intended hook for "do not send me work, my disk is filling up". If the script's condition never clears, the loop never ends.

Confirm it

RUST_LOG=info

then look at the scheduler's periodic pass: the action keeps appearing in Oldest actions in state as Queued, its age grows, and it never reaches a retry limit. The worker itself is quiet; the script's exit status is logged only at TRACE (Preconditions script returned), so RUST_LOG=info on the worker shows nothing. If the worker still holds other work you will also see, in the scheduler's matching diagnostics:

Worker <id> cannot accept work: is_paused=true, is_draining=false, inflight=1/0

is_paused=true is only ever set while inflight is non-zero; a worker with nothing in flight that is still refusing work does not appear here at all, because it is, as far as the scheduler can tell, a perfectly available worker.

Stop the bleeding

Fix or remove the precondition script and restart the worker. Nothing can clear is_paused from the outside; it is set and cleared by the scheduler in response to what the worker returns.

If the precondition script is doing its job correctly, and the host really is out of disk, then this is the disk-full incident wearing a different mask. Go to The disk fills up.

Related limits that block rather than reject

Several NativeLink limits queue instead of erroring, which is why "hung" is a more common symptom than "failed":

The FD limit is the one that catches people. If the kernel's hard limit caps the raise below what you asked for, the semaphore silently shrinks to 80% of what it actually got, and the only evidence is a startup warning (if the raise call fails outright, the semaphore stays at 80% of 24576 regardless of what the kernel allows):

The new open file limit (1024) is below the recommended value of 24576.
Consider raising max_open_files.

Restarting the scheduler

Not an incident so much as a procedure that behaves worse than you expect.

Worker reconnect is a flat 0.5-second retry with no exponential backoff, no jitter, no cap and no configuration field. A scheduler outage therefore produces a 2 Hz reconnect storm from every worker, for the whole outage.

Worse, on reconnect each worker calls kill_all() and destroys every action it had in flight, to avoid resource-locking itself against the new work the scheduler will send. A scheduler restart is disruptive to in-flight builds by design.

Before restarting a scheduler with running work, drain each worker (see above) and wait for inflight=0.

Log lines to grep for during a restart:

Worker registered with scheduler
Worker disconnected from scheduler
Actions in transit did not reach zero before we disconnected from the scheduler

The third means the 10-second drain window expired with work still in transit.

What's next