Troubleshooting

Who this is for: something is wrong and you have a symptom but not a cause. What you'll have at the end: the cause, and either a fix or a pointer to the procedure that fixes it. Time: find your row, follow it.

Search this page for the error string you have. If your symptom is a hang rather than an error, start at Actions queue forever and workers sit idle; NativeLink's most common failure mode produces no error string at all.

If you already know which incident you are having and need the procedure rather than the diagnosis, go to Runbooks.

The index

Actions queue forever and workers sit idle

The single most important asymmetry in NativeLink:

Unknown platform property 'os_famly'

That second case is not a bug so much as a consequence of how matching works ("no worker matches yet" is indistinguishable from "no worker matches ever"), but its operational effect is a build that hangs with no diagnostic.

max_action_executing_timeout_s does not help, because it applies only to the Executing state and the action never reaches it. The only thing that ever ends the wait is client_action_timeout_s (default 60 s) noticing the client has stopped listening, which produces a message about clients listening and says nothing about properties.

Diagnose it

Raise the log level. Every useful diagnostic here is at INFO, and the shipped examples run at warn.

RUST_LOG=info

Then match what you see:

All of those lines, and the property mismatch lines below, are only written during the periodic full-logging pass, which is controlled by worker_match_logging_interval_s, default 10 seconds. -1 or 0 disables it; anything invalid logs an error and disables it. The same pass emits Oldest actions in state, capped at five per stage, the intended "why is my queue stuck" tool.

Property types that constrain nothing

A scheduler key configured as priority when it should be exact or minimum is the classic version of this bug, and it fails in the opposite direction: actions get scheduled onto workers that cannot run them. See Scaling workers for the fleet-shaped version of this problem and Platform properties for the full matching semantics.

The converse hazard is documented in the config itself: if a worker fails to advertise a property the scheduler requires (cpu_arch, say), the scheduler will never send it any work at all.

Actions fail with a CAS miss

The string you will actually see

Bazel reports FAILED_PRECONDITION. Not NOT_FOUND. That translation happens on the worker, which checks for a NotFound whose message contains not found in either fast or slow store and converts it before returning:

Missing CAS inputs during prepare_action, returning FAILED_PRECONDITION

The error your developers report will not contain the word "missing" and will not look like a cache problem. The underlying message is:

Object <digest> not found in either fast or slow store. If using multiple
workers, ensure all workers share the same CAS storage path.

The same warning is pre-announced at worker startup, which is worth grepping for when you first bring a fleet up:

Starting worker '<id>'. IMPORTANT: If running multiple workers, all workers
must share the same CAS storage path to avoid 'Object not found' errors.

The two causes

Workers do not share CAS storage. The fix is exactly what the message says: every worker's fast_slow slow tier must resolve to the same CAS the clients upload to. In Compose, docker-compose-multi-worker.yml does that by pointing each worker's slow store at the cas-server over gRPC (the cas-data volume it also mounts into the workers is not what shares the data); in Kubernetes, a shared backing store rather than per-pod local storage. See Deploy with Docker Compose.

Cache pressure evicted the blob. A different string, and a much more helpful one:

<digest> not found in filesystem store. This may indicate the file was evicted
due to cache pressure. Consider increasing 'max_bytes' in your filesystem
store's eviction_policy configuration.

Take that advice literally, and read the disk runbook before you pick a number; the store can reach roughly twice max_bytes on disk.

A near-miss worth knowing

<digest> is a zero-digest; FilesystemStore does not persist zero-byte files.

Zero-byte outputs are not stored. Callers materialise them directly. If you see this in a stack trace it is a caller bug, not a cache problem.

The timeout message that tells you nothing

Operation timed out having no more clients listening

This means the client stopped waiting, and client_action_timeout_s (default 60 s) cleaned the operation up. It is a statement about the client, not about why the action never ran.

It is the message operators find when chasing a hung build, and it is consistently misleading: the action was almost certainly stuck in the silent queue, and this string is just the tombstone.

Timeouts

The timeout defaults that matter: max_action_timeout_s 20 minutes, max_upload_timeout_s 10 minutes, max_cleanup_wait_s 30 seconds, worker_timeout_s 5 seconds, client_action_timeout_s 60 seconds, retain_completed_for_s 60 seconds.

retain_completed_for_s is not an execution timeout; it is how long a completed action stays findable. Set it too low and a client reconnecting after a network blip cannot find its own finished action.

Zero means default

Setting a field to 0 to disable it usually does the opposite.

No single rule applies here, which is exactly why the table exists. When in doubt, check the config reference for the field rather than assuming.

Worker connection churn

A worker that logs at a steady ~2 Hz is reconnecting:

Worker disconnected from scheduler

Reconnect is a flat 0.5-second retry: no backoff, no jitter, no cap, and no configuration field. That cadence is the signature, and a scheduler outage produces it from every worker simultaneously for the duration.

Every reconnect calls kill_all(), destroying every in-flight action on that worker. If your builds are failing intermittently and your workers are churning, those are the same incident.

The worker sends a keepalive every half of its worker_api_endpoint.timeout (default 5 s, so every 2.5 s), and the scheduler evicts a worker it has not heard from for worker_timeout_s (default 5 s), checking once a second. The scheduler counts what it receives in nativelink_worker_keepalives_total and departures by reason in nativelink_worker_disconnections_total.

Limits that block instead of erroring

If the symptom is "everything got slow and nothing errored", you are probably sitting on one of these:

The FD limit deserves special attention. NativeLink uses only 80% of the limit it actually achieved, not 80% of what you configured. In a container with a hard nofile cap, the semaphore silently shrinks and the only evidence is a startup warning:

The new open file limit (1024) is below the recommended value of 24576.
Consider raising max_open_files.

Or, if the raise failed entirely:

set_open_file_limit() failed to assign open file limit. Maybe system does not
have ulimits, continuing anyway.

Every filesystem operation takes a permit: opens, creates, hard links, chmod, directory creation, directory reads, symlink reads. When permits run out, everything using the filesystem store queues.

Startup failures

These all abort the process before it serves anything. All of them are config wiring, and all of them name the offending key.

Config parsing uses deny_unknown_fields, so a misspelled key is a startup error naming the key, not a silently ignored setting. That is a feature; trust the message.

One string that looks like a startup failure and is not: Could not parse the value of precondition_script: '<v>'. The experimental_precondition_script value is split with shlex before every action, not at startup, so an unparseable value fails each action as it arrives and the worker stays up.

Retries

Retry exhaustion is not a distinct error. It is the last underlying error with a suffix appended:

... On attempt 5

The code and message you see are the cause, and the suffix tells you it had been happening repeatedly.

Non-retryable codes: InvalidArgument, FailedPrecondition, OutOfRange, Unimplemented, NotFound, AlreadyExists, PermissionDenied, Unauthenticated. Everything else retries, which is why a full disk, arriving as Unknown or Internal, burns the whole retry budget before failing.

Two other retry strings: Retry stream ended abruptly on attempt N, and Not retrying permanent error.

Where else to look

Scaling workers has a "when it doesn't work" section covering the fleet-shaped versions of several problems above: queue depth high while workers idle, adding workers not helping, scale-in causing failed actions, two workers on one host behaving erratically. It complements this page rather than duplicating it.

Observability covers the metrics pipeline itself: no metrics appearing, missing cache metrics, out-of-order samples. Note that those are problems with the collector, not with NativeLink.

Security hardening covers what is deliberately absent rather than broken: there is no inbound authentication anywhere, and the committed certificates expired in 2024.

What's next