Autonomous operation

Who this is for: whoever would otherwise be paged about the build farm. What you'll have at the end: an accurate picture of how close NativeLink gets you to a hands-off farm, and exactly which parts you build. Time: about fifteen minutes.

The shape of the problem

Build load is spiky in a way that punishes static capacity. It is near zero at night, ragged through the morning, and vicious in the hour before a release. A farm sized for the peak is idle most of the time and expensive all of the time. A farm sized for the average has a queue during every moment anybody cares about.

The way out is a farm that changes size on its own. That is not a NativeLink feature you switch on. It's a property that emerges when the components have the right characteristics, and most of this page is about which characteristics those are and which ones you have to supply.

Why workers can be treated as disposable

Autoscaling only works if adding and removing a machine is uneventful. Three properties of the worker make it so.

Workers hold no authoritative state. A worker executes actions and writes results to the CAS. Everything durable lives in the store, not on the worker, so a worker that vanishes takes nothing with it that isn't reproducible.

Workers announce themselves. A worker dials the scheduler and registers, rather than the scheduler being told a list of workers. Starting five more is enough; there is no membership to update. That's why running multiple workers is mostly a matter of starting more processes.

Lost work is retried, not lost. When a worker disappears mid-action, the scheduler notices, increments an attempt counter, and requeues the action, with its worker affinity cleared, so it goes to whichever worker is free rather than waiting for a machine that is gone.

That last property is what makes preemptible and spot instances viable, and it is the single largest cost lever available. The retry budget is max_job_retries on the scheduler, defaulting to 3.

A worker is declared dead when it stops keepaliving for worker_timeout_s, which defaults to 5 seconds and is checked once a second. That is a deliberately short leash: fast detection means fast requeue, and the cost of a false positive is one repeated action rather than a stall.

Draining, for the shutdown you know about

Retries handle the machines that die without warning. Machines that give you notice deserve better, and there's a specific hook for it.

The admin interface exposes a drain endpoint:

POST {admin_path}/scheduler/{instance_name}/set_drain_worker/{worker_id}/1

A draining worker accepts no new actions and finishes the ones it is running. That is exactly the right response to a spot-termination notice or a rolling node upgrade: call it when the notice arrives, and the in-flight work completes instead of being thrown away and redone.

Nothing in the repository wires this to any cloud provider's interruption signal. The handler is a short script you write, and it is the highest-value piece of glue on this page.

What the scheduler remembers, and what it forgets

This is the constraint that shapes a hands-off design, so it's worth being precise.

By default the scheduler keeps its queue in process memory. Restarting it loses the in-flight queue. Clients retry and the work gets done, so this is not data loss in any lasting sense, but it does mean the scheduler is the one component you cannot casually cycle under load, and that a single scheduler is a single point of interruption.

A Redis-backed alternative exists, and the configuration names it honestly: the field is experimental_backend. It moves queue state out of the process so schedulers stop being special. It also carries the rough edges the name implies: there's a fallback_match_interval_s in the config specifically to paper over eventually-consistent searches.

The signal you scale on

An autoscaler needs a number that says "more work is waiting than there is capacity for." That number is queue depth: actions sitting in the queued stage rather than executing.

NativeLink emits it as execution.active.count, an up-down counter attributed by execution.stage.

Three things about that metric decide your architecture, and all three surprise people:

Telemetry is OTLP push, not Prometheus pull. The binary has no /metrics endpoint to scrape. It pushes OpenTelemetry to a collector, and the collector is what a Prometheus-shaped world reads from. deployment-examples/metrics/ is the working example of that pipeline.

Queue depth is a derived series, not a metric name. What you scale on is a recording rule that sums the counter filtered to the queued stage. The example rules in deployment-examples/metrics/prometheus-recording-rules.yml define nativelink:queue_depth, nativelink:worker_utilization and nativelink:actions_per_worker. An HPA reads the recording rule through an external-metrics adapter; there is no metric the binary emits that an HPA can target directly.

The metrics module is Business Source Licensed. nativelink-util/src/metrics.rs carries a header requiring an enterprise license agreement, unlike the Functional Source License covering most of the repository. Production observability is exactly the case that header describes, so settle the licensing question before you design around it.

What the repository ships, and what it doesn't

Being direct, because this is the part most likely to be assumed:

The repository contains no autoscaling manifests for NativeLink. The Helm charts for Kubernetes ship with NativeLink Enterprise, not with this repository. The Kustomize building blocks under kubernetes/ set fixed replica counts. No HorizontalPodAutoscaler for workers, no KEDA ScaledObject, no cluster-autoscaler or Karpenter configuration, and no node pools or tolerations for build workers. The one HPA in the tree scales the OpenTelemetry collector, not NativeLink.

What the repository does give you is the harder half: a worker that is safe to kill, a scheduler that requeues what it loses, a drain hook for planned shutdown, a queue-depth signal, and a working metrics pipeline to carry it.

What you assemble is the standard Kubernetes half: an HPA on the derived queue metric via an external-metrics adapter, a cluster autoscaler or Karpenter for the nodes underneath, spot node pools with the drain hook wired to interruption notices, and node affinity for heterogeneous pools. These are not NativeLink concepts, which is the point: nothing here needs a bespoke controller.

What "no human in the loop" actually means

A realistic end state, in the order the pieces earn their keep:

Workers scale on queue depth and fall to a small floor overnight. Most of the fleet is preemptible, because retries make interruption a cost rather than an incident, and the drain hook converts warned shutdowns into clean ones. Node capacity follows pod demand through your existing autoscaler. The CAS scales separately and more conservatively, because it holds the state everything else depends on. Alerts fire on the things automation cannot resolve (sustained queue depth that scaling isn't clearing, error rates, storage approaching its limit) rather than on capacity, which now handles itself.

The queue-depth alert is the one to get right. It's the signal that your autoscaler has hit a ceiling it cannot pass, whether that's a quota, a node pool's maximum, or an exhausted spot market. Everything else is noise by comparison.

Common questions