An autoscaling reference deployment

Who this is for: you have run scaling workers and scaling the CAS by hand, you trust the numbers, and you want the pool to size itself. What you'll have at the end: a worker deployment that grows and shrinks on queue depth, with probes, drain, and disruption budgets wired up. Time: a day to assemble, a week of watching before you trust it.

The two scaling pages tell you what to scale on and why. This page is the deployment: the manifests, the config changes they depend on, and the latency between a build backing up and a pod existing.

What ships, and what this page supplies

Being blunt about this up front, because it changes how you read everything below.

The repository contains no autoscaling artifacts for NativeLink. It has no HorizontalPodAutoscaler, no KEDA ScaledObject, no VPA, and no metrics adapter configuration for any NativeLink workload. The one HPA in the tree scales the OpenTelemetry collector, and the one PodDisruptionBudget protects the same collector. Neither has anything to do with your workers.

Everything on this page is written by these docs against the shipped configuration schema and the shipped recording rules. It is a worked reference, not a file you can point kubectl apply at from a checkout. Treat the YAML as a starting shape you adapt, and read the prerequisites section before you assume any of it drops onto the shipped manifests unchanged.

The reaction-time budget

The signal an autoscaler reads is not emitted by NativeLink. It is derived, several hops downstream, and each hop adds staleness. Here is the whole chain with its shipped defaults.

Worst case, an action that queues right after an export window closes is invisible to the HPA for about two and a half minutes. Then the HPA's own stabilization window applies on top.

Three of those five hops are yours to fix and two are not worth fixing.

The export interval is the big one, and it is an environment variable. The binary builds its meter provider with a periodic exporter and never sets an interval, so it takes the OpenTelemetry SDK default of 60 seconds. Set OTEL_METRIC_EXPORT_INTERVAL on the scheduler pod to bring it down.

init_tracing

The recording-rule interval is the second. nativelink:queue_depth lives in the nativelink_performance group in deployment-examples/metrics/prometheus-recording-rules.yml, and that group is declared interval: 60s. The execution group next to it uses 30s. Either move the rule into a group with a short interval, or skip the recording rule for scaling and point the adapter at the raw series.

The collector batch timeout is 10 seconds in the shipped collector config and is not worth chasing; it overlaps the scrape interval anyway.

Five gaps you close first

The shipped Kubernetes manifests cannot support an HPA as they stand. Not because of anything subtle, but because of five specific absences. Each one is a short fix and all five are prerequisites.

AdminConfig

The health endpoint does less than you expect

Before you write a readiness probe, understand what /status actually measures, because the answer is surprising in both directions.

Health indicators are registered only by leaf stores: memory, filesystem, S3, GCS, Azure, Redis, and Mongo. Every composing store (existence_cache, verify, fast_slow, size_partitioning, compression, dedup, shard, completeness_checking, ref_store) inherits the trait's default register_health, which does nothing and does not recurse into its children. The single exception is cache_metrics, which forwards registration to its backend.

StoreDriver

That has a sharp consequence. In the shipped Kubernetes config, CAS_MAIN_STORE is an existence_cache at the top and AC_MAIN_STORE is a completeness_checking at the top. Neither registers anything. The health registry is empty, /status serialises an empty list, and the endpoint returns 200 OK unconditionally, including when the filesystem underneath is gone.

Now the other direction. When a leaf store is at the top of a composition, what the check costs depends on which leaf it is. The filesystem store (the compose example's AC_MAIN_STORE is a bare filesystem) only stats its content_path, with a two-second ceiling. GCS does a single object_exists on a fixed never-present key, Redis sends PING under health_check_timeout_ms, and Mongo pings the database. Memory, S3 and Azure fall back to the trait's default, which is a full round trip: it generates one megabyte of deterministic data, writes it with update_oneshot, calls has, reads it back, and compares.

check_health check_health

The digest is seeded from the store name, so it is the same blob every time and the store does not grow. But against S3 or Azure that is a PUT, a HEAD and a GET of a megabyte on every probe, per store, per replica. At a ten-second probe period on three replicas that is eighteen megabytes a minute of pure probe traffic and a billable request count to match.

global.default_digest_size_health_check lowers the payload. The round trip itself you cannot turn off short of not enabling the health service. Whatever the probe, the endpoint returns 503 if any registered indicator reports failed or timeout (HealthConfig.timeout_seconds, default 5 s, bounds each check) and 200 otherwise.

Build the deployment

1. Make the signal fresh. Two changes, one on the scheduler pod and one in Prometheus.

   # scheduler Deployment, container env
   env:
     - name: OTEL_EXPORTER_OTLP_ENDPOINT
       value: http://otel-collector-collector.default.svc:4317
     - name: OTEL_EXPORTER_OTLP_COMPRESSION
       value: zstd
     - name: OTEL_METRIC_EXPORT_INTERVAL
       value: "15000"   # milliseconds; SDK default is 60000

   # a rule group of your own, evaluated fast enough to scale on
   groups:
     - name: nativelink_autoscaling
       interval: 15s
       rules:
         - record: nativelink:queue_depth_fast
           expr: |
             sum by (instance_name) (
               nativelink_execution_active_count{execution_stage="queued"}
             )

   Keep the shipped nativelink:queue_depth for dashboards. Scale on the fast copy, summed without execution_priority so the HPA sees one series per instance rather than one per priority band.

2. Give the worker a server so it can be probed. Every shipped worker config has servers: [], which means the process listens on nothing at all. Add a minimal server with only the health service on it.

   servers: [{
     name: "worker_health",
     listener: {
       http: {
         socket_address: "0.0.0.0:50070",
       },
     },
     services: {
       health: {},
     },
   }],

   The worker's own stores register health the same way any other process's do. In the shipped worker configs the named stores are a fast_slow and two grpc stores, none of which registers an indicator, so the report is empty and the endpoint answers 200 whenever the process is up. Read it as a liveness signal ("the process is up and serving"), not as a statement about the CAS.

3. Name the worker after its pod. LocalWorkerConfig.name is the prefix of the worker id the scheduler assigns, in the form {name}{uuidv6}, and it is shell-expanded, so the downward API can populate it.

   workers: [{
     local: {
       name: "${POD_NAME:-worker}-",
       max_inflight_tasks: 16,
       // ...
     },
   }],

   env:
     - name: POD_NAME
       valueFrom:
         fieldRef:
           fieldPath: metadata.name

   LocalWorkerConfig

   This is what makes a worker id traceable back to a pod, in logs and in the execution_worker_id metric attribute. You need it for step 7.

4. Add resources and probes to the worker.

   containers:
     - name: nativelink-worker
       image: nativelink-worker
       ports:
         - name: health
           containerPort: 50070
       resources:
         requests:
           cpu: "4"
           memory: 8Gi
         limits:
           memory: 8Gi
       readinessProbe:
         httpGet:
           path: /status
           port: health
         periodSeconds: 30
         failureThreshold: 2
       livenessProbe:
         httpGet:
           path: /status
           port: health
         periodSeconds: 30
         failureThreshold: 4

   Set requests.cpu to match the concurrency max_inflight_tasks implies, not to the node size. A CPU limit is deliberately absent: a compile that gets throttled at the cgroup boundary shows up as a slow action rather than a failure, which is harder to diagnose than an occasional noisy neighbour.

5. Enable the admin service on the scheduler. It is a service entry, and it is absent from every kubernetes/ config. Put it on the worker-API server block, which is already on its own port and its own permission boundary.

   {
     listener: {
       http: {
         socket_address: "0.0.0.0:50061",
       },
     },
     services: {
       worker_api: {
         scheduler: "MAIN_SCHEDULER",
       },
       admin: {},
       health: {},
     },
   },

   The admin path has no authentication

   The drain route is plain HTTP with no auth of any kind. Keep it on the internal worker-API port, keep that port off any Ingress, and restrict it with a NetworkPolicy to the namespace that runs your drain job.

6. Wire the external metric to an HPA. Kubernetes cannot read Prometheus directly; something has to serve the external.metrics.k8s.io API. Nothing in the repository does this, so this is a Prometheus Adapter configuration written for the purpose.

   # prometheus-adapter values
   rules:
     external:
       - seriesQuery: 'nativelink:queue_depth_fast'
         resources:
           overrides:
             namespace: {resource: "namespace"}
         name:
           matches: "^nativelink:queue_depth_fast$"
           as: "nativelink_queue_depth"
         metricsQuery: 'max(<<.Series>>{<<.LabelMatchers>>})'

   apiVersion: autoscaling/v2
   kind: HorizontalPodAutoscaler
   metadata:
     name: nativelink-worker
   spec:
     scaleTargetRef:
       apiVersion: apps/v1
       kind: Deployment
       name: nativelink-worker
     minReplicas: 3
     maxReplicas: 60
     metrics:
       - type: External
         external:
           metric:
             name: nativelink_queue_depth
           target:
             type: AverageValue
             averageValue: "8"
     behavior:
       scaleUp:
         stabilizationWindowSeconds: 30
         policies:
           - type: Percent
             value: 100
             periodSeconds: 60
           - type: Pods
             value: 8
             periodSeconds: 60
         selectPolicy: Max
       scaleDown:
         stabilizationWindowSeconds: 600
         policies:
           - type: Pods
             value: 2
             periodSeconds: 120

   AverageValue: "8" means "aim for eight queued actions per worker": the HPA divides the metric by the replica count, so the target is a per-worker backlog rather than an absolute queue length. Pick the number from the per-worker throughput you measured in scaling workers; eight is a placeholder, not a recommendation.

   The asymmetry between the windows is deliberate and matters more than the thresholds. Scaling up fast costs money. Scaling down fast costs work, because every worker you remove mid-action produces retries.

7. Drain on scale-in, as far as the API allows. This is the part where honesty is more useful than a clean recipe.

   The drain endpoint takes a worker id the pod does not know at startup; the scheduler generates it at registration and returns it over the wire (the worker logs it once, in its Worker registered with scheduler line).

   POST {admin_path}/scheduler/{scheduler_name}/set_drain_worker/{worker_id}/1

   The second path segment is the scheduler's name from the schedulers list (MAIN_SCHEDULER in the shipped config), not the REAPI instance name. Because you set name to the pod name in step 3, the id is discoverable: it is the execution_worker_id label on the execution series the scheduler emits for that worker's actions, prefixed with the pod name. nativelink_execution_active_count does not carry that label (it is attributed by stage only); nativelink_execution_stage_transitions_total and nativelink_execution_completed_count_total do. A preStop hook can look it up and call drain.

   terminationGracePeriodSeconds: 1800
   lifecycle:
     preStop:
       exec:
         command:
           - /bin/sh
           - -c
           - |
             WID=$(wget -qO- "http://prometheus:9090/api/v1/query?query=\
             nativelink_execution_stage_transitions_total%7Bexecution_worker_id%3D~%5C%22\
             ${POD_NAME}-.%2A%5C%22%7D" \
               | sed -n 's/.*"execution_worker_id":"\([^"]*\)".*/\1/p' | head -1)
             if [ -n "$WID" ]; then
               wget -qO- --post-data='' \
                 "http://nativelink:50061/admin/scheduler/MAIN_SCHEDULER/set_drain_worker/$WID/1"
             fi
             sleep 1500

   Three caveats, all of which you should weigh before adopting this.

   It only finds an id once that worker has reported at least one action to the scheduler since the scheduler started; the counter it queries is cumulative, so a worker that ran something an hour ago is still findable. A worker that has never run anything has no series, the lookup returns nothing, and the hook falls through, which is harmless, because such a worker has nothing to drain.

   It depends on Prometheus being reachable and current from inside a terminating pod. If that is unacceptable in your environment, drop the lookup and rely on terminationGracePeriodSeconds alone.

   The sleep is what actually holds the pod open while in-flight actions finish; the drain call only stops new work arriving. Size the grace period to your longest action, and cross-check it against max_action_timeout_s on the worker.

   Without any of this, work is still not lost

   A worker that disappears mid-action has its actions requeued with worker affinity cleared, up to max_job_retries. Drain turns a retry storm into a clean handover; it is not the difference between correct and incorrect.

8. Protect the pool from everything that isn't the HPA. Voluntary disruptions (node upgrades, cluster autoscaler consolidation) will otherwise take workers out in parallel with your scale-in.

   apiVersion: policy/v1
   kind: PodDisruptionBudget
   metadata:
     name: nativelink-worker
   spec:
     minAvailable: 50%
     selector:
       matchLabels:
         app: nativelink-worker
   ---
   # in the worker pod spec
   topologySpreadConstraints:
     - maxSkew: 1
       topologyKey: kubernetes.io/hostname
       whenUnsatisfiable: ScheduleAnyway
       labelSelector:
         matchLabels:
           app: nativelink-worker

   The shipped worker manifest has neither, so replicas: 3 can and does land three workers on one node.

9. Leave the CAS out of the autoscaler. The HPA scales workers and nothing else. A CAS process holds per-process state that makes a second replica a project rather than a replica count; scaling the CAS covers which state and why. What the CAS needs from this deployment is a readiness probe with enough patience for its startup scan.

   readinessProbe:
     httpGet:
       path: /status
       port: 50061
     periodSeconds: 10
     failureThreshold: 60      # ten minutes of scan
     initialDelaySeconds: 10

   The filesystem store walks its entire content directory before the process binds any socket, so during a restart the probe gets connection-refused rather than a 503. A short failureThreshold turns a slow restart into a crash loop.

What this deployment does not do

Worth stating so the gaps are choices rather than surprises.

It does not scale the CAS or the scheduler. Both are single-replica in this reference. The scheduler's default backend is in-memory, so a second scheduler is a second independent queue, not a bigger one.

It does not do predictive or scheduled scaling. Queue depth is reactive by construction: pods start after work is already waiting. If your builds arrive on a known schedule, a CronJob that raises minReplicas before the wave beats any metric-driven policy.

It does not give you node capacity. An HPA creates pods; something has to create nodes. Cluster Autoscaler or Karpenter underneath, sized so that maxReplicas is actually reachable.

It does not survive a move to the Redis scheduler backend. execution.active.count, the series this whole page depends on, is emitted only by the in-memory awaited-action store. Under experimental_backend, the queue-depth series is not emitted at all and the HPA has nothing to read.

When it doesn't work

NextObservability

The pipeline this deployment reads from: how metrics get from the binary to Prometheus, what the catalog contains, and what the shipped dashboards show.

SidewaysTuning

The lever table. Once the pool sizes itself, tuning is what decides how much each of those pods is worth.