Production configuration

Who this is for: you have NativeLink working and now other people are going to depend on it. What you'll have at the end: a three-process deployment whose ports, state, and failure behaviour you can explain to someone else. Time: an afternoon, plus however long your change process takes.

A production configuration is not the quickstart configuration with bigger numbers. It differs in one structural way: the CAS, the scheduler, and the workers become three separate processes. Everything else on this page (the port split, the store composition, the hardening checklist) follows from that one decision.

Every config on this page is adapted (paths and hostnames changed) from deployment-examples/docker-compose/, which CI builds and runs end to end on every pull request: run_integration_tests.sh brings the three processes up with docker compose and then drives real Bazel builds through them.

The shape

Three facts about that picture are the whole design:

The scheduler owns no storage. It reaches the CAS through grpc stores, the same way a client would. That is why you can restart it without losing blobs, and why it needs no volume.

The worker also owns no durable storage. Its local disk is a read cache in front of the CAS, not a source of truth. A worker can be deleted mid-action; the scheduler retries the action elsewhere.

Only the CAS process is stateful. It is the one thing you back up, size for growth, and think carefully about before restarting.

Process 1: the CAS and Action Cache

This is the process that holds bytes. Two stores, one for content and one for action results, each with its own eviction budget:

{
  stores: [
    {
      name: "CAS_MAIN_STORE",
      compression: {
        compression_algorithm: { lz4: {} },
        backend: {
          filesystem: {
            content_path: "/var/lib/nativelink/content_path-cas",
            temp_path: "/var/lib/nativelink/tmp_path-cas",
            eviction_policy: {
              // 10gb. This is a cache: size it to your working set,
              // not to your total build output.
              max_bytes: 10000000000,
            },
          },
        },
      },
    },
    {
      name: "AC_MAIN_STORE",
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-ac",
        temp_path: "/var/lib/nativelink/tmp_path-ac",
        eviction_policy: {
          // 500mb. The AC is tiny next to the CAS; it stores
          // result metadata, not outputs.
          max_bytes: 500000000,
        },
      },
    },
  ],
  servers: [
    {
      listener: { http: { socket_address: "0.0.0.0:50051" } },
      services: {
        cas: [{ cas_store: "CAS_MAIN_STORE" }],
        ac: [{ ac_store: "AC_MAIN_STORE" }],
        capabilities: [],
        bytestream: [{ cas_store: "CAS_MAIN_STORE" }],
      },
    },
  ],
}

stores is an array of named stores, not an object map; every store carries its own name, and other parts of the config refer to it by that name. filesystem requires both content_path and temp_path; the config parser uses deny_unknown_fields, so a typo or an invented key is a startup failure rather than a silently ignored line.

Each entry in cas, ac, bytestream, and capabilities carries an optional instance_name (default: the empty string, which is also what Bazel sends by default). If you set a non-empty one, every client has to pass it too (--remote_instance_name= for Bazel). An empty capabilities: [] still answers GetCapabilities with cache capabilities for any instance name; an entry with remote_execution is only needed to advertise remote execution.

Moving off local disk

filesystem is the right terminal store for a single CAS node with a real disk. For a CAS you intend to replicate, put an object store underneath instead and keep the local disk as the fast tier:

{
  name: "CAS_MAIN_STORE",
  fast_slow: {
    fast: {
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-cas",
        temp_path: "/var/lib/nativelink/tmp_path-cas",
        eviction_policy: { max_bytes: 100000000000 },
      },
    },
    slow: {
      experimental_cloud_object_store: {
        provider: "aws",
        region: "us-east-1",
        bucket: "nativelink-prod-cas",
        key_prefix: "cas/",
        retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
      },
    },
  },
}

The store key is experimental_cloud_object_store with a provider discriminator: aws, gcs, azure, ontap, r2, or oci. No s3_store key exists.

The full backend list and the tradeoffs between them are in Storage backends and Store overview.

Process 2: the scheduler

The scheduler holds the queue and matches actions to workers. It reaches the CAS over gRPC rather than owning a store:

{
  stores: [
    {
      name: "GRPC_CAS_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "cas",
      },
    },
    {
      name: "GRPC_AC_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "ac",
      },
    },
  ],
  schedulers: [
    {
      name: "MAIN_SCHEDULER",
      simple: {
        supported_platform_properties: {
          cpu_count: "minimum",
          OSFamily: "priority",
          "container-image": "priority",
          ISA: "exact",
        },
      },
    },
  ],
  servers: [
    {
      // Client-facing. Anyone who can reach this can queue work.
      listener: { http: { socket_address: "0.0.0.0:50052" } },
      services: {
        ac: [{ ac_store: "GRPC_AC_STORE" }],
        execution: [{ cas_store: "GRPC_CAS_STORE", scheduler: "MAIN_SCHEDULER" }],
        capabilities: [{ remote_execution: { scheduler: "MAIN_SCHEDULER" } }],
      },
    },
    {
      // Worker-facing. Never expose this outside the cluster.
      listener: { http: { socket_address: "0.0.0.0:50061" } },
      services: {
        worker_api: { scheduler: "MAIN_SCHEDULER" },
        health: {},
      },
    },
  ],
}

schedulers, like stores, is an array of named entries.

supported_platform_properties is the matching contract between this scheduler and its workers. Each value is a match rule (exact, minimum, priority, or ignore). A worker that advertises a property the scheduler does not list is rejected at registration (Unknown platform property), and an action that requests one is rejected the same way. Getting this out of sync with the workers is the single most common reason actions sit in the queue forever; see platform properties for how the two halves line up.

Process 3: the workers

Workers are the fleet you scale. Note servers: []: a worker serves nothing and needs no inbound port at all. It dials out to the scheduler and the CAS.

{
  stores: [
    {
      name: "GRPC_CAS_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "cas",
      },
    },
    {
      name: "GRPC_AC_STORE",
      grpc: {
        instance_name: "",
        endpoints: [{ address: "grpc://cas.internal:50051" }],
        store_type: "ac",
      },
    },
    {
      name: "WORKER_FAST_SLOW_STORE",
      fast_slow: {
        fast: {
          filesystem: {
            content_path: "/var/lib/nativelink/worker/content_path-cas",
            temp_path: "/var/lib/nativelink/worker/tmp_path-cas",
            eviction_policy: { max_bytes: 10000000000 },
          },
        },
        // "get" means the local disk serves reads but is not written
        // through on upload; inputs get cached, outputs go straight out.
        fast_direction: "get",
        slow: { ref_store: { name: "GRPC_CAS_STORE" } },
      },
    },
  ],
  workers: [
    {
      local: {
        worker_api_endpoint: { uri: "grpc://scheduler.internal:50061" },
        cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
        upload_action_result: { ac_store: "GRPC_AC_STORE" },
        work_directory: "/var/lib/nativelink/worker/work",
        platform_properties: {
          cpu_count: { query_cmd: "nproc" },
          OSFamily: { values: [""] },
          "container-image": { values: [""] },
          ISA: { values: ["x86-64"] },
        },
        use_namespaces: true,
        use_mount_namespace: true,
      },
    },
  ],
  servers: [],
}

worker_api_endpoint, cas_fast_slow_store, work_directory, and platform_properties are all required. upload_action_result takes ac_store directly; it is not nested twice.

work_directory is where actions are staged and executed. It sees every input file and every output file of every action, so put it on the fastest local disk the machine has, and size it for the largest action tree you run rather than the average one.

Which ports are public

health is served alongside worker_api in the reference config. It is the only readiness signal the binary exposes; there is no metrics port to probe.

What's durable and what's disposable

The rule this table encodes: exactly one component is stateful. If you find yourself needing to protect a scheduler's disk or a worker's disk, something has drifted from this shape.

Before you go live

Raise the file-descriptor limit

Every open blob is a file descriptor. The binary gates file opens on a semaphore sized to 80% of the limit it actually achieved, so a CAS at the default limit does not error under load; it waits.

global: {
  // The reference Kubernetes worker sets 524288. The binary's own
  // default is 24576 when `global` is omitted or this is 0.
  max_open_files: 524288,
},

global accepts exactly three keys: max_open_files (required whenever the global block is present), default_digest_hash_function, and default_digest_size_health_check. It has no metrics or tracing section; telemetry is configured by environment variable, below.

GlobalConfigset_open_file_limit

Turn on namespaces on Linux workers

Bazel in particular can leave zombie processes behind on a worker. Two opt-in flags contain that:

workers: [{
  local: {
    use_namespaces: true,
    use_mount_namespace: true,
    // ...the rest of the worker config
  },
}],

use_namespaces puts each action in its own process namespace; use_mount_namespace additionally isolates the worker root in a new mount namespace, and only works when use_namespaces is on. Both are off by default and both are recommended in production. The container running the worker needs the privileges to create namespaces; the reference Compose stack marks the executor service privileged: true for exactly this.

Put TLS on the client-facing listeners

servers: [{
  listener: {
    http: {
      socket_address: "0.0.0.0:50071",
      tls: {
        cert_file: "/etc/nativelink/tls/server.crt",
        key_file: "/etc/nativelink/tls/server.key",
        // Optional: require client certificates (mTLS).
        client_ca_file: "/etc/nativelink/tls/clients-ca.crt",
        client_crl_file: "/etc/nativelink/tls/clients.crl",
      },
    },
  },
  services: { /* ... */ },
}],

Setting client_ca_file makes the listener require a client certificate signed by that CA, and client_crl_file lets you revoke one. That is the whole authorization model on the listener: any client with a valid, unrevoked certificate gets full access to that listener's services. No per-identity allow-list exists and no mapping from certificate subject to permissions. If you need finer-grained access than "trusted or not," put a proxy in front.

The certificates checked into deployment-examples/ are self-signed and named do-not-use-in-prod for a reason. Generate your own.

Point telemetry somewhere

The config file has no metrics section, and the binary has no Prometheus scrape endpoint in the binary. Telemetry is OTLP over gRPC, configured by environment:

# Where to send traces, metrics, and logs. Must include an explicit
# host AND port; the client panics on a URL without both.
export NL_OTEL_ENDPOINT="http://otel-collector.internal:4317"

# Log format: "compact", "json", or anything else for pretty.
export NL_LOG=json

# Standard tracing filter.
export RUST_LOG=info

Prometheus sees NativeLink through an OpenTelemetry Collector that exports to it, or through Prometheus' own OTLP receiver. Observability has the collector config and the Grafana dashboard.

Opt in to cache metrics, if you want them

Execution metrics are emitted unconditionally. Cache metrics are not: a store only reports hits, misses, and errors on its reads, writes, and deletes if you wrap it:

{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: {
      // ...the store you would otherwise have written here
    },
  },
}

Without the wrapper there are no nativelink_cache_* series at all, and any dashboard panel or alert built on them stays empty forever. The wrapper is opt-in because it costs hot-path timing on every operation.

What to watch

Four signals cover most of what goes wrong. All four are recording rules in deployment-examples/metrics/prometheus-recording-rules.yml:

• nativelink:queue_depth: actions waiting, by priority. Rising and staying risen means you need workers; spiking and draining is normal.
• nativelink:worker_utilization: the fraction of workers actually executing. Low utilization with a deep queue means a matching problem, not a capacity problem.
• nativelink:execution_queue_time_p95: how long an action waits before a worker picks it up. This is the number your users experience as "the build farm is slow."
• nativelink:cache_hit_rate: requires the cache_metrics wrapper above.

Observability covers the pipeline that produces these.

What to tune

Once the shape is right, Tuning is the table of levers: the signal you observe, the knob that moves it, and what it costs you.

What to do when it breaks

Troubleshooting indexes symptoms to causes, and Runbooks has the procedure for each of the four incidents this shape actually produces.

FAQ