Servers and services

Who this is for: anyone deciding what a NativeLink process should expose, and on which port. What you'll know at the end: what a listener is, what each of the eleven services does, how instance_name partitions a cache, and why the worker API must not share a port with anything else. Time: thirty minutes.

A server is a listener plus a set of servicesCopy Anchor Link

The servers array is the only entry point into a NativeLink process. Each entry binds one socket and decides what is reachable on it:

servers: [
  {
    name: "public",
    listener: {
      http: {
        socket_address: "0.0.0.0:50051",
      },
    },
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      capabilities: [{ instance_name: "main" }],
    },
  },
],

Four things are going on there, and each is worth naming.

name is documented as a label for telemetry and logs, defaulting to the server's index in the array. Set it; it's the cheapest documentation in the file.

listener currently has exactly one variant, http, which serves HTTP/HTTP2 and therefore gRPC. socket_address is parsed as a Rust SocketAddr, so it must be an IP literal plus port: 0.0.0.0:50051 to bind every IPv4 interface, [::]:50051 for IPv6, 127.0.0.1:50051 for loopback only. A hostname or a bare :port is rejected as an invalid address. TLS, HTTP/2 tuning, and compression are options on the listener, not on the services.

services is optional in the schema, but the binary refuses to start a server without it ('services' must be configured), so in practice every server entry carries one.

Each service entry names a store or a scheduler as a string, the name from the stores or schedulers array. This is where the wiring described in the section overview actually happens.

The eleven servicesCopy Anchor Link

In development builds, ByteStream rejects a completed upload whose byte count differs from the size in its CAS resource name, even without a verify store. See upload integrity.

Development implementation.

A cache-only deployment needs the first four. Adding execution adds execution and worker_api. The rest are opt-in.

capabilities is the one that surprises people: it carries almost no config, but Bazel queries it before doing anything else and fails the build if the call is not served, with an error about querying capabilities rather than one naming a missing service. And if capabilities has no remote_execution block, the response carries no execution_capabilities, so the client is told this endpoint caches but does not execute, which is correct for a cache-only Getting started deployment and a silent bug once you add Remote execution.

capabilities: [
  {
    instance_name: "main",
    remote_execution: { scheduler: "MAIN_SCHEDULER" },
  },
],

Download remote assets on a cache missCopy Anchor Link

In development builds, fetch can download HTTP and HTTPS assets when a URI mapping is missing or its CAS blob has been evicted. This supports a shared dependency downloader for short-lived CI clients, as discussed in issue #2770. Omitting downloader keeps the existing lookup-only behavior.

fetch: [{
  instance_name: "main",
  fetch_store: "DOWNLOAD_INDEX",
  downloader: {
    cas_store: "CAS_MAIN_STORE",
    http_proxy_secret_file: "/run/secrets/fetch-proxy-url",
    no_proxy: ["localhost", "127.0.0.1", ".example.internal"],
    max_size_bytes: 0, // No size limit; set a positive value to enforce one.
    max_concurrent_downloads: 4,
    timeout_s: "300s",
  },
}],

cas_store must be the CAS exposed to the same client instance. The mapping store is a dedicated download index: it contains URI/qualifier metadata, not artifact bytes. Do not share this index with a Push service. A cache hit checks CAS availability before returning the digest; a missing blob triggers another download. The existing mapping expiry is also checked.

Configure either http_proxy (an HTTP and HTTPS forward proxy URL) or http_proxy_secret_file (a file containing that URL). The file is read once at startup and surrounding whitespace is removed. The default HTTP client connects directly. An explicit proxy adds a concurrent attempt; the first fully verified CAS upload wins, and a failing path does not cancel the other path. This preserves availability when only one network path can reach an origin. no_proxy controls the configured proxy's exceptions; an empty list bypasses no hosts. Proxy settings come only from configuration: HTTP_PROXY, HTTPS_PROXY, ALL_PROXY and NO_PROXY environment variables (including lowercase variants) are ignored. Clients, limits and timeouts belong to each Fetch instance. HTTPS origins use the forward proxy's CONNECT support. Both clients disable automatic Referer headers so redirects do not disclose credentials in the source URL.

The downloader tries candidate URIs in order. checksum.sri is verified using its declared algorithm. The existing digest implementations support sha256- and blake3- values. A checksum.sri value may contain whitespace-separated alternatives; at least one supported checksum must match. Unsupported algorithms are ignored when a supported alternative is present. A list with no supported algorithm, malformed checksums, or duplicate checksum.sri qualifiers is rejected before cache lookup. Requests without a checksum are allowed by default, matching Bazel's optional sha256 and integrity attributes. Set require_checksum: true to require one.

bazel.canonical_id is accepted as URL-index identity metadata. The downloader rejects http_header: and http_header_url: qualifiers before cache lookup or downloading because it does not implement their HTTP header semantics. Other qualifiers are accepted as opaque URL-index metadata by default; set allow_unknown_qualifiers: false to reject them. This switch is independent of require_checksum and also applies on cache hits. Lookup-only mode is unchanged.

The index is keyed by URI plus qualifiers and stores a complete digest, including its size. Different URI/qualifier keys have independent mappings. When none of the supplied URIs has a usable mapping, the selected candidate is downloaded even if its checksum matches bytes already in CAS; only that URI is registered. Repeating the same URI and qualifiers reuses its record when the mapping is unexpired and the CAS blob is still present. New download records include fetched_at, the time origin retrieval started. When a request supplies oldest_content_accepted, reuse also requires fetched_at to be at or after that time. Older records without a timestamp remain usable for requests without a freshness constraint; constrained requests download again and update the record. Bazel requests without a checksum can deliberately require a future timestamp to prevent cache reuse. Checksum verification protects integrity during download; it does not perform hash-only CAS lookup or create mappings for other mirror URLs.

The CAS digest follows the requested digest function, falling back to the server's default when none is specified. When the checksum uses the same algorithm, one hash calculation serves both purposes; each different checksum algorithm is computed once, even when several alternatives use it. HTTP errors, oversized bodies and checksum mismatches are not published to CAS. Empty bodies are valid blobs.

Each active download spools to an anonymous temporary file before publishing verified bytes in bounded chunks to CAS. File size is unlimited by default: max_size_bytes: 0 disables the size limit. Set a positive value to reject larger objects, including responses without Content-Length. The concurrency limit bounds active requests, but temporary disk usage has no fixed size bound unless an object limit is configured. A request may run two concurrent attempts when a proxy is configured, so with an object limit the disk bound is twice that limit times the active request limit. Timeouts cover waiting for a download slot, all URI attempts and CAS publication. A caller timeout is clamped to the configured maximum. Cancellation releases the file and slot. No detached background fetch continues after RPC cancellation.

Enabling downloads grants callers access to the server's outbound HTTP and HTTPS network, including redirects. Use trusted clients and network egress rules to restrict reachable destinations; this downloader is not an SSRF filter. The default path remains eligible even when an explicit proxy is configured; this configuration is not an enforced outbound-proxy boundary. See the configuration reference for the fields and defaults.

Development implementation.

Instance namesCopy Anchor Link

Most services are declared as an array, and each element carries an instance_name. That field is a routing key: it selects which store (or scheduler) serves a request. The instance name is not part of the storage key, so two instance names pointing at different stores are fully isolated, and two pointing at the same store share every blob.

cas: [
  { instance_name: "", cas_store: "CAS_MAIN_STORE" },
  { instance_name: "main", cas_store: "CAS_MAIN_STORE" },
],

That pattern (the same store registered twice, once under the empty name and once under main) appears throughout the shipped examples, and it relies on the sharing: clients differ in what they send, some omit the instance name entirely, and registering both means either kind of client works, against one cache, without a config change on their side.

Pick one name per logical environment. main for the production cache, <repo-name> for per-repo isolation, <something>-experiments for a sandbox that shouldn't pollute the real cache, each pointed at its own store. Every client targeting a given cache must use the same one.

The port splitCopy Anchor Link

The worker_api service is different from every other service on this page, and the difference is a security boundary rather than a design preference.

A client calling the CAS can upload and fetch blobs. A worker connected to the worker API is handed actions to execute and reports their results as authoritative. Those are not the same permission, and the worker API has no authentication of its own to distinguish them.

The rule: worker_api gets its own listener, on its own port, with no other service sharing it. The shipped examples put the public API on :50051 and the worker API on :50061, and the second port is the one you never expose beyond the cluster:

servers: [
  {
    name: "public",
    listener: { http: { socket_address: "0.0.0.0:50051" } },
    services: {
      cas: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      ac: [{ instance_name: "main", ac_store: "AC_MAIN_STORE" }],
      bytestream: [{ instance_name: "main", cas_store: "CAS_MAIN_STORE" }],
      execution: [{
        instance_name: "main",
        cas_store: "CAS_MAIN_STORE",
        scheduler: "MAIN_SCHEDULER",
      }],
      capabilities: [{
        instance_name: "main",
        remote_execution: { scheduler: "MAIN_SCHEDULER" },
      }],
    },
  },
  {
    name: "private_workers_servers",
    listener: { http: { socket_address: "0.0.0.0:50061" } },
    services: {
      worker_api: { scheduler: "MAIN_SCHEDULER" },
      admin: {},
      health: {},
    },
  },
],

Note that worker_api takes a single object rather than an array; it has no instance_name, because it isn't part of the client-facing protocol at all.

admin and health sit on the private port in the examples for the same reason. A build event protocol endpoint (experimental_bep), when you enable it, can go on its own listener so it can be scaled and secured independently of both.

TLS, when you need itCopy Anchor Link

TLS is a property of the listener:

listener: {
  http: {
    socket_address: "0.0.0.0:50071",
    tls: {
      cert_file: "/etc/nativelink/tls/server.crt",
      key_file: "/etc/nativelink/tls/server.key",
    },
  },
},

Adding client_ca_file turns that into mutual TLS (clients must present a certificate signed by that CA) and client_crl_file lets you revoke one. That's the closest thing NativeLink has to built-in client authentication, and it is a reasonable answer for a cache exposed beyond a trusted network.

FAQCopy Anchor Link