Stores

Who this is for: anyone who has seen a NativeLink config with a store nested five levels deep and wanted to know why. What you'll know at the end: how to declare a store, how to name one, how composition works, and which layers are worth reaching for. Time: thirty minutes.

A store is one entry in an array

The stores array holds every storage backend the process can use. Each entry is an object with a name and exactly one type key:

stores: [
  {
    name: "CAS_MAIN_STORE",
    memory: {
      eviction_policy: { max_bytes: 1000000000 }, // 1 GB
    },
  },
  {
    name: "AC_MAIN_STORE",
    memory: {
      eviction_policy: { max_bytes: 100000000 }, // 100 MB
    },
  },
],

That's a complete, working cache backend pair: a content-addressable store for blobs and an action cache for results. Everything lives in RAM, so a restart wipes it, which is exactly right for a demo or a short-lived CI runner, and exactly wrong for anything else.

The name is arbitrary. CAS_MAIN_STORE and AC_MAIN_STORE are conventions the shipped examples use, not keywords; the services in your servers block will refer to whatever names you pick.

Backing it with disk

Swap the type key and the same two stores survive a restart:

stores: [
  {
    name: "CAS_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/cas",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 50000000000 }, // 50 GB
    },
  },
  {
    name: "AC_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/ac",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 500000000 }, // 500 MB
    },
  },
],

content_path is where the bulk of the data lives. temp_path is a staging area for in-flight writes and deletes, and it must be on the same block device as content_path so a completed write can be moved into place atomically rather than copied. Everything in temp_path is deleted on every startup.

On boot, content_path is scanned and every file found is added back to the cache; that scan is what makes the store durable across restarts.

Stores compose

Here is the idea the rest of this page is about. Most store types don't store anything. They wrap another store and change its behaviour, and the store they wrap is written inline, as a full store spec:

{
  name: "CAS_MAIN_STORE",
  compression: {
    compression_algorithm: { lz4: {} },
    backend: {
      filesystem: {
        content_path: "/var/lib/nativelink/cas",
        temp_path: "/var/lib/nativelink/tmp",
        eviction_policy: { max_bytes: 100000000000 }, // 100 GB
      },
    },
  },
},

compression has no storage of its own. It compresses on write, decompresses on read, and hands the bytes to whatever is in backend. The client sees raw bytes either way. Build artifacts are usually highly compressible, so this saves disk in exchange for CPU; the only algorithm offered is lz4, which is fast and gives up early on data that won't compress.

Because the wrapped store is a full spec rather than a reference to a type, wrappers nest arbitrarily. A verify around a size_partitioning whose upper half is a dedup whose index is a fast_slow of memory-over-disk is not an exotic config; it's nativelink-config/examples/filesystem_cas.json5, and each layer is doing one comprehensible job.

ref_store: naming instead of nesting

Nesting alone can't express sharing. If two stores should write to the same place, inlining the backend twice gives you two independent stores that happen to have identical settings.

ref_store is the escape hatch. It's a store type whose entire body is the name of another top-level store:

stores: [
  {
    name: "FS_CONTENT_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/content",
      temp_path: "/var/lib/nativelink/tmp",
      eviction_policy: { max_bytes: 2000000000 },
    },
  },
  {
    name: "CAS_MAIN_STORE",
    size_partitioning: {
      size: 262144, // 256 KB
      lower_store: { ref_store: { name: "FS_CONTENT_STORE" } },
      upper_store: { ref_store: { name: "FS_CONTENT_STORE" } },
    },
  },
],

Both halves now land in the same physical store. Swap either ref_store for an inline spec and they diverge.

Two properties of ref_store are worth knowing before you use it:

Order doesn't matter. A store can reference a name declared later in the array. nativelink-config/examples/redis.json5 has AC_MAIN_STORE referring to CAS_MAIN_STORE, which is declared below it. References are resolved in a post-init pass after every store has been constructed.

A typo is not a parse error. deny_unknown_fields catches a misspelled field, but ref_store: { name: "CAS_MAIN_STROE" } is structurally valid; the failure surfaces in that post-init pass (Failed to find store 'CAS_MAIN_STROE' in StoreManager in RefStore), still at startup and still before any socket is bound, but after the parse itself has succeeded.

The layers worth knowing

You don't need all eighteen store types to write a good config. These are the ones that appear in nearly every non-trivial one:

Two of these are load-bearing for correctness rather than performance. verify with verify_hash: true catches a blob whose content doesn't match the digest it was uploaded under. completeness_checking wraps an action cache and checks the CAS still holds every blob a cached result refers to; without it, an eviction in the CAS turns cached results into builds that fail on a missing output.

That's why most of the shipped backend examples (filesystem_cas.json5, s3_backend_with_local_fast_cas.json5, and the other object-storage examples) wrap the CAS in verify, why redis.json5 wraps the AC in completeness_checking, and why you should do both.

The full list, with every field and default, is in the store reference. Recipes for specific backends (object storage, Redis, OCI registries) live in How-to guides.

What keys the store

A store is content-addressed, so what goes in a key is the digest, and every server, worker, and build client sharing a cache must compute that digest the same way. SHA-256 and BLAKE3 produce different keys for identical bytes, so uploads made under one function are cache misses for clients using the other.

NativeLink defaults to SHA-256. Declare it explicitly anyway, in the global block, so the file says what it depends on:

{
  global: {
    max_open_files: 24576,
    default_digest_hash_function: "sha256",
  },
  // ... stores, servers, schedulers, workers ...
}

Then match it on the client. For Bazel, in .bazelrc:

startup --digest_function=sha256

NativeLink accepts uploads whose client-declared digest function differs from the server default, but logs a warning the first time it sees each mismatch: mixed digest functions divide the cache and reduce shared hits.

max_open_files is in that block for a related reason: a filesystem store holds file descriptors. At startup NativeLink tries to raise its own soft nofile limit to this value (default 24576) and sizes an internal semaphore from whatever the kernel actually granted, so if the hard limit is lower the effective value is lower too, and a warning is logged.

Which function to pick

We compared both algorithms on an Apple M2 Max with 12 CPU cores and 32 GB of memory, running macOS 15.6 and Bazel 9.1.1. Each of two alternating runs built //:nativelink against a fresh local NativeLink filesystem cache with hash and size verification enabled. Uploads were synchronous, and the Bazel output and NativeLink cache directories were isolated by algorithm.

Both population builds performed 2,620 actions. Each cached rebuild reported 1,458 remote cache hits. SHA-256 was 1.9% faster while populating the cache and 6.3% faster on the cached rebuild in this test, so SHA-256 remains the default.

These are whole-build measurements from one macOS machine, not a universal hashing benchmark; compilation, storage, scheduling, and operating-system caching dominate much of the elapsed time. Measure your own representative workload before changing algorithms. Keeping every participant aligned matters far more than a small isolated timing difference.

FAQ