Filesystem store

Who this is for: anyone who wants the cache to survive a restart without paying for object storage, and anyone building the fast tier of a larger composition. What you'll have at the end: a filesystem store sized to your disk, with eviction that actually runs. Time: fifteen minutes.

The recipe

stores: [
  {
    name: "CAS_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/cas/content",
      temp_path: "/var/lib/nativelink/cas/tmp",
      eviction_policy: {
        max_bytes: "50gb",
        evict_bytes: "5gb",
      },
    },
  },
  {
    name: "AC_MAIN_STORE",
    filesystem: {
      content_path: "/var/lib/nativelink/ac/content",
      temp_path: "/var/lib/nativelink/ac/tmp",
      eviction_policy: {
        max_bytes: "5gb",
      },
    },
  },
],

Create the parent directory and start NativeLink. The store scans content_path on boot and adopts every file whose name parses as a store key (<hash>-<size> under d/, or a plain string key under s/); a file whose name does not parse is deleted. The scan reads metadata, not content, so it does not check that bytes still match their digest. That scan is what makes the cache survive restarts, and it is also why boot time grows with the number of cached objects.

The two paths

content_path holds the data. temp_path holds objects mid-upload and mid-delete, while their contents cannot yet be trusted.

Everything in temp_path is deleted on every startup. Nothing you care about should ever be written there by hand.

The eviction policy you cannot skip

eviction_policy is optional in the schema and mandatory in practice. Without it, nothing is ever removed and the store grows until the disk is full.

Eviction is LRU: touching an entry refreshes its timestamp, and inserts run the policy until the store fits again. Four independent limits are available, all defaulting to 0, which means "don't evict on this axis":

Set evict_bytes to something like 10% of max_bytes. Without it, the store sits exactly at the limit and every single insert triggers an eviction; a low watermark turns that thrash into one bulk pass per 10%.

Size accounting uses block_size (default 4 KiB), not the byte length of the file, because a one-byte object still consumes a whole block. If your filesystem uses a different block size, set it, or your max_bytes will be a significant underestimate of real disk use.

Tuning that matters on real hardware

max_concurrent_writes (default 0, unlimited). Every write streams into a temp file and calls sync_all(). Enough concurrent writes will saturate disk I/O and start blocking the async runtime. On a busy shared CAS, set this to something bounded (a few dozen) rather than leaving it open.

read_buffer_size (default 32 KiB). Leave it alone unless you are benchmarking.

evict_page_cache (default false). When true, the store advises the kernel to drop the page cache for each blob after it is read or written.

If this store feeds a worker

A worker's cas_fast_slow_store must have a filesystem store as its fast half. The worker builds each action's input tree by hard-linking files out of that store, and hard links require real files on a real filesystem; a memory or object store cannot supply them.

The worker's work_directory must be on the same filesystem as that store's content_path, for the same reason. See Schedulers and workers.

Steps

1. Create the directories and make sure the process user owns them: mkdir -p /var/lib/nativelink/cas/content /var/lib/nativelink/cas/tmp.

2. Add the store with both paths under one parent and an eviction_policy whose max_bytes leaves headroom on the device.

3. Start NativeLink and run a build against it.

4. Restart NativeLink and run the same build again. The second run should be a cache hit.

When it doesn't work