Content-defined chunking

Who this is for: anyone re-transferring large artifacts that change a little at a time: container layers, linked binaries, archives. What you'll have at the end: clients uploading and downloading only the chunks that changed. Time: twenty minutes.

When a large build output changes slightly (a relinked binary, a container layer with one file modified), its digest changes, and a conventional remote cache re-transfers the whole blob. Content-defined chunking (CDC) splits blobs into chunks at content-derived boundaries, so clients upload and download only the chunks that actually changed. In our measurements a one-file change to a 16.8 MB tar layer re-uploaded 2.1 MB (87.5% less), and a one-source-file change to a linked binary re-uploaded 80.8% less.

NativeLink implements the server side of the REAPI blob split/splice extension: SpliceBlob re-assembles chunked uploads (verifying the digest before committing anything), and SplitBlob serves chunk layouts for downloads, chunking blobs on demand with FastCDC 2020 when they were uploaded whole, which is what makes chunked downloads work for outputs produced by remote execution workers.

Requirements

• Bazel 9.1.1+ or 8.7.0+ on the client, with --experimental_remote_cache_chunking. Avoid 9.1.0: it has a client bug that corrupts outputs when the chunking flag is combined with --disk_cache (fixed in 9.1.1).
• Chunking is optional and off by default. Without the configuration below, NativeLink behaves exactly as before and does not advertise chunking support, so clients fall back to regular transfers.

Enabling it

Add an experimental_chunking block to the CAS service and give it a small store for chunk layouts. The index store must not verify content digests and must not be the CAS store itself (naming the same store for both is rejected at startup):

stores: [
  {
    name: "CAS_MAIN_STORE",
    // ... your existing CAS store ...
  },
  {
    // Blob-to-chunks layouts: roughly 80-140 bytes per chunk.
    name: "CHUNK_INDEX_STORE",
    filesystem: {
      content_path: "/tmp/nativelink/data/content_path-chunk-index",
      temp_path: "/tmp/nativelink/data/tmp_path-chunk-index",
      eviction_policy: { max_bytes: 100000000 },
    },
  },
],
servers: [
  {
    // ...
    services: {
      cas: [
        {
          cas_store: "CAS_MAIN_STORE",
          experimental_chunking: {
            index_store: "CHUNK_INDEX_STORE",
            // Optional; the REAPI-recommended default. Must be between
            // 1 KiB and 1 MiB. Clients derive min and max chunk sizes
            // from it (a quarter and four times this value).
            avg_chunk_size_bytes: 524288,
            // Optional; blobs producing more chunks than this are served
            // without chunking (~25 GiB at the default average).
            max_chunk_count: 50000,
          },
        },
      ],
      // The capabilities service advertises chunking support; clients
      // only use it when advertised.
      capabilities: [{}],
      // ...
    },
  },
],

A complete runnable example lives at chunking_cas.json5. For instances whose cas_store is a grpc proxy store, omit index_store: the chunking RPCs are forwarded to the backend, which owns the layouts.

Then build with:

bazel build //... \
  --remote_cache=grpc://your-nativelink:50051 \
  --experimental_remote_cache_chunking

When it helps, and when it doesn't

Chunking pays off when clients reach the cache across a real network (WAN, metered links, cross-region) and artifacts change incrementally: uncompressed archives, linked binaries, and container layers typically save 80 to 90% of transfer bytes per change. It does little on same-rack links (saved bytes only save time when the wire is the bottleneck) and little for compressed artifacts, where everything after the first changed byte re-transfers. Small blobs, below the maximum chunk size (4x the average), are not worth chunking and clients leave them alone, so hot small-object traffic is unaffected.

The server verifies every spliced blob's digest before committing it and materializes the full blob, so non-chunking clients and every existing read path see ordinary blobs.

CasServer::inner_splice_blob Storage grows by roughly the chunk bytes for chunk-eligible blobs; pairing the CAS with a dedup or compression store composes normally; see Compose stores.

Steps

1. Add a chunk index store: a small filesystem store of its own, not the CAS store, and not wrapped in verify.

2. Add experimental_chunking to the CAS service pointing at it, and make sure the instance has a capabilities entry so support is advertised.

3. Build with --experimental_remote_cache_chunking, change one source file, and build again.

4. Compare the re-upload bytes between the two builds. That difference is the whole point.

When it doesn't work