S3 and compatible

Who this is for: anyone who wants durability they don't operate, on S3 or something that speaks S3. What you'll have at the end: a bucket-backed CAS behind a local fast tier, with credentials that aren't in the config file. Time: thirty minutes, most of it bucket and IAM setup.

Three providers share one store type and one block of common options. aws and r2 share one implementation (R2 is the S3 store with a derived endpoint and region auto); ontap has its own, with a few differences noted below. They differ mainly in how the endpoint and credentials are derived:

GCS, Azure Blob and OCI Object Storage use the same store type with their own provider values and have their own pages.

Amazon S3

stores: [
  {
    name: "CAS_MAIN_STORE",
    fast_slow: {
      fast: {
        filesystem: {
          content_path: "/var/lib/nativelink/cas/content",
          temp_path: "/var/lib/nativelink/cas/tmp",
          eviction_policy: { max_bytes: "50gb" },
        },
      },
      slow: {
        experimental_cloud_object_store: {
          provider: "aws",
          region: "us-east-1",
          bucket: "my-nativelink-cas",
          key_prefix: "cas/",
          retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
        },
      },
    },
  },
],

region and bucket are the only fields you have to set. Credentials come from the standard AWS chain (environment variables, the shared credentials file, the EC2/ECS/EKS instance or task role), so nothing secret goes in the config.

key_prefix lets one bucket hold several logical stores. Give the CAS cas/ and the Action Cache ac/ rather than provisioning two buckets; lifecycle rules can then treat them differently.

Cloudflare R2

R2 has no regions from the client's point of view: the endpoint is derived entirely from your account ID.

{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "r2",
    account_id: "${R2_ACCOUNT_ID}",
    bucket: "nativelink-cas",
    access_key_id: "${R2_ACCESS_KEY_ID}",
    secret_access_key: "${R2_SECRET_ACCESS_KEY}",
    key_prefix: "cas/",
    retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
  },
},

access_key_id and secret_access_key are optional in the schema (omitting both falls back to the AWS credential chain), but R2 tokens don't live in that chain, so in practice you set them, from environment variables. The ${VAR} syntax is expanded at config-load time; see The config file.

R2 charges no egress, which changes the arithmetic on how small a fast tier you can get away with.

NetApp ONTAP S3

On-premises S3 needs an explicit endpoint, the storage VM serving it, and usually a private CA bundle:

{
  name: "CAS_SLOW_STORE",
  experimental_cloud_object_store: {
    provider: "ontap",
    endpoint: "https://ontap-s3-endpoint:443",
    vserver_name: "your-vserver",
    bucket: "your-bucket",
    root_certificates: "/etc/nativelink/ontap-ca.pem",
    key_prefix: "cas/",
    retry: { max_retries: 6, delay: 0.3, jitter: 0.5 },
    multipart_max_concurrent_uploads: 10,
  },
},

root_certificates is optional and points at a PEM bundle for the CA that signed the endpoint's certificate. Without it, the system roots are used and an internally-signed endpoint fails TLS verification. Credentials come from the same AWS chain as aws (AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY in practice), and vserver_name doubles as the signing region.

The ONTAP implementation differs from aws in three ways: it always uses path-style addressing, its default max_retry_buffer_per_request is 20 MB rather than 5 MB, and it is HTTPS-only with HTTP/1.1 and HTTP/2 both enabled, so insecure_allow_http and disable_http2 have no effect on it.

ONTAP has one extra store type of its own, ontap_s3_existence_cache, which keeps a persisted index of what the bucket contains and refreshes it on an interval:

{
  name: "CAS_SLOW_STORE",
  ontap_s3_existence_cache: {
    index_path: "/var/lib/nativelink/ontap-index.json",
    sync_interval_seconds: 300,
    backend: {
      endpoint: "https://ontap-s3-endpoint:443",
      vserver_name: "your-vserver",
      bucket: "your-bucket",
      key_prefix: "cas/",
    },
  },
},

It exists because FindMissingBlobs against an on-premises S3 endpoint is often the slowest thing in the build, and the answer is usually stale-tolerant.

The options all three share

Every provider flattens the same block of common options. These are the ones worth setting deliberately:

retry is exponential with jitter: up to max_retries retries after the first attempt, a base delay in seconds that is doubled before the first retry and again for each retry after it, and jitter as a fraction of the computed delay (0.5 means each wait is randomised within plus or minus 25%). The shipped examples all use { max_retries: 6, delay: 0.3, jitter: 0.5 }: waits of about 0.6, 1.2, 2.4, 4.8, 9.6 and 19.2 seconds, roughly forty seconds of retrying across seven attempts.

Objects smaller than 5 MB with a known size are uploaded in one PutObject request; anything larger, or of unknown size, goes through a multipart upload in parts of at least 5 MB, each part held in memory and retried on its own.

consider_expired_after_s is the one people are surprised by. When set, an object whose last_modified is older than this many seconds is reported as not existing, even though it is still in the bucket. Pair it with a bucket lifecycle rule whose window is longer by a comfortable margin (a few days): the rule deletes the object, and this setting stops NativeLink handing out a digest that is about to disappear. Clients that get a NotFound re-upload.

Steps

1. Create the bucket and a least-privilege credential that can head, get and put objects and create, upload to, complete and abort multipart uploads on it, nothing else. The store never lists or deletes objects (only the ontap_s3_existence_cache wrapper lists the bucket, to build its index).

2. Add a lifecycle rule that aborts incomplete multipart uploads after a few days. A process killed mid-upload leaves orphaned parts you are billed for.

3. Export the credentials into the process environment, and reference them with ${VAR} rather than pasting them into the config.

4. Add the store as the slow half of a fast_slow pair whose fast half is a local filesystem store.

5. Run a build twice: once cold, once after clearing the local fast tier.

When it doesn't work