Schedulers and workers

Who this is for: anyone adding execution to a config that currently only caches. What you'll know at the end: the minimum scheduler and worker blocks, how the two are wired to each other, and which of the many optional fields actually change behaviour you'll notice. Time: thirty minutes.

The minimum scheduler

A scheduler is one entry in the schedulers array, with a name and a type:

schedulers: [
  {
    name: "MAIN_SCHEDULER",
    simple: {
      supported_platform_properties: {
        cpu_count: "minimum",
        OSFamily: "exact",
        "container-image": "priority",
      },
    },
  },
],

simple is the scheduler you want. The other four types (grpc, cache_lookup, property_modifier, and historical_resource) either forward to another scheduler or wrap one to change its behaviour; none of them is a starting point.

supported_platform_properties is the whole contract. It declares which property keys this scheduler understands and how each is matched:

A property key an action requests that is not in this map is an error. A key that is in the map but that no worker advertises means the action never matches anything and queues silently. That failure mode gets its own page, Platform properties, because it's the first wall almost everyone hits.

The minimum worker

A worker is one entry in the workers array. local is currently the only type; it means the worker runs actions in this process, on this machine:

workers: [
  {
    local: {
      name: "WORKER_1",
      worker_api_endpoint: {
        uri: "grpc://${SCHEDULER_ENDPOINT:-127.0.0.1}:50061",
      },
      cas_fast_slow_store: "WORKER_FAST_SLOW_STORE",
      upload_action_result: {
        ac_store: "AC_MAIN_STORE",
      },
      work_directory: "/tmp/nativelink/work",
      platform_properties: {
        cpu_count: { values: ["16"] },
        OSFamily: { values: ["linux"] },
        "container-image": { values: [""] },
      },
    },
  },
],

Four of those fields are load-bearing.

worker_api_endpoint is the address of the scheduler's worker_api service, the private port from Servers and services, not the public one. The worker dials out; the scheduler never dials in. That direction is why a worker fleet can sit behind NAT with no inbound rules.

cas_fast_slow_store must name a fast_slow store whose fast half is a filesystem store. The worker builds each action's input tree by hard-linking files out of that fast store into the work directory, which requires real files on a real filesystem. The slow half must eventually resolve to the same CAS the client and scheduler use, or be a noop, when the fast tier is the only storage.

work_directory must be on the same filesystem as the fast store's content_path, for the same hard-linking reason. It is fully managed by the worker and purged on startup.

platform_properties is the worker's half of the contract. Each key is either a static list of values or a query_cmd that is run (as a command, not through a shell) each time the worker connects to the scheduler, with its output split on newlines:

platform_properties: {
  cpu_count: { query_cmd: "nproc" },
  OSFamily: { values: ["linux"] },
},

Every key the scheduler declares as minimum or exact must appear here, or this worker will never be matched for any action requesting it.

How they find each other

Nothing in the scheduler names a worker. Workers are not declared to the scheduler at all; they connect to it, announce their platform properties, and are added to the pool. Removing a worker means stopping it.

That is what the three pieces of wiring add up to:

• the execution service names the scheduler, so clients can submit actions;
• the worker_api service names the same scheduler, on a different port, so workers can join;
• each worker names the worker_api address, so it knows where to join.

In a single-process config all three are the same process and the endpoint is 127.0.0.1. In a fleet, the scheduler is its own deployment and that endpoint is a service address. The config shape does not change.

The fields that change what you'll notice

SimpleSpec has ten optional fields. Most can stay at their defaults forever. These four are the ones worth setting deliberately:

worker_timeout_s (default 5 s): how long a silent worker stays in the pool. Five seconds is aggressive for a fleet on a congested network; a worker evicted mid-action has its action re-queued.

max_action_executing_timeout_s (default 0, disabled): caps how long an action can sit in Executing with no update, even from a worker that is otherwise healthy. This is the setting that catches a worker stuck on one specific action rather than dead. Without it, such an action hangs until a client gives up.

max_job_retries (default 3): how many times an action that fails with an internal error is retried before the last error is returned to the client. It exists to stop one poisonous action from cycling through and destabilising your whole fleet.

retain_completed_for_s (default 60 s): how long a completed action's result stays available for a late WaitExecution call. Clients that disconnect and reconnect need this window to be longer than their reconnect time.

On the worker side, max_inflight_tasks (default 0, meaning unlimited) is the one to set on any real machine. Unlimited means the worker accepts every action the scheduler offers and lets the OS sort out the contention.

Putting the pieces together

Adding execution to a cache-only config takes four edits, all of which you've now seen:

1. Add a schedulers array with one simple entry, naming the platform properties you intend to match on.

2. Add execution to the public server's services, naming the CAS store and the scheduler.

3. Add a second server on a private port with worker_api naming the same scheduler, and nothing else client-facing on that port.

4. Add a workers array with one local entry pointing at that private port, plus the fast_slow store it needs.

Your first full config does exactly this, from an empty file, one block at a time.

FAQ