Upgrade to a newer version

Who this is for: anyone running a pinned version who wants a newer one. What you'll have at the end: a rehearsed upgrade, a rollback that works, and your existing cache intact. Time: thirty minutes for the rehearsal, minutes for the upgrade itself.

NativeLink releases every week or two. Most releases change nothing you depend on, which makes it tempting to upgrade casually and equally tempting to never upgrade at all. Both go wrong in the same way: the thing that breaks an upgrade is almost never the code, it is the config.

Nearly every block of the config is parsed with deny_unknown_fields. A field that was renamed or removed in the new version is not ignored: it is a startup failure with a message naming the field (renamed fields sometimes keep the old name as a serde alias, in which case both spellings parse). That is the behaviour you want, because the alternative is a silently ignored setting and a deployment that is quietly not doing what the file says. But it means an upgrade is a config compatibility exercise first and a binary swap second.

What is stable and what isn't

The client-facing API is the Remote Execution API. It is a published, versioned protocol, and Bazel talking to NativeLink over :50051 and :50052 is the most stable surface here. You do not generally need to upgrade clients in step with the server.

The config schema is the surface that moves. Fields get added constantly, which is harmless; occasionally one is renamed or removed, which is not.

The worker API between a worker and the scheduler is internal. Nothing promises that a worker from one release talks to a scheduler from another, so treat scheduler and workers as a unit that upgrades together.

Stored data is content-addressed, which does most of the work for you: a blob's key is its digest, so bytes written by an old version are found by a new one. What is not automatically portable is the on-disk or in-Redis layout a store uses to organize those bytes. Layout changes are rare (the filesystem store still migrates its old single-folder layout in place at startup), and they are one of the things to look for when you read the changelog.

Read the changelog for four things

CHANGELOG.md is generated per release, grouped into features, bug fixes, documentation, testing and CI, miscellaneous, and bumps and version updates, with a compare/v1.6.4..v1.6.5 link in each release heading that shows every commit between the two.

Skimming the whole thing is not the point. Read it for:

Renamed, removed or newly-required config fields: these are the ones that stop your process from starting.

Changes to default values, which do not fail loudly and are the reason to set explicitly anything you actually care about.

Store layout or serialization changes, which determine whether existing cached data is still reachable.

Anything about the worker protocol, which decides whether you can upgrade workers and scheduler independently.

Skip several versions and you must read every intervening release's entries, not just the newest, because the breaking change may be three releases back.

Rehearse the config before you touch production

No --validate flag exists. The binary takes one positional argument, the config file (plus the usual --help and --version), and parses it at startup, which is exactly what you want, because the check is just "does the new binary accept this file":

docker run --rm \
  -v $(pwd)/cas.json5:/config.json5 \
  ghcr.io/tracemachina/nativelink:v1.6.5 /config.json5

A config error fails within a second, before any port is bound, with a message naming the offending field. If instead it starts up and begins listening, the config parsed and you can stop the container.

Run this for every config file you have (the CAS, the scheduler, and each distinct worker) against the version you are moving to. A parse failure here is a five-minute fix. The same failure during a rolling restart is an outage.

Upgrade in the right order

1. Pin, don't float. If anything in your deployment says :latest, fix that first. You cannot control an upgrade you did not choose, and you cannot roll back to a tag that means something different now.

2. Rehearse every config file against the new image as above, and fix whatever fails.

3. Upgrade the CAS servers first. They hold the data, they serve the most traffic, and they are the piece other processes depend on. A filesystem store rescans its content_path at startup to rebuild its index, so a restart with a large cache is not instant. Expect a pause proportional to the number of objects, and do not interpret it as a hang.

4. Upgrade the scheduler and the workers together. The worker API between them is internal. With a simple scheduler the queue is in memory, so restarting it drops queued actions. Clients retry, but do it when a retry storm is cheap.

5. Run a real build, then run it again. The first proves the new version works; the second proves it can still read what the old version wrote, which is the question that actually matters.

Rolling back

Rollback is the same operation in reverse, with one asymmetry worth internalizing.

Config is not backward compatible in the direction you need it to be. If you added a field that only exists in the new version, the old binary rejects that file, because deny_unknown_fields cuts both ways. Roll back the config and the image together, as a pair. Keeping the two versions of each config file next to each other for a release or two costs nothing and turns a rollback into a single decision.

Data written by the newer version stays readable by the older one in the normal case, because content addressing does not change. If the changelog mentioned a store layout change, that is the case where it does not, and it is also the case where rolling back means accepting a cold cache. Reading for that in advance is what makes the difference between a rollback and an incident.

When it doesn't work