Metrics reference

NativeLink v1.7.7 declares 56 OpenTelemetry instruments. This page is generated from the declarations in nativelink-util/src/metrics.rs and from every .add() and .record() call site across the workspace, so the Emitted column is not a claim, it is the result of looking.

That column is the reason this page is generated rather than written. Declaring an instrument costs nothing and emits nothing; the instrument only produces a series when some code path calls it. A catalogue written by hand cannot tell those two states apart, and the gap between them is where dashboards go quietly wrong.

Every instrument at a glance

How an instrument becomes a Prometheus series

NativeLink speaks OTLP only; there is no Prometheus scrape endpoint in the binary. The names in the middle column above are what the shipped OpenTelemetry collector produces after two transformations:

1. The Prometheus exporter in otel-collector-config.yaml is configured with namespace: nativelink, which prefixes every series with nativelink_.
2. The exporter rewrites the OTLP name into Prometheus form: dots become underscores, and monotonic counters gain a _total suffix. Histograms fan out into _bucket, _sum and _count series.

cache.operations (a counter) is therefore scraped as nativelink_cache_operations_total, and cache.operation.duration (a histogram) as nativelink_cache_operation_duration_bucket and friends. Change the collector's namespace and every name on this page changes with it.

Where a name in the Prometheus series column is unmarked, it is not a prediction: it is the name that actually appears in the shipped rules, dashboards and alerts under deployment-examples/metrics/ and kubernetes/. A name marked (unreferenced) is derived from the two rules above, because no shipped artifact mentions that metric at all: nothing has ever queried it, so nothing has confirmed the exporter's spelling of it.

Declared but never emitted

Every instrument declared in metrics.rs has at least one call site.

Cache metrics

Global cache metrics instruments.

Declared as CACHE_METRICS in nativelink-util/src/metrics.rs:449, backed by the CacheMetrics struct.

Every cache instrument is emitted from one place: the cache_metrics store wrapper. A store that is not wrapped emits nothing, no matter how the collector is configured.

cache.operation.duration

Histogram of cache operation durations in milliseconds

Histogram buckets: 19 explicit boundaries, from 0.001 to 5000.0 ms. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

cache.operations

Counter of cache operations by type and result

cache.io

Counter of bytes read/written during cache operations

cache.size

Current total size of all cached data in bytes

cache.entries

Current number of entries in cache

cache.item.size

Histogram of individual cache entry sizes in bytes

Execution metrics

Global remote execution metrics instruments.

Declared as EXECUTION_METRICS in nativelink-util/src/metrics.rs:550, backed by the ExecutionMetrics struct.

Execution instruments are emitted from the scheduler process as actions change state, plus the per-action CPU time and peak memory a worker reports back when it finishes. Worker processes emit no execution metrics of their own.

execution.stage.duration

Histogram of stage durations in seconds

Histogram buckets: 15 explicit boundaries, from 0.001 to 3600.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.total.duration

Histogram of total execution durations in seconds

Histogram buckets: 13 explicit boundaries, from 0.01 to 7200.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.queue.time

Histogram of queue wait times in seconds

Histogram buckets: 12 explicit boundaries, from 0.001 to 600.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.active.count

Current number of actions in each stage

execution.completed.count

Total number of completed executions

execution.stage.transitions

Number of stage transitions

execution.cpu.time

CPU time an action used, as sampled by the worker.

Histogram buckets: 12 explicit boundaries, from 0.1 to 7200.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.peak.memory

Peak memory an action used, as sampled by the worker.

Histogram buckets: 8 explicit boundaries, from 1_048_576.0 to 68_719_476_736.0 By. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.output.size

Histogram of output sizes in bytes

Histogram buckets: 8 explicit boundaries, from 1_024.0 to 10_737_418_240.0 By. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

execution.retry.count

Counter for execution retries

INPUT_MOUNT_METRICS

Input mount counters, emitted by workers rather than schedulers.

Declared as INPUT_MOUNT_METRICS in nativelink-util/src/metrics.rs:892, backed by the InputMountMetrics struct.

worker.input_mount.cache_hits

Cached directories reused for input mount preparation.

worker.input_mount.cache_misses

Input mount directories requiring cache construction.

worker.input_mount.fallbacks

Actions using ordinary input preparation despite configured input mounts.

worker.input_mount.prepare_failures

Failed input mount preparations retried without mounts.

Worker fleet metrics

Global worker fleet metrics instruments.

Declared as WORKER_METRICS in nativelink-util/src/metrics.rs:929, backed by the WorkerMetrics struct.

Worker fleet instruments are emitted from the scheduler process as workers connect, heartbeat, pause, drain and leave. They describe the pool as this scheduler instance sees it.

worker.connected.count

Workers currently connected.

worker.connections

Workers that have joined, cumulative.

worker.disconnections

Workers that have left, cumulative, by reason.

worker.input.relinks

Input hardlinks made again after their source was evicted mid-staging, by kind.

worker.prepare.retries

Input fetches tried again on the worker after a transient store failure: retried per attempt made again, then one of recovered, exhausted, failed or abandoned.

scheduler.dispatch.requeues

Dispatches requeued untried, by reason.

worker.keepalives

Keepalives received, cumulative.

worker.keepalive.gap

Seconds between consecutive messages from a worker.

Histogram buckets: 12 explicit boundaries, from 1.0 to 60.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

worker.state.count

Connected workers currently paused or draining.

gRPC server metrics

Global gRPC serving metrics.

One duration histogram covers rate, errors and latency: the count gives rate, the status attribute separates errors, and the buckets give latency.

Declared as RPC_METRICS in nativelink-util/src/metrics.rs:1107, backed by the RpcMetrics struct.

Emitted by the OtlpLayer tower middleware on every gRPC server NativeLink starts, so a CAS, a scheduler and a worker API endpoint each report their own.

rpc.server.duration

Duration of inbound gRPC calls, by service, method and status.

Histogram buckets: 15 explicit boundaries, from 0.001 to 300.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

Scheduler metrics

Global scheduler metrics.

Declared as SCHEDULER_METRICS in nativelink-util/src/metrics.rs:1160, backed by the SchedulerOtlpMetrics struct.

Emitted from the scheduler process once per matching pass, whether or not the pass assigned anything.

scheduler.matching.duration

How long a matching pass takes. The saturation signal: this climbing while the queue is non-empty means matching is the bottleneck.

Histogram buckets: 13 explicit boundaries, from 0.0001 to 10.0 s. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

scheduler.matching.passes

Matching passes, by result.

scheduler.unsatisfiable.queued

Queued actions seen in the last matching pass that no worker connected to this scheduler could run even when idle. Per scheduler instance, judged against that instance's own workers.

scheduler.unsatisfiable.failed

Queued actions failed because no worker could ever run them, by the properties that could not be satisfied.

scheduler.queue.depth

Queued actions the last matching pass read.

scheduler.queue.age

Queued actions the last matching pass read, by age bucket.

scheduler.queue.oldest_age

Age of the oldest action the last matching pass read, in seconds.

scheduler.executing.age

Executing actions the last executing sweep read, by age bucket.

scheduler.executing.oldest_age

How long the longest-executing action the last sweep read has been executing, in seconds.

scheduler.queue.retired

Queued actions retired for having no client.

scheduler.matching.parked_dispatched

Actions a pass parked for want of room and dispatched when room opened before the pass ended, ahead of the newer actions it was reading.

scheduler.worker.holds

Workers held for a queued action that kept finding no room, by what began or ended the hold.

scheduler.worker.held

Workers held for such an action right now.

scheduler.awaited_action.orphans

Listed operations whose record was gone on read, by site.

scheduler.sweep.failures

Abandoned-queue sweeps that failed before finishing.

scheduler.hints.resolutions

Actions sized by the historical resource scheduler, by source.

STORE_METRICS

Global object-store metrics.

Declared as STORE_METRICS in nativelink-util/src/metrics.rs:1509, backed by the StoreMetrics struct.

store.object.throttles

Throttled responses, by provider and operation.

store.object.read_limit

The adaptive read limit as it stands, by provider.

store.object.read_waits

Reads that waited for a read permit, by provider.

Store tier metrics

Global tiered-store metrics.

Declared as STORE_TIER_METRICS in nativelink-util/src/metrics.rs:1571, backed by the StoreTierMetrics struct.

Emitted from the fast_slow store wrapper only. A deployment without a fast_slow store emits none of these.

store.tier.operations

Reads served per tier, by result.

store.tier.io

Bytes moved per tier and direction.

Health check metrics

Global health-check metrics.

Declared as HEALTH_METRICS in nativelink-util/src/metrics.rs:1622, backed by the HealthMetrics struct.

Emitted each time the health check endpoint runs, one observation per registered health indicator.

health.checks

Health check results, by namespace and status.

Connection pool metrics

Global connection-pool metrics.

Declared as CONNECTION_METRICS in nativelink-util/src/metrics.rs:1653, backed by the ConnectionMetrics struct.

Emitted by the gRPC connection manager and the Redis store as they hand out and re-establish connections. A deployment that uses neither emits none of these.

connection.pool.available

Free slots at the moment a connection was taken.

Histogram buckets: 11 explicit boundaries, from 0.0 to 512.0 {connection}. Anything past the last boundary lands in the +Inf bucket, so a quantile computed above it is a lower bound, not a measurement.

connection.pool.acquisitions

Acquisitions, by pool and result.

connection.reconnects

Reconnects, by pool.

RETRY_METRICS

Global metrics for operations run under a [crate::retry::Retrier].

Declared as RETRY_METRICS in nativelink-util/src/metrics.rs:1753, backed by the RetryMetrics struct.

retry.outcomes

Retry events, by client and outcome.

Attributes

Every attribute key NativeLink attaches to a metric, and the closed set of values it can take. The Prometheus label is the key with dots replaced by underscores.

Cache attributes

Execution attributes

Worker attributes

gRPC attributes

Scheduler attributes

Store tier attributes

Health attributes

Connection attributes

cache.operation.name values

cache.operation.result values

execution.stage values

execution.result values

Reading the source

If anything here disagrees with the binary, the source wins:

• nativelink-util/src/metrics.rs: every declaration
• nativelink-util/src/telemetry.rs: the OTLP exporter and its interval
• nativelink-store/src/cache_metrics_store.rs: every cache call site
• nativelink-scheduler/src/simple_scheduler_state_manager.rs: every execution call site

Where to go next

NextWire up the telemetry pipeline

The collector, the exporter interval and the queries worth alerting on: the operational half of this reference.

SidewaysConfiguration reference

Every configuration field, generated the same way from the same source of truth.