Observability

Who this is for: you are running NativeLink for other people and need to see queue times, cache hit rates and worker behaviour on a dashboard. What you'll have at the end: a collector receiving OTLP from every NativeLink process, Prometheus holding the series under the names the shipped recording rules query, and a Grafana dashboard drawing them. Time: 30 minutes.

NativeLink pushes; it is never scraped

The binary has no /metrics endpoint and no configuration that creates one. NativeLink builds three OTLP exporters at startup (logs, traces and metrics) and pushes to whatever endpoint the environment names. Every Prometheus-shaped thing in this pipeline happens downstream of the process.

That namespace: nativelink on the collector's Prometheus exporter is load-bearing, and it is the single most misunderstood part of this pipeline. The instruments NativeLink declares are named cache.operations, execution.queue.time and so on, with no nativelink prefix anywhere in the binary. The prefix is added by the collector. Bypass the collector and you lose it, along with every shipped recording rule and dashboard panel. See what happens if you skip the collector.

The configuration

Point every NativeLink process at a collector. Two variables carry the whole decision:

export NL_OTEL_ENDPOINT=http://otel-collector:4317
export RUST_LOG=info

NL_OTEL_ENDPOINT builds one client-side load-balanced gRPC channel shared by all three exporters. Leave it unset and each exporter falls back to the OpenTelemetry SDK's own resolution: OTEL_EXPORTER_OTLP_ENDPOINT (or the per-signal OTEL_EXPORTER_OTLP_METRICS_ENDPOINT and friends), defaulting to http://localhost:4317. Both work; the explicit channel is what you want in a cluster, because it re-resolves the collector's DNS as pods move. Either way the transport is OTLP over gRPC: the binary is built with only the gRPC exporter, so OTEL_EXPORTER_OTLP_PROTOCOL cannot switch it to HTTP.

The rest of the surface is small enough to state in full:

The complete list, including the variables the config file expands and the ones a worker forwards to actions, is in CLI and environment.

Bring up the pipeline

git clone https://github.com/TraceMachina/nativelink
cd nativelink/deployment-examples/metrics
docker compose up -d

export NL_OTEL_ENDPOINT=http://localhost:4317
export OTEL_METRIC_EXPORT_INTERVAL=10000
nativelink /path/to/config.json5

kubectl apply -f deployment-examples/metrics/kubernetes/prometheus.yaml
kubectl apply -f deployment-examples/metrics/kubernetes/otel-collector.yaml

env:
  - name: NL_OTEL_ENDPOINT
    value: "http://otel-collector:4317"
  - name: OTEL_METRIC_EXPORT_INTERVAL
    value: "10000"
  - name: OTEL_RESOURCE_ATTRIBUTES
    value: "deployment.environment=prod,k8s.cluster.name=main"

What happens if you skip the collector

The namespace: nativelink setting on the collector's Prometheus exporter is what produces every metric name you have ever seen in a NativeLink dashboard. Send the OTLP stream anywhere else (another collector without that setting, an OTLP-native backend, or the shipped collector's own otlphttp/prometheus pipeline into Prometheus's receiver) and the names change:

Everything shipped under deployment-examples/metrics/ (the recording rules, the Grafana dashboard, the alert expressions) is written against the first spelling. Against the second they all evaluate to an empty vector, which on a dashboard renders as a flat healthy zero rather than as a missing signal.

If you take the un-prefixed path anyway, either rewrite the rules without the prefix or add a metric_relabel_config that puts it back. The collector path is the one the repository supports.

Turn on cache metrics

Every family except cache metrics is emitted with no configuration. Cache metrics exist only for stores you explicitly wrap:

{
  name: "CAS_MAIN_STORE",
  cache_metrics: {
    cache_type: "cas",
    backend: {
      filesystem: {
        content_path: "/var/lib/nativelink/content_path-cas",
        temp_path: "/var/lib/nativelink/tmp_path-cas",
      },
    },
  },
}

With cache_metrics absent, NativeLink constructs the same store graph without a wrapper, a timer, an attribute allocation or an OpenTelemetry call on the hot path. The feature costs nothing when it is off, which is why it is off.

Wrap the stores you actually want to measure, usually CAS_MAIN_STORE and the AC. Wrapping every layer of a fast_slow composition gives you one series per layer and a hit-rate query that no longer means what you think it means.

Which metrics exist

The authority is the metrics reference, which is generated from nativelink-util/src/metrics.rs and its call sites. It records every declared instrument, its type and unit, its histogram buckets, and (the part no hand-written table stays honest about) whether anything in the binary ever emits it.

As of v1.6.5 the 30 declared instruments fall into eight families. Which process emits each one decides where you have to point the collector:

Three things worth knowing before you build a dashboard:

nativelink_cache_size and nativelink_cache_entries are declared and never emitted. The shipped recording rules and the metrics README query both. Those rules have always returned nothing.

Execution and worker fleet metrics come from the scheduler process only, and they describe the fleet as that scheduler instance sees it. With several scheduler replicas, aggregate across instances before reading a fleet total.

The shipped recording rules and dashboard predate the worker, gRPC, scheduler, store tier, health and connection pool families. Nothing under deployment-examples/metrics/ queries them yet, so the queries below are the starting point.

Queries worth having

Cache hit rate, per cache type:

sum(rate(nativelink_cache_operations_total{cache_operation_result="hit"}[5m])) by (cache_type) /
sum(rate(nativelink_cache_operations_total{cache_operation_name="read"}[5m])) by (cache_type)

How long actions wait before a worker picks them up, the number that tells you whether to add workers:

histogram_quantile(0.95,
  sum(rate(nativelink_execution_queue_time_bucket[5m])) by (le, instance_name)
)

Actions currently queued, which is the closest thing to a queue depth:

sum(nativelink_execution_active_count{execution_stage="queued"})

Success rate:

sum(rate(nativelink_execution_completed_count_total{execution_result="success"}[5m])) /
sum(rate(nativelink_execution_completed_count_total[5m]))

Workers connected to each scheduler instance, and how many of them are paused or draining:

sum(nativelink_worker_connected_count) by (service_instance_id)
sum(nativelink_worker_state_count) by (worker_state)

Why workers are leaving, which separates a scaling event from a fleet that is being evicted for timeouts:

sum(rate(nativelink_worker_disconnections_total[5m])) by (worker_disconnect_reason)

Matching pass latency, which is the scheduler's own cost per pass rather than how long actions wait:

histogram_quantile(0.95, sum(rate(nativelink_scheduler_matching_duration_bucket[5m])) by (le))

gRPC error rate per method, across every process:

sum(rate(nativelink_rpc_server_duration_count{rpc_grpc_status_code!="0"}[5m])) by (rpc_service, rpc_method) /
sum(rate(nativelink_rpc_server_duration_count[5m])) by (rpc_service, rpc_method)

Fast tier hit rate of a fast_slow store, which is what tells you whether the local disk in front of a bucket is earning its keep:

sum(rate(nativelink_store_tier_operations_total{store_tier="fast",store_result="hit"}[5m])) /
sum(rate(nativelink_store_tier_operations_total{store_tier="fast"}[5m]))

Peak memory per action at the 95th percentile, which is the number to size worker pods from (the scheduler records this sample with an empty execution_instance, so there is nothing useful to group by):

histogram_quantile(0.95, sum(rate(nativelink_execution_peak_memory_bucket[5m])) by (le))

The repository ships 34 more as recording rules in prometheus-recording-rules.yml, grouped into nativelink_execution, nativelink_cache, nativelink_performance and nativelink_slo. Loading that file gives you nativelink:execution_queue_time_p95 and friends as pre-computed series. The shipped dashboard does not query them; its panels read the raw nativelink_execution_* series directly.

The Grafana dashboard

A reference dashboard lives at nativelink-overview.json. The Compose stack provisions it automatically. Elsewhere: Dashboards → New → Import, paste the JSON, pick your Prometheus data source. It draws execution throughput, success rate, active actions by stage and stage transitions from nativelink_execution_*; it has no cache, worker or gRPC panels.

If you add panels bound to nativelink_cache_size or nativelink_cache_entries, they will be empty forever. That is the metric, not your pipeline.

Troubleshooting

FAQ

What's next