Monitoring & alerting¶

observability.md covers the setup — the three signals (Prometheus /metrics, structured logs, OTel spans) and how to wire them. This page is the interpretation: what to alert on, the expression to alert with, what a firing alert means, and what to do about it. It assumes the /metrics endpoint is scraped (since 0.5.0) and the JSON request logs are aggregated.

Counters on /metrics are per-process and reset on restart — always wrap them in rate(), and sum() across instances in a multi-process deployment. The metric surface is the table in observability.md § Prometheus /metrics.

Alert catalogue¶

Each row is a page-worthy signal. PromQL is illustrative — tune the thresholds and windows to your traffic and SLOs.

Queue backlog / backpressure¶

# Backlog growing and not draining (tune 50 / 10m to your throughput)
kneo_runs_queued > 50 and deriv(kneo_runs_queued[10m]) > 0

Means: runs are arriving faster than the worker pool drains them, or workers are wedged. Do: check kneo_worker_count is non-zero and kneo_runs_running is moving; scale KNEO_SERV_WORKER_CONCURRENCY or add instances; if a deployment runs with KNEO_SERV_MAX_QUEUE_DEPTH set, sustained backlog at the cap means clients are getting 503 load-shed (those runs are terminalized failed{queue_full}, not silently dropped) — alert on rate(kneo_runs_rejected_total[5m]) > 0, the direct load-shed counter.

Worker starvation¶

kneo_worker_count == 0 and kneo_runs_queued > 0

Means: queued work with no live worker to claim it. Workers poll persistently until shutdown (0.10.0), so a zero count with a backlog points at a crashed pool or a process that never called start_worker. Do: check the process is up and /readyz is green; restart drains the queue (leases are reclaimed).

Failure & dead-letter rate¶

# Failures as a fraction of completions over 5m
rate(kneo_runs_failed_total[5m])
  / clamp_min(rate(kneo_runs_completed_total[5m]), 1) > 0.1

# Any dead-lettering is worth attention
rate(kneo_runs_dead_lettered_total[5m]) > 0

Means: a rising failed-ratio is a provider outage, a bad spec, or store pressure; dead-lettering means a run exceeded KNEO_SERV_QUEUE_MAX_ATTEMPTS re-claims (it repeatedly crashed its worker). Do: read the failing runs' traces (GET /v1/runs/{id}/trace) and the JSON logs for the exception; a dead-letter usually means a poison run — fix the spec/tool, don't just raise the cap.

Latency regression¶

histogram_quantile(0.95, sum by (le) (rate(<proxy_request_duration_bucket>[5m]))) > <slo>

/metrics exports run counters and queue gauges, not request-latency histograms — read p95 from the reverse-proxy access logs or the OTel spans, or from the duration_ms field on the JSON request logs. Means: provider slowness, queue wait, or DB pressure. Do: correlate with kneo_runs_queued (queue wait) and provider latency in the spans.

Token spend¶

rate(kneo_tokens_total[1h]) > <budget_per_hour>

Means: consumption above your cost envelope. Do: the per-run hard ceiling is TokenBudgetMiddleware (a run that hits it ends failed{token_budget_exceeded}); this alert catches aggregate spend trending over budget across runs.

Readiness / dependency health¶

# From the proxy/LB health checks, not /metrics:
/readyz returns 503 for more than one probe interval

Means: a dependency probe (store, queue, configured provider secret) is failing — /readyz reports per-check detail. Do: read the checks block in the /readyz body; see incident_response.md.

Signals that aren't on `/metrics`¶

Two state-growth signals matter operationally but are read from the API / store, not the scrape endpoint (see checkpoint_and_state_lifecycle.md):

Continuation backlog — GET /v1/human-tasks (or a COUNT on the continuation store). A count that only grows means human tasks are neither answered nor expired. Give human tasks an on_timeout and run prune_expired_human_tasks.
Checkpoint / store growth — track store size or a checkpoint COUNT. Rising size with a flat run rate means the retention sweep (prune_retention) isn't running. Note the 0.10.0 liveness guard intentionally retains live (blocked/running) runs' checkpoints regardless of age, so a large blocked-run population is expected store.

Wiring¶

The service does not push its own alerts — it exposes signals. Point your Prometheus/Alertmanager at /metrics, your log pipeline at the JSON records, and your proxy's health checks at /readyz. Keep /metrics unauthenticated but not publicly reachable (it carries operational counts, no run content); bind it behind the reverse proxy per tls_and_proxy.md.