Skip to content

Observability & monitoring

How to watch the Dashboard BFF's own health in production: the opt-in /metrics endpoint, the signals it exposes, and an alert catalogue. This is the Dashboard's operational telemetry — distinct from the platform's time-series (which you view in Grafana via the Overview deep-link) and from the Health page (which reflects the platform's readiness). The full metric spec is telemetry.md.

Enabling /metrics

GET /metrics is opt-in and disabled by default — it returns 404 unless a scrape token is set:

  • Set KNEO_DASH_METRICS_TOKEN (≥ 32 chars; environment). With it set, /metrics serves Prometheus text exposition to a request bearing the token; a missing/wrong token gets 401 (and /metrics is 404 when the token is unset — disabled).
  • Also network-restrict /metrics at your reverse proxy — don't expose it publicly even with the token.
  • It's an instantaneous exposition — your Prometheus scrapes + stores it; Grafana graphs it. The Dashboard is not a time-series store (ADR-005).

Signals (shipped 0.6.0)

Metric Type Watch it for
kneo_dash_build_info gauge (=1, version label) which version a replica is running
kneo_dash_http_requests_total counter (method·route·status_class) request volume + error ratio (BFF-side)
kneo_dash_http_request_duration_seconds histogram BFF request latency (the queryable access-log)
kneo_dash_sse_active_streams gauge live trace-stream occupancy vs the per-operator cap
kneo_dash_audit_write_failures_total counter best-effort audit append failures (the log isn't lossless)
kneo_dash_store_reachable gauge (1/0) a trivial store read succeeds (1) or not (0)
kneo_dash_store_state_bytes gauge state-store disk footprint (SQLite main + WAL + SHM)

Counters are process-local and reset on restart (use rate()/increase(), not the raw total); on multi-replica each replica exposes its own /metrics (no built-in aggregation — your Prometheus aggregates). store_state_bytes is absent (not 0) on Postgres.

The platform-dependency histogram/counter (kneo_dash_platform_request_duration_seconds / kneo_dash_platform_requests_total{operation,outcome}) land in 0.8.0: BFF latency minus platform latency isolates BFF-side time, so a /v1-only regression points at kneo-serv/network rather than the dashboard.

State-growth metering (0.8.0): kneo_dash_sessions_active and kneo_dash_store_rows{table="audit_log"|"launches"} expose the store's growth at scrape time. When the row gauges trend up unbounded, the operator-safe kneo-dash prune CLI (export-before-delete, self-auditing — see the deployment guide) is the remediation; conservative pre-soak watch guidance: investigate around ~500k audit rows / ~50k launches on the certified single-instance topology (tuned thresholds land post-soak).

Ready-to-run monitoring assets (0.8.0)

examples/monitoring/ ships a Prometheus scrape config (15s interval, bearer-token file, plus the self-scrape the soak's continuity check reads) and alerts.yml — the enforceable form of the catalogue below with conservative pre-soak thresholds (tuning happens in those files, deployment-side). The 0.8.0 soak harness (scripts/soak/) drives + grades the beta acceptance workload against this stack.

Alert catalogue

Guidance, not committed SLOs — the Dashboard ships no product SLOs; these are signals to watch, with untuned thresholds (tune to your traffic; the soak-derived numbers land in 0.8.0). Start with:

Alert Signal Why
Audit trail degrading rate(kneo_dash_audit_write_failures_total[5m]) > 0 the best-effort local audit is dropping appends — attribution gaps (treat the window as known-incomplete)
SSE nearing capacity kneo_dash_sse_active_streams trending toward your connection ceiling operators will start getting 503 Retry-After on new trace streams
BFF errors rate(kneo_dash_http_requests_total{status_class="5xx"}[5m]) elevated BFF-side failures (a 4xx spike is usually client/auth, not the BFF)
Store unreachable kneo_dash_store_reachable == 0 the state store isn't answering — readyz will be 503, sessions/annotations/settings fail
Disk growth kneo_dash_store_state_bytes + kneo_dash_store_rows{table} growth audit + launch-history are append-only — bound them with kneo-dash prune (0.8.0) and size the volume for steady-state between prunes (data handling)
  • Operational telemetry spec — the full metric inventory + label rules.
  • Audit & health — the platform-health page + the Overview Grafana deep-link.
  • Deployment — the livez/readyz/healthz probes (distinct from /metrics).
  • Data handling — retention + the best-effort audit posture behind the failure counter.