Backup, restore & recovery¶

Protecting the Kneo Agent Dashboard's own state store, and restoring it safely. The dashboard is a thin BFF — it holds no platform truth — so this covers only its state store (ADR-006). The platform (kneo-serv) backs up its own data separately.

Targets: RPO ≤ 24 h (a daily backup) · RTO ≤ 1 h (restore → recover → reconcile completes within the hour). These are the ADR-012 §7 commitments a beta operator should design around.

What's in the store — and why a restore is a security event¶

The state store holds two very different classes of data:

Class	Data	On restore
Presentation	operator annotations · saved filters · preferences · launch history	restore freely
Security-sensitive	active sessions · the Access map (role → capabilities) · connection mappings · the local audit trail	must be reconciled — see below

Restoring an old backup is not a neutral convenience — it can resurrect revoked sessions, restore a formerly-permissive Access map, undo a security/config change, and lose up to RPO of audit records. Treat a restore as a security event, never as a transparent rollback.

The golden rule: run `kneo-dash recover` after every restore¶

The dashboard ships a fail-safe recovery path (ADR-012 §7) that makes a restored store safe before it serves traffic. Never start the server directly on a restored database — run recover first. It is out-of-band (CLI, server stopped), idempotent, and interruption-safe (the marker is written first, so a crash mid-way leaves the instance in recovery, never open).

The production recipe enforces this mechanically — including automatic restore detection. docker-compose.prod.yml includes a recover-gate one-shot that runs kneo-dash recover --status on every up; the server depends_on it completing successfully, so it refuses to start the server whenever recovery is required. --status fails (and arms recovery) in two cases: (a) recovery is already active, or (b) a restore-detection sentinel finds this database is an older copy than the one last served — so even a fresh file-level restore that carries no marker is caught and blocked, not just a store you manually entered recovery on.

The sentinel is a monotonic generation stored both inside the DB and in a sidecar file (<db>.generation) next to it on the state volume. A cp state.db backup does not copy the sidecar, so a restored older DB lands with a generation behind the sidecar → the restore is detected. It fails closed: if the sidecar is missing or corrupt while the DB has already been served (a lost external anchor — e.g. a replacement-volume restore), that is treated as a restore, not a fresh install. It anchors on the store's resolved file, so it also covers the default DB (when KNEO_DASH_DB_URL is unset) and bare / .sqlite paths — not only sqlite:///… URLs. A bare pip install deployment (no recipe) gets the same sentinel: the server arms recovery at startup instead of serving a restored DB.

Scope: the supported SQLite topology. Cases it cannot distinguish (they rely on the runbook + recover):

a restore of a backup taken within the same server boot with no restart since — the generation hasn't advanced, so the copy looks current;

a whole-volume-snapshot restore that carries the sidecar back to the same older generation as the DB (a snapshot that loses or mismatches the sidecar does fail closed);

a pre-sentinel / legacy backup restored onto a fresh (replacement) volume — it has neither a generation row nor a sidecar, so it is indistinguishable from a genuine first install and enrolls as first boot (the first approved boot then enrolls it — a positive generation + sidecar — and the guarantees hold from there). After restoring a legacy backup, run recover manually.

Postgres (best-effort) has no local sidecar anchor, so it keeps the manual recover requirement throughout.

kneo-dash recover (enter):

Sets the recovery marker first — while it is set, get_client fails closed (every platform-dependent route → 503) and /api/readyz is red, so your proxy/orchestrator de-routes the instance. Nothing privileged can run.
Deletes every session — no cookie from before the restore survives.
Installs a deny-by-default Access lockdown — only admin settings.write; every platform capability is withheld. The lockdown stays in force until you reconcile — reconciliation replaces it with your validated Access map and only then clears the marker (it is not silently dropped), so the formerly-permissive map in the backup can never take effect.

Then reconcile — the only way out of recovery. You must supply the Access map you intend and make an explicit decision about the restored connections, then the marker clears in one validated step. There is no raw marker-only exit: because platform read routes aren't capability-gated, dropping the marker without reconciling would let any authenticated user reach the restored connection mappings, so the marker may only clear after both the Access map and the connections are validated (ADR-012 §7).

# keep the restored connections (you trust them):
kneo-dash recover --reconcile --access-map access.json --keep-connections

# …or replace them with a validated set:
kneo-dash recover --reconcile --access-map access.json --connections connections.json

access.json is your intended map, {role: [capabilities]}:

{
  "admin":    ["settings.write", "run.control", "launch", "policy.write",
               "annotate", "filter.write", "audit.read", "credentials.read"],
  "operator": ["run.control", "annotate", "filter.write", "credentials.read"],
  "viewer":   []
}

connections.json (for --connections) is {env: {profile: ...}}, replacing the restored set.

Everything is validated before anything is cleared (admin must retain settings.write; unknown roles/capabilities rejected; a connection decision is mandatory). If any step fails, the instance stays in recovery (exit non-zero) — fail-closed. On success the real map + connection decision are applied and the marker clears; /api/readyz goes green. --reconcile is out-of-band (no web-login dependence — it works in a DR scenario where OIDC may be down).

Connections are held behind the marker during recovery (get_client fails closed), and the restored set only becomes reachable once you clear via --reconcile — where you either keep it (--keep-connections) or replace it (--connections). Never assume the restored connections are safe without that deliberate decision.

Backing up¶

Take a backup daily (RPO ≤ 24 h). A "validated" backup is one a restore drill has actually exercised (below) — not just a file on disk.

SQLite (default)Postgres (best-effort)

Checkpoint the WAL first so the single file is self-contained, then copy it:

# DB path from KNEO_DASH_DB_URL (container default: /var/lib/kneo-dash/state.db)
sqlite3 /var/lib/kneo-dash/state.db "PRAGMA wal_checkpoint(TRUNCATE);"
cp /var/lib/kneo-dash/state.db "backup-$(date +%F).db"

Copying without the checkpoint can miss data still in the -wal file. Best done during a brief quiescent window (or snapshot the volume).

pg_dump "$KNEO_DASH_DB_URL" > "backup-$(date +%F).sql"

Postgres is best-effort (see the deployment guide's support matrix). Follow your normal Postgres backup practice; the restore + recover procedure below is the same.

Store backups off the instance and rotate them.

Restoring (the fail-safe procedure)¶

Complete within RTO ≤ 1 h:

Stop the server.
Restore the data — copy the SQLite file back into place (or psql < backup.sql for Postgres) at the path KNEO_DASH_DB_URL points to.
Enter recovery: kneo-dash recover (marker + session wipe + deny-default lockdown).
Reconcile (the only exit): kneo-dash recover --reconcile --access-map access.json --keep-connections (or --connections connections.json to replace them) — validated → applied → marker cleared. Fails closed if any check fails.
Start the server. Confirm /api/readyz → 200.

Steps 3–4 are the security reconciliation — do not skip them.

With the production Compose recipe¶

The recover-gate service is the entrypoint for these steps (same image, same state volume/DB), and it blocks a normal up while recovery is active — so the sequence is:

C="docker compose -f docker-compose.prod.yml --env-file .env"

# 1–2. stop the server + restore the DB into the dash-state volume
$C down                                   # leaves the volume intact
#     …restore your backup into the volume (e.g. via a helper container)…

# 3. enter recovery (writes the marker); mount the dir holding your JSON for step 4
$C run --rm -v "$PWD/recovery:/recovery" recover-gate recover

# `up` is now BLOCKED — the gate exits non-zero while in recovery:
$C up -d                                  # kneo-dash will NOT start (dependency failed)

# 4. reconcile (the only exit) — validated → marker cleared
$C run --rm -v "$PWD/recovery:/recovery" recover-gate \
     recover --reconcile --access-map /recovery/access.json --keep-connections

# 5. now the gate greens → the server starts
$C up -d                                  # serves; confirm https://<host>/api/readyz → 200

You can check the gate at any time with $C run --rm recover-gate recover --status (exit 0 = safe to serve, 3 = in recovery).

Rolling back a failed upgrade¶

Migrations are forward-only, applied at startup; there is no down-migration. So rollback = restore the pre-upgrade backup (which is why it inherits the RTO ≤ 1 h target — a rollback is a restore). If you start an older image against a newer schema, the migration runner fails fast with a clear "downgrade unsupported — restore a backup" error rather than risk a partial downgrade.

Procedure: stop → restore the pre-upgrade backup → run the same recover → reconcile steps → start the older image. Keep a backup taken immediately before every upgrade.

Verifying a restore (the drill)¶

A backup you have never restored is a hope, not a backup. Periodically:

Restore the backup into a throwaway instance (a scratch DB path).
Run kneo-dash recover → --reconcile against it.
Boot the server and read back — annotations/filters present, /api/readyz green, an operator can log in with the reconciled roles.
Time the whole restore → recover → reconcile and confirm it lands under RTO ≤ 1 h.

Disaster-recovery checklist¶

[ ] Daily backups run and are rotated off-instance (RPO ≤ 24 h).
[ ] A backup is taken immediately before every upgrade.
[ ] The restore → recover → reconcile runbook is documented for your environment, with your intended access.json kept somewhere retrievable out-of-band.
[ ] A restore drill has been run and timed against RTO ≤ 1 h.
[ ] Operators know that post-restore they must re-login (all sessions are wiped).

What this page does not cover¶

Platform data — kneo-serv owns its own backup/restore; this is only the dashboard's BFF state store.
Per-operator audit attribution across a restore — losing ≤ RPO of audit records on a restore can lose attribution; a documented, accepted beta tradeoff.