Checkpoint & state lifecycle

An operator's guide to the durable state a run accumulates — checkpoints, trace events, and human-task continuations — what writes it, how it grows, when retention prunes it, and how to read it. For what each run status means see run_lifecycle.md; for backup/restore of the whole store see backup_and_recovery.md.

What a run persists

Every run writes three kinds of durable state to the configured store (SQLite or PostgreSQL — see deployment.md):

Record	What it is	Why it exists
Run row	The RunState: status, output, error, deadline, the redacted final trace.	The authoritative run record GET /v1/runs/{id} returns.
Checkpoints	Per-step snapshots written as the run progresses, each carrying the trace delta (events since the previous checkpoint) plus a redacted state snapshot.	Resume/replay: a paused or interrupted run is rebuilt from its checkpoints; GET /v1/runs/{id}/trace reassembles the timeline from them.
Continuations	The paused-run record for a human task (the pending request + the message thread).	A blocked run resumes from its continuation when the human responds.

Checkpoints and trace events are redacted at write time — secrets and PII never land in the persisted snapshot (see observability.md and security_hardening.md).

How it accumulates

• Checkpoints are appended per workflow step / agent iteration. A long sequential workflow or a high-iteration agent writes one checkpoint per step, so checkpoint volume scales with steps × runs, not wall-clock.
• Trace events within a single run's live buffer are bounded by KNEO_SERV_TRACE_MAX_EVENTS (default 10 000; 0 disables the cap). Past the cap, further events are counted but not buffered — the full timeline still reassembles from the checkpoints via GET /v1/runs/{id}/trace, which merges checkpoint deltas with the run's events. A resumed run trims the oldest seed events from its live buffer to reserve headroom for the new leg; those trimmed events are recovered from the prior-leg checkpoints on read.
• Continuations accumulate one per blocked run (a run waiting on a human task). They persist until the run resumes, is cancelled, times out, or the human task expires.

Reading it

Endpoint	Returns
GET /v1/runs/{id}/trace	The full OTel-style trace, merged from checkpoints + the run's events (deduped by event_id).
GET /v1/runs/{id}/checkpoints	The raw checkpoint list for the run.
GET /v1/runs/{id}/checkpoints/diff	A diff view between checkpoints (replay/debug).
GET /v1/human-tasks	Continuations currently waiting on a human task.

Retention — what prunes, and what is protected

State does not grow without bound if you run the retention sweep. Retention is operator-driven (prune_retention / the maintenance pass — no built-in scheduler); each category has its own window on the RetentionPolicy:

Knob	Prunes
runs_days	Terminal run rows older than the window.
checkpoints_days	Checkpoints older than the window whose run has reached a terminal status — see the liveness guard below.
continuations_days	Continuation records older than the window.
queue_days	Completed/failed queue records.
audit_days	Audit events.

Checkpoint liveness guard (0.10.0). checkpoints_days prunes by age only for runs that are already terminal (completed / failed / cancelled / timed_out / expired) or whose run row no longer exists. The checkpoints of a live run — running, blocked, created, paused — are retained regardless of age, so a run that has been blocked on a human task for longer than checkpoints_days keeps the checkpoints it needs to resume. (Before 0.10.0, checkpoints were pruned purely by age, which could delete the resume state out from under a long-paused run.)

Expired human tasks. prune_expired_human_tasks transitions a human-blocked run to expired once its task deadline passes (or applies its on_timeout policy), and clears the stale continuation. Run it on a cadence so abandoned approvals don't pin continuations and their checkpoints forever. A run driven to a terminal status this way then becomes eligible for the age-based checkpoint prune above.

Operating guidance

• Set a retention policy. Without one, terminal runs' checkpoints and continuations persist for the life of the store. A typical on-prem policy keeps a few weeks of history; size it against your run volume × steps-per-run and your disk.
• Run the sweeps on a schedule. Wire prune_retention and prune_expired_human_tasks into cron / a sidecar (the service ships no scheduler). Pair them: expiring stale human tasks first makes their runs terminal, which then lets the checkpoint prune reclaim their state.
• Watch continuation backlog and checkpoint growth — see monitoring_and_alerting.md. A continuation count that only grows means human tasks are never being answered or expired; rising store size with a flat run rate means the retention sweep isn't running.
• Long-lived human-in-the-loop workflows keep their continuation and checkpoints for as long as they stay blocked — that is by design (the liveness guard), but it means an unbounded population of never-answered approvals is unbounded state. Give human tasks a timeout (on_timeout) so they can't pile up indefinitely.