Run lifecycle & failure semantics

The authoritative operator reference for what each run status means, how runs move between them, what happens when a step fails, times out, is cancelled, or pauses for a human — and which statuses retention prunes. The 0.9.0 cut made the state machine honest under failure; this page is the contract those fixes implement, and the surface the contract-stability policy binds to for run semantics.

Related deep-dives: human_in_the_loop.md (the pause/resume walkthrough), service_api.md (routes and error envelopes), troubleshooting.md.

Status reference

Status	Terminal?	Meaning
created	no	Run row persisted; not yet executing.
queued (queue record)	no	Waiting for a worker (async mode). The run row itself stays created/running; queued is the queue record's status.
running	no	A worker (or the synchronous caller) is executing it under a lease.
blocked	no	Paused at a human step; a WorkflowContinuation holds the resume context.
paused	no	Reserved/legacy pause marker; human pauses use blocked.
completed	yes	Finished successfully.
failed	yes	A step or the runtime raised past its error policy. Eligible for /continue (sequential workflows).
cancelled	yes	An operator cancel landed (cooperatively honored mid-run).
timed_out	yes	The run exceeded its timeout_seconds deadline.
expired	yes	A blocked human task hit its timeout with on_timeout: fail.

Terminal statuses are final: every write path fences on the canonical terminal set, so a late worker result, a resume, or a cancel can never overwrite one (409 run_state_conflict where a client asks for it).

stateDiagram-v2
    [*] --> created
    created --> running : execute / worker claim
    created --> cancelled : cancel before start
    running --> completed
    running --> failed : error past on_error policy
    running --> cancelled : cancel honored
    running --> timed_out : deadline exceeded
    running --> blocked : human step reached
    blocked --> running : resume with decision /<br/>on_timeout continue
    blocked --> expired : task timeout, on_timeout fail
    blocked --> cancelled : cancel (continuation deleted)
    blocked --> timed_out : run deadline exceeded<br/>(continuation deleted)
    failed --> running : POST /runs/{id}/continue
    completed --> [*]
    failed --> [*]
    cancelled --> [*]
    timed_out --> [*]
    expired --> [*]

Step failure: on_error semantics

Each workflow step/node declares on_error (default fail). The policy applies after the step's retry budget (max_retries) is exhausted:

• fail — the run fails; the error lands in state.error and the run_failed checkpoint (with the collected trace).
• retry — the step retries up to max_retries before fail semantics apply.
• continue — the failed step is skipped: its input passes through unchanged to the next step, the failure is recorded in the step's checkpoint metadata (and a step_failed checkpoint on graph paths), and the run proceeds. In graph workflows, edge conditions see the real outcome — a status_is: failed condition routes the failure branch.
• fallback — the referenced step/node (fallback_ref) runs in place of the failed one, under the fallback's own retry/timeout policy but never its on_error policy (fallback chains cannot recurse); traversal then continues. Note: a sequential fallback_ref is another step of the same workflow, so it also runs at its own chain position.

Cancellation and human-intervention pauses are control flow, never step failures — no on_error policy can swallow them. One unsupported corner: a fallback_ref pointing at a human step cannot pause — keep human steps in the main chain. /specs/validate rejects that shape (E_STEP_FALLBACK_HUMAN), along with a fallback_ref to a disabled graph node (E_GRAPH_NODE_FALLBACK_DISABLED).

Timeouts, cancellation, pause/resume

• Run timeout (timeout_seconds): the sweep marks the run timed_out, signals any still-executing worker to stop cooperatively, and deletes a blocked run's continuation.
• Cancel (POST /runs/{id}/cancel): cooperative mid-run; on a blocked run it also deletes the continuation (the task disappears from GET /human-tasks). Cancelling an already-terminal run is a no-op that returns the unchanged state.
• Pause/resume: a human step persists a continuation and the run goes blocked. Resume re-enters under the continuation lock and refuses any run that is no longer blocked with 409 run_state_conflict — a cancel that raced the reviewer wins. The resumed leg runs under the run's cancellation token, and the full pre-pause trace survives on GET /runs/{id}/trace.
• /continue (crash/failure recovery, sequential only): re-enters a failed run from its last checkpoint. Terminal-but-not-failed runs are refused, as are running runs whose worker lease is still live (double-execution guard); an expired lease admits the legitimate crash-recovery case.

Async run creation returns 202 Accepted

POST /v1/runs with async_mode=true returns 202 Accepted: the run is dispatched to the worker queue and the caller polls GET /runs/{id} for progress. Synchronous creates (async_mode=false) return 200. POST /v1/specs/run shares the same contract as of 0.12.0 — async_mode=true returns 202 + a queued run_id to poll, async_mode=false runs inline at 200.

0.11.0 breaking change: through 0.10.x this returned 200; the 200 → 202 move was held for the contract-stability boundary and shipped in 0.11.0 (see contract_stability.md and upgrade.md § 0.11.0). The response body shape is unchanged — only the status code. (The run row progresses created → running → terminal; queued is the queue record's status, not the run's — see the status table above.)

What retention prunes

Each knob is days-based; None/unset means "never prune" (see deployment.md for configuration):

Knob	Prunes	Notes
runs_days	Terminal runs	Default statuses: completed, failed, cancelled, timed_out, expired; override via KNEO_SERV_RETENTION_RUN_STATUSES.
checkpoints_days	Checkpoint rows	By created_at, except checkpoints of live runs (running/blocked/created/paused) — those are retained regardless of age so an aged pause can still resume.
queue_days	Finished queue records	completed/failed queue statuses.
continuations_days	Stale continuations	Continuations of still-blocked runs are protected — age alone never bricks a pending human task.
audit_days	Audit events	By created_at.
idempotency_days	Idempotency records	By updated_at (a re-upserted key refreshes its lifetime).
artifacts_days / logs_days	Files under KNEO_SERV_ARTIFACT_PATH / KNEO_SERV_LOG_PATH	By mtime.

Error taxonomy quick reference

HTTP	error	When
400	spec_invalid / invalid_request	Spec fails compile; malformed input.
403	environment_policy_blocked	The environment's stored policy blocks the deployment (diagnostics included).
404	not_found	Unknown run/continuation id.
409	run_state_conflict	Resume/continue against a run whose status forbids it.
409	resource_locked / idempotency_key_conflict	Concurrent operation holds the lock; same key, different payload.
413	payload too large	Body over the cap — including chunked uploads.
422	guardrail_violation	A guardrail blocked the content (violation type included).
422	spec_path_confined	A caller-supplied spec_path / overlay / skills[].source resolved outside the spec root (KNEO_SERV_SPEC_ROOT, or the working directory by default). Default-on as of 1.0.0.
422	(native envelope)	Request-shape validation errors.
503	queue_full	Backpressure: retry after the queue drains (Retry-After).
503	store_unavailable	The persistence backend is unreachable; transient (Retry-After).

Full envelopes and examples: service_api.md.