Skip to content

Upgrade guide

Conventions for upgrading Kneo Agent Platform (kneo-serv) between releases, plus version-specific notes when a release has breaking changes.

For the release process itself (gates, tagging, artifacts), see release_checklist.md. For the supported kneo_agent SDK range, see sdk_alignment.md.

Versioning

Kneo Agent Platform follows semantic versioning:

  • Patch (0.1.0 → 0.1.1): bug fixes; persistence schemas, route contracts, CLI commands, and env-var names do not change.
  • Minor (0.1.x → 0.2.0): additive changes. Persistence schemas may add new tables or columns with migrations; routes and CLI may add new surfaces. Existing surfaces remain available with the same shape unless the release notes call out an exception.
  • Major (1.x → 2.0): may remove or change surfaces. Read the release notes before upgrading; expect to update calling code.

As of 1.0.0 (GA), the /v1 HTTP API and the kneo CLI are stable contracts under semantic versioning: additive changes within the major, deprecation windows before any removal, and no silent /v1 breaks. (Pre-1.0, behavior corrections could land in a minor under the fixes-vs-breaks policy; that latitude is retired.)

The HTTP API is also versioned at the URL prefix (/v1); legacy unversioned routes remain available alongside /v1. See design.md § 13 and contract_stability.md.

Standard upgrade procedure

  1. Read the release notes for every minor/major version between your current and target version. Patch upgrades only need the latest patch's notes. The reading-order index (newest first) is releases/README.md; the current release's notes are release_notes_1.2.0.md.
  2. Pin the target version in your dependency manifest: 
    kneo-serv[deploy]==X.Y.Z
  3. Stop traffic to the service (or drain via a load balancer). Background runs that are queued will be reclaimed by the worker after restart; in-flight runs that complete during the drain will record normally.
  4. Back up persistence. Follow backup_and_recovery.md (pg_dump for PostgreSQL, backup_sqlite_database() for SQLite). Keep the backup until you have verified the new version through at least one business cycle.
  5. Install the new version in your deployment image or environment.
  6. Restart the service. Migrations apply automatically at startup. Watch the structured log for migration events and any migration_failed errors.
  7. Verify with GET /readyz and the deployment smoke script: 
    python scripts/deployment_smoke.py --base-url http://<host>:<port>
  8. Resume traffic.

If GET /readyz does not return 200 within a few seconds of restart, see troubleshooting.md § 1.2.

Persistence migrations

Every store that has a schema (SQLiteRunStateStore, PostgresRunStateStore) tracks its schema version and applies forward-only migrations on first connection. Migrations are idempotent and never drop columns or rows on their own. The file-based stores have no schema; they tolerate older record shapes through the row decoder.

If a migration fails, the service refuses to serve requests rather than running on a partially-migrated schema. Fix the underlying cause (usually a permissions or disk-space problem), then restart.

Downgrades are not supported. Restore from backup if you need to revert.

For contributors authoring new migrations (conventions, the dialect portability rules, the test patterns), see docs/dev/migrations.md.

Spec migrations

The YAML spec format is versioned at version: v1. The compiler accepts older shapes through automatic normalization, but for clarity the CLI can write upgraded specs to disk:

kneo spec migrate legacy_agent.yaml --output migrated_agent.yaml
kneo spec migrate migrated_agent.yaml --check --json

Specs that pass kneo spec validate on the source version will continue to compile after upgrading; specs that hit deprecation warnings should be migrated proactively before a future release removes the fallback.

Signed bundles created with kneo spec bundle sign are tied to the signing key, not the kneo-serv version, so bundles signed before an upgrade continue to verify after as long as the signing key is unchanged.

SDK compatibility

kneo-serv declares a kneo-agent range in pyproject.toml. When upgrading kneo-serv, let pip resolve the matching SDK; do not pin SDK versions outside that range. The compatibility tests (tests/test_sdk_compatibility.py) assert the SDK surface used by the service, so a version mismatch surfaces as a test failure.

If you maintain custom runtimes or middlewares that import directly from kneo_agent, run those compatibility tests after upgrading and update imports in lockstep.

Configuration changes

Environment-variable names and defaults are part of the public surface. Changes are recorded in environment.md and called out in release notes:

  • New variables default to behavior consistent with the previous release.
  • Renamed variables retain a deprecation alias for at least one minor release; a startup warning is emitted when the alias is used.
  • Removed variables are removed only at major versions.

After upgrading, diff your env file against the latest deploy/production.env.example (or staging.env.example) to spot any new optional variables.

CLI changes

The kneo CLI is regenerated each release; see cli_reference.md for the current shape. New subcommands are additive within minor releases. Subcommand behavior may change at major releases — check the release notes.

CLI profiles stored at ~/.kneo_serv/profiles.json carry forward across releases. The profile schema is itself versioned and migrated in place.

Version-specific notes

This section grows as releases ship. Each entry should describe what changed, what action operators must take, and how to verify the upgrade.

0.1.0 — initial release

No upgrade applies; this is the first published version. See release_notes_0.1.0.md for scope, capabilities, and verified release-candidate steps.

0.2.0 — first public distribution

This is the first cut to publish a real kneo-serv package. 0.1.0 and 0.1.1 shipped as GitHub Release artifacts only; 0.2.0 is the first version available via pip install kneo-serv and docker pull ghcr.io/kneo-agent/kneo-serv.

Version trajectory on PyPI: 0.0.0 → 0.2.0. The kneo-serv 0.0.0 placeholder published on 2026-05-14 reserved the distribution name; it shipped an empty importable module with no kneo CLI binary (no [project.scripts] entry). Any user who tried pip install kneo-serv && kneo --version during the placeholder window saw kneo: command not found — 0.2.0 is the first cut to install the binary. The placeholder is yanked once 0.2.0 ships; existing explicit ==0.0.0 pins still resolve, but default pip install kneo-serv jumps straight to 0.2.0.

Install paths: - pip install kneo-serv — first time this works end-to-end. - docker pull ghcr.io/kneo-agent/kneo-serv:0.2.0 (and :0.2, and :latest) — first time the image is available without a local build.

Deployment migration for operators on 0.1.x using compose.yaml with the bundled build: context: .: - Default flow becomes docker compose pull && docker compose up -d against the GHCR image. - The build: block stays in compose.yaml for contributors and the CI smoke test (docker compose up --build). - No required changes to deploy/production.env or deploy/staging.env from 0.1.1.

Persistence schemas: unchanged from 0.1.1. No migrations required.

Feature additions visible to operators (full per-feature detail in release_notes_0.2.0.md): - kneo spec lint — CI-friendly validator subcommand that exits non-zero on any warnings or errors. - Retention windows now live in .kneo/config.yaml under a retention: block, with env vars as the operator override. - Human-task expiration via PlatformManager.prune_expired_human_tasks() — paused runs whose human-step deadline has passed transition to a new expired status and emit human.expired audit events. - Two new reference example specs: concurrent_review_workflow.yaml and group_chat_workflow.yaml. - Docker-based local PostgreSQL integration testing via python scripts/postgres_test.py.

No breaking changes to spec syntax, HTTP API contracts, CLI command names, env-var names, or persistence schemas. Specs that validated under 0.1.1 continue to validate under 0.2.0.

0.2.1 — /healthz version and Docker /app permission fix

Patch release fixing two regressions discovered while smoke-testing the published 0.2.0 image. Both are bug fixes; no new features, no contract changes.

Upgrade: - pip install -U kneo-serv (resolves to 0.2.1). - docker pull ghcr.io/kneo-agent/kneo-serv:0.2.1 — :0.2 and :latest now resolve to the 0.2.1 digest.

What was broken in 0.2.0: - GET /healthz returned "version":"0.1.0" from the 0.2.0 image because HealthResponse.version was a hardcoded string literal. 0.2.1 resolves the field dynamically via importlib.metadata.version("kneo-serv"). - Plain docker run -p 8000:8000 ghcr.io/kneo-agent/kneo-serv:0.2.0 crashed on startup with PermissionError: [Errno 13] Permission denied: '.kneo' because /app was root-owned but the container drops to the non-root kneo user before creating the SQLite-fallback path. 0.2.1 adds chown -R kneo:kneo /app to the install layer. The Docker Compose deployment path was unaffected (it pins KNEO_SERV_DATABASE_URL to PostgreSQL).

Persistence schemas: unchanged from 0.2.0. No migrations required.

No breaking changes to spec syntax, HTTP API contracts, CLI command names, env-var names, or persistence schemas.

0.2.2 — FastAPI info.version fix + post-0.2.0 docs sweep

Patch release fixing one regression in the same family as 0.2.1 plus a documentation sweep. No feature changes, no contract changes, no schema changes.

Upgrade: - pip install -U kneo-serv (resolves to 0.2.2). - docker pull ghcr.io/kneo-agent/kneo-serv:0.2.2 — :0.2 and :latest now resolve to the 0.2.2 digest.

What was broken in 0.2.1: - GET /openapi.json returned info.version: "0.1.0" from the 0.2.1 image because the FastAPI app constructor in kneo_serv/service/app.py still pinned a hardcoded literal. The 0.2.1 cut fixed HealthResponse.version but missed this parallel occurrence. 0.2.2 resolves both via the same importlib.metadata.version("kneo-serv") helper, called at app-construction time.

Documentation: - Forward-looking plan docs and "as of 0.1.0" framing in user/dev docs swept to match the 0.2.x shipped reality. No content lost — historical files (CHANGELOG entries, shipped release notes, the archived 0.2.0 tracker, ADRs) are unchanged.

Persistence schemas: unchanged from 0.2.1. No migrations required.

0.3.0

Next additive minor on the 0.2.x line. No breaking changes to spec syntax, HTTP API contracts, CLI command names, env-var names, or persistence schemas. Full narrative in release_notes_0.3.0.md.

Upgrade: - pip install -U kneo-serv (resolves to 0.3.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.3.0 — :0.3 and :latest now resolve to the 0.3.0 digest. The image is now signed (cosign keyless via Sigstore) and ships with a CycloneDX SBOM attestation; verification commands are in supply_chain_review.md § Verification commands.

SDK floor bump: - The kneo-agent SDK floor moves from >=1.1.1 to >=1.2.0. Pip auto-resolves on pip install -U kneo-serv, but operators pinning the SDK separately (e.g. via a constraints file or a monorepo lockfile) must ensure their install is on 1.2.0 or newer. The compat test suite passed against kneo-agent 1.2.0 throughout the 0.2.x line; the floor was kept low to avoid forcing 0.1.x users to upgrade. 0.3.0 is the natural inflection point to lift it.

New timed_out lifecycle status: - Runs that hit their run-level deadline transition to a new terminal timed_out status (alongside completed, failed, cancelled, expired). Operator tooling that switches on state.status should accept it as terminal — e.g. dashboards, alerting rules, retention sweeps (which the platform's own RetentionPolicy.run_statuses already includes). - The error.type field on a timed-out run is run_timed_out, distinct from human_task_expired (which the existing expired status uses).

New runtime surfaces: - start_run_from_spec(..., timeout_seconds=N) and run_from_spec(..., timeout_seconds=N) accept an optional wall-clock deadline. Operator-callable PlatformManager.prune_timed_out_runs() walks runs and force-cancels those past their deadline. Same operator-cron pattern as prune_retention() and prune_expired_human_tasks() — no built-in scheduler. - The human-task on_timeout: continue and on_timeout: escalate literals are now wired in the runtime (they were accepted by the spec but silently treated as fail in 0.2.x). Operators with specs that declared these literals will see the documented behaviour for the first time. Audit consumers should expect new event types: human.continued, human.continue_failed, human.escalated, run.timed_out. - New route GET /v1/runs/{run_id}/policy-report returns the spec policy report for a stored run, no spec bundle required client-side. Auth: specs:read scope (same as the existing POST /v1/specs/policy-report).

New observability surfaces: - Three new platform-side OpenTelemetry spans (kneo.queue.dispatch, kneo.worker.lease, kneo.continuation.lock) join the SDK's agent-boundary spans when KNEO_SERV_OTEL_ENABLED=true. Pre-existing OTel pipelines pick them up automatically once telemetry is enabled — no extra configuration required. See observability.md § Platform-side spans.

Persistence schemas: unchanged. The new RunState.deadline_at and Checkpoint.iteration fields default to None and 1 respectively in the dataclass, so existing rows round-trip cleanly through the JSON-payload SQLite / PostgreSQL stores.

0.4.0

Next additive minor on the 0.3.x line. No breaking changes to spec syntax, HTTP API contracts, CLI command names, env-var names, or persistence schemas. Specs that validated under 0.3.x continue to validate under 0.4.0. The cut is a docs + tooling release — runtime semantics are identical to 0.3.0. Full narrative in release_notes_0.4.0.md.

Upgrade: - pip install -U kneo-serv (resolves to 0.4.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.4.0 — :0.4 and :latest now resolve to the 0.4.0 digest. Image continues to be signed (cosign keyless via Sigstore) and ships with a CycloneDX SBOM attestation; the 0.4.0 cut adds a Trivy CVE scan report attached to the GitHub Release. Verification commands are in supply_chain_review.md § Verification commands.

SDK floor: unchanged. The kneo-agent floor stays at >=1.2.0 — same as 0.3.0. No operator action required for operators pinning the SDK separately.

New auto-generated API reference: the docs site at kneo-agent.github.io/kneo-serv/ gains a new top-level API Reference nav section with 17 pages (16 subpackages + sdk), rendered at build time by mkdocstrings from the Python docstrings. Operator surface unchanged — the API ref is a developer lookup surface, not a runtime change. See docs/api/README.md for the index.

Image vulnerability scanning (Trivy): the release pipeline now scans the pushed GHCR image with Trivy under the CVSS≥7 policy (HIGH/CRITICAL findings block the publish step). On every release-tag build, the JSON scan report is attached to the GitHub Release as the trivy-report-<version> artifact, 90-day retention. Deployers can re-run the scan locally with trivy image ghcr.io/kneo-agent/kneo-serv:<tag>; full policy + escape hatch documented in security_hardening.md § Image vulnerability scanning.

Developer-facing changes (no operator surface impact): - Ratcheting ruff D-rule gate (D100/D101/D102) now enforced project-wide for kneo_serv/**/*.py. New public classes / methods without docstrings fail CI. Forks adding code should follow the Google docstring convention; the chain-reference files are security/secrets.py and platform/manager.py. - Full mypy strict coverage across kneo_serv/. The [[tool.mypy.overrides]] block in pyproject.toml now covers every public module. Forks that subclass or extend public types should expect disallow_untyped_defs + warn_return_any + strict_equality. - mkdocstrings[python]>=0.27 added to the docs optional-dep block. Operators using pip install kneo-serv (without [docs]) are unaffected — the dep is build-time only for the rendered site.

New 0.3.0-feature worked examples: - examples.md picked up a Timeout branches subsection on the human_approval_workflow.yaml entry covering the on_timeout: fail/continue/escalate literals (all wired since 0.3.0). - New examples/run_with_timeout.py walks through start_run_from_spec(..., timeout_seconds=N) + prune_timed_out_runs(). Companion to the human-task timeout example above.

Persistence schemas: unchanged. No new fields, no migrations.

0.5.0

Next additive minor on the 0.4.x line. No breaking changes to spec syntax, HTTP API contracts, CLI command names, env-var names, or persistence schemas. Specs that validated under 0.4.x continue to validate under 0.5.0.

Upgrade: - pip install -U kneo-serv (resolves to 0.5.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.5.0 — :0.5 and :latest resolve to the 0.5.0 digest; image signing, SBOM attestation, and the Trivy scan gate are unchanged from 0.4.0.

SDK floor: unchanged. The kneo-agent floor stays at >=1.2.0.

Bug fix — checkpoint-callback metadata now survives on the final run record: in 0.4.x and earlier, execute_run and continue_run saved a stale in-memory RunState at the end of a run, overwriting the step_iteration_counts and execution_context that the checkpoint callback had written to RunState.metadata during the run. The values survived on the checkpoints but were missing from the run record. 0.5.0 re-reads the persisted run before the final save. Operator impact: if you read per-step iteration metadata off the final RunState (via GET /v1/runs/{id} metadata or the PlatformManager API) and worked around its absence, that metadata is now present. Checkpoint contents are unchanged. No migration or data backfill — the fix only affects runs executed under 0.5.0 onward.

New performance and capacity guide: performance.md documents throughput, latency, the SQLite-vs-PostgreSQL capacity trade-off, and the tuning knobs, with a reproducible bench harness (scripts/bench). No runtime change — guidance and tooling only.

Single-team on-prem operability (all additive; defaults preserve 0.4.x behaviour): - Worker concurrency. KNEO_SERV_WORKER_CONCURRENCY (default 1) runs a pool of in-process worker threads; KNEO_SERV_WORKER_LEASE_SECONDS (default 300) sets the queue lease. Default 1 reproduces the prior single-worker behaviour. Sizing guidance in performance.md. - Prometheus /metrics. New unauthenticated GET /metrics (root path only), opt-out via KNEO_SERV_METRICS_ENABLED=false. Operator action: restrict it to your monitoring network (reverse proxy or the env flag). See observability.md § Prometheus /metrics. - Overload backpressure. KNEO_SERV_MAX_QUEUE_DEPTH (default 0 = unlimited, i.e. unchanged). When set, POST /v1/runs (async) returns 503 with Retry-After: 5 once the queue is full — make async callers retry on 503. - Poison-run dead-letter. KNEO_SERV_QUEUE_MAX_ATTEMPTS (default 5) fails a run re-leased past the cap with a dead_letter error + run.dead_lettered audit event. Behaviour change: before 0.5.0 a run that repeatedly crashed its worker was re-leased indefinitely; it is now dead-lettered after 5 attempts. Set 0 to restore unbounded retries. - Graceful drain. On SIGTERM the worker pool finishes in-flight runs and stops claiming new ones, so container rollouts no longer interrupt a run.

Persistence schemas: unchanged. No new fields, no migrations.

0.6.0

Multi-process PostgreSQL hardening + uptake of the Kneo Agent SDK 2.x line. Additive persistence migrations only (a new checkpoint uniqueness index); no breaking spec-syntax, CLI, or env-var-removal changes. Two intentional observable changes (error codes, RunConfig defaults) are called out below.

Upgrade: - pip install -U kneo-serv (resolves to 0.6.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.6.0 — :0.6 and :latest resolve to the 0.6.0 digest; image signing, SBOM attestation, and the Trivy scan gate are unchanged.

SDK floor bumped to kneo-agent>=2.2.0,<3.0.0 (from >=1.2.0,<2.0.0). The service installs this for you; if you pin kneo-agent yourself, move your pin onto the 2.x line. The bump picks up SDK security fixes (MCP cross-origin redirect refusal + URL-scheme validation; broader secret redaction). Two SDK 2.x behaviour deltas a deployer may notice: - The SDK no longer auto-retries tool calls. kneo-serv's own retry knobs (KNEO_SERV_PROVIDER_RETRIES, KNEO_SERV_MCP_RETRIES, …) are unchanged and still apply; only the SDK's internal per-tool retry default flipped off. - Spec/agent with_defaults for temperature / max_iterations now actually apply — see the RunConfig change below.

Observable change — error-code remap (action may be required). The public error field in error responses is now a stable, snake_case code decoupled from internal Python class names:

Status Old error New error
404 KeyError not_found
400 ValueError / FileNotFoundError invalid_request
500 <ExceptionClassName> (+ raw message) internal_error (generic message)

queue_full, resource_locked, unauthorized, forbidden, and the idempotency codes are unchanged. If a client matched on the old class-name codes (KeyError, ValueError, …), update it to the new codes. The error envelope — {"detail": {"error": "...", "message": "..."}} — is otherwise unchanged and is now published in the OpenAPI schema as ErrorResponse / ErrorDetail. 500 responses no longer echo the exception message, and /readyz probe failures no longer leak the underlying error detail (both are logged server-side instead).

Observable change — RunConfig defaults now merge. Before 0.6.0, a run that didn't specify max_iterations / temperature had them force-set to 10 / 0.7, silently overriding a spec author's with_defaults(...). 0.6.0 leaves them unspecified so the agent/skill defaults apply (SDK 2.x merge semantics). If a spec set temperature: 0.2 (or a custom max_iterations) and you relied on the run ignoring it, the spec value now takes effect. To force a value regardless of the spec, set it explicitly on the run config. A malformed temperature (non-numeric / bool / NaN / inf) now returns 400 invalid_request instead of a 500.

New operator knobs (all additive; defaults preserve 0.5.x behaviour): - KNEO_SERV_RETENTION_AUDIT_DAYS (and project-config retention.audit_days) — prune audit events older than N days. The audit table is otherwise unbounded; set this on long-lived deployments. - KNEO_SERV_SHUTDOWN_TIMEOUT_SECONDS (default 30) — how long SIGTERM shutdown waits for in-flight runs to finish. A run still executing past the timeout is interrupted by process exit but stays claimed and is re-leased / retried (not lost); set this and your orchestrator's termination grace period ≥ your longest run step to drain without a restart. - Token-usage metrics. /metrics now exposes kneo_tokens_input_total, kneo_tokens_output_total, and kneo_tokens_total counters; usage is also on the run record and run.created audit metadata when the runtime reports it. - Idempotency in-progress. A duplicate POST arriving while the first same-key request is still in flight now returns 409 idempotency_key_in_progress (previously the two could race). Treat 409 as "retry shortly".

Persistence schemas: one additive migration — a UNIQUE(run_id, sequence) index on checkpoints (migration v3), which de-duplicates any pre-existing duplicate (run_id, sequence) rows on first start. PostgreSQL queue/lease timestamp columns are widened REAL → DOUBLE PRECISION in place (a precision fix). No data backfill or operator action required; both apply automatically on the first start under 0.6.0.

0.7.0

Finishing the 0.6.0 lease-liveness story plus an on-prem operability cluster. No breaking changes, no persistence migration, no SDK-floor change — the schema version is unchanged from 0.6.0 and the SDK floor stays kneo-agent>=2.2.0,<3.0.0. Two behaviour notes below are worth reading before you upgrade; everything else is additive and default-off / default-unset.

Upgrade: - pip install -U kneo-serv (resolves to 0.7.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.7.0 — :0.7 and :latest resolve to the 0.7.0 digest. The release pipeline now runs an in-pipeline cosign verify self-check (cosign verify + verify-attestation against the pushed digest); image signing, SBOM attestation, and the Trivy gate are otherwise unchanged.

Behaviour note — worker_lease_seconds is now a liveness window, not a run-time cap. A worker now renews its queue lease for the life of a run (a heartbeat renewing at ~worker_lease_seconds / 3), so a healthy long run never lets its lease lapse and get reclaimed mid-flight. The lease therefore no longer bounds how long a run may take — it bounds how long a crashed worker's run stays unreclaimable. If you raised worker_lease_seconds in 0.5.x/0.6.x to "fit" your longest run, you can lower it back toward your crash-detection latency. No action is required; the default is unchanged and shorter leases are now safe.

Behaviour note — the per-run token ceiling is a post-run boundary check, not a mid-flight kill. A run can be capped at a maximum input+output token budget via the SDK's TokenBudgetMiddleware, configured per agent with model.token_budget in the spec (a positive integer — a non-positive value is rejected at spec validation, not silently ignored) or deployment-wide with KNEO_SERV_TOKEN_BUDGET (the spec field wins). The middleware checks reported usage after each run and then raises TokenBudgetExceeded, surfaced as 400 token_budget_exceeded. Consequences: - A run that overshoots within a single step finishes that step before failing — size the ceiling as a spend backstop, not a precise hard stop. - on_missing="ignore": a runtime that doesn't report metadata["usage"] never spuriously fails the ceiling. Unset (the default) means no ceiling.

New operator knobs (all additive; defaults preserve 0.6.x behaviour): - KNEO_SERV_AUDIT_EXPORT_ENABLED — when set, every persisted (already redacted) audit event is also emitted as a JSON line on the dedicated kneo_serv.audit logger, from the single record_audit_event chokepoint. Attach a logging handler to forward to a file / syslog / SIEM. Off by default; export failures never break the run path. - KNEO_SERV_TOKEN_BUDGET — deployment-wide per-run token ceiling (see the behaviour note above). A spec's model.token_budget overrides it. - Local / self-hosted LLM endpoints. The native (openai) runtime now reads model.extra.base_url and an API key — model.extra.api_key_ref resolved through the SecretResolver, or a literal api_key escape hatch — and threads them into the OpenAI-compatible client, so a spec can target Ollama / vLLM / llama.cpp / LocalAI. Unset fields preserve the hosted-OpenAI default; a literal api_key is redacted from audit / list surfaces.

New spec fields (optional, additive — old specs are unaffected): - Human-request taxonomy. components.humans.* accepts request_type (approval / review / correction / selection / freeform), options, default_option, context, and response_role. validate_semantics rejects a default_option outside options and a selection without options. When response_role is set, the reviewer's reply folds into the resumed run's message thread with that role. GET /v1/human-tasks/{id} now also returns the paused run's redacted messages thread alongside the pending request (same auth scope; no new route). A client that only reads the existing request field is unaffected.

Persistence schemas: no migration. The store schema version is unchanged from 0.6.0; RunStateStore gains a schema_version / close Protocol surface (behaviourally a no-op on the schema-less stores), but no on-disk change applies on upgrade.

0.9.0

Reliability & retention. No breaking changes, no persistence migration, no SDK-floor change — the spec schema version is unchanged and the SDK floor stays kneo-agent>=2.2.0,<3.0.0. Persistence additions (the idempotency prune, count queries) are additive-only; rollback to 0.8.0 is safe (no persisted-field removals).

Behavior corrections to review before upgrading. Each corrects shipped behavior that contradicted its own documented/validated contract (the fixes-vs-breaks test in the new contract-stability policy, adopted this cut). If you built automation against the old behavior, adjust:

  • Handoff round_robin runs report completed after a full rotation (previously every successful rotation persisted as failed / max_iterations). Alerts keyed on that false failure will go quiet.
  • on_error: continue / fallback execute. Workflows that declared error tolerance but relied on the hard failure will now proceed: continue passes the step's input through; fallback runs the referenced step. See the run lifecycle guide.
  • List total is the true store count — it previously capped silently at the 10 000-row fetch window. Dashboards asserting total ≤ 10000 should read the pagination block. Run list items now carry trace_event_count instead of each run's full trace_events array (the trace lives at GET /runs/{id}/trace).
  • Token-usage metrics survive redaction (input_tokens etc. were [REDACTED] everywhere). Cost dashboards start receiving real values.
  • Resume/continue are fenced: resuming a run that is not blocked, or continuing a terminal-but-not-failed / live-leased run, returns 409 run_state_conflict instead of silently re-executing. Cancelling a blocked run removes its task from GET /human-tasks.
  • A per-attempt timeout no longer retries by default — the abandoned attempt may still be running, so the retry double-executed non-idempotent calls. This applies to provider calls AND to workflow steps/nodes that set timeout_seconds + max_retries. Opt back in per surface: KNEO_SERV_PROVIDER_RETRY_ON_TIMEOUT=true (or retry_on_timeout in spec retry config) for providers, KNEO_SERV_WORKFLOW_RETRY_ON_TIMEOUT=true for workflow steps/nodes. MCP connect timeouts are the exception: they cancel the connect coroutine cleanly, so configured MCP retries do retry them.
  • Stricter validation (pure checks; previously these crashed at runtime): graph kind: human nodes (E_GRAPH_NODE_HUMAN_UNSUPPORTED), memory blocks without policy (E_MEMORY_POLICY_REQUIRED), guardrail items missing id/type (E_GUARDRAIL_FIELDS). Invalid path-based specs now return 200 {valid: false} from /specs/validate (was 400).
  • Environment policies set via REST are enforced on run/compile when the request names the environment — a deployment blocked by policy returns 403 environment_policy_blocked. Verify your stored policies say what you mean before upgrading production.
  • Stricter env-var parsing: invalid numeric values in KNEO_SERV_WORKER_* / queue knobs now fail startup instead of silently running defaults; same for the new strictly-parsed knobs.

New knobs (all optional; see environment.md): KNEO_SERV_RETENTION_IDEMPOTENCY_DAYS, KNEO_SERV_RETENTION_RUN_STATUSES, KNEO_SERV_TRACE_MAX_EVENTS, KNEO_SERV_MCP_CONNECT_TIMEOUT_SECONDS, KNEO_SERV_IDEMPOTENCY_LOCK_TTL_SECONDS, KNEO_SERV_PROVIDER_RETRY_ON_TIMEOUT, KNEO_SERV_WORKFLOW_RETRY_ON_TIMEOUT, KNEO_SERV_ARTIFACT_PATH / KNEO_SERV_LOG_PATH.

0.8.0

Declarative spec parity along the tools / MCP / skills axis. No breaking changes, no persistence migration, no SDK-floor change — the schema version is unchanged and the SDK floor stays kneo-agent>=2.2.0,<3.0.0. One behaviour note below is worth reading before you upgrade; everything else is additive and default-unset.

Upgrade: - pip install -U kneo-serv (resolves to 0.8.0). - docker pull ghcr.io/kneo-agent/kneo-serv:0.8.0 — :0.8 and :latest resolve to the 0.8.0 digest. Image signing, SBOM attestation, and the Trivy gate are unchanged from 0.7.0.

Behaviour note — overlays is no longer silently ignored. POST /v1/runs (sync + async) and the /v1/specs/run / /compile / /validate / /policy-report routes accepted an overlays list but dropped it without applying it. From 0.8.0 the overlays are threaded through compile/run, persisted in run metadata, and replayed on resume. If any stored client request or automation passes overlays, audit it before upgrading — those overlays now actually change the compiled spec. overrides / strict are likewise now honored on the /specs/* routes that previously dropped them.

Trust note — spec_path and overlays are filesystem-trusted inputs. Both name paths the server reads at compile time. Grant runs:write / specs:read-scoped keys to callers you trust with that read surface, and see security_hardening.md for the posture before exposing these fields to semi-trusted callers.

New spec surface (all optional, additive — old specs are unaffected):

  • Declarative MCP transports. A top-level mcp_servers block (transport: stdio | http | sse, with command/args/env/cwd or url/sse_url/message_url/headers/timeout, plus max_response_bytes / sse_read_timeout knobs and verify / ca_bundle / client_cert / client_key TLS fields) and a tool.mcp = {server: <name>, name?: <remote_tool>} reference. Construction happens at build time; the connection is lazy on first tool call, so the spec compiles offline. Prefer client_key_ref (resolved via the SecretResolver) over inline client_key — the inline spec is persisted unredacted into run metadata, and the TLS field names are redaction terms only on audit/list surfaces. verify: false draws a validation warning.
  • Agent-as-tool. tool.agent: <name> backs a tool with another declared agent. A tool must be backed by exactly one of implementation / mcp / agent — a tool with none is now a validation error (E_TOOL_NO_BACKING) instead of being silently dropped at build.
  • Workflow-as-agent. agent.as_agent: <workflow> backs an agent with a declared workflow; only name / description / system_prompt are legal alongside it. Cyclic or dangling references across all of these fail at /specs/validate (E_BUILD_CYCLE etc.), not at runtime.

New API surface (additive):

  • GET /v1/skills — read-only catalog of declared + default discoverable skills; specs:read scope, standard pagination, no side effects.
  • RunCreateRequest.skills — per-request {add, disable} skills overlay. add only enables skills already declared in the spec; out-of-scope overlays are rejected; every overlay is audited (run.skills_overlay) and preserved across resume.
  • GET /v1/human-tasks?status=pending|escalated — a real filter now; an unknown value returns 422 where it was previously a silent no-op.
  • POST /runs/{id}/continue accepts an Idempotency-Key and replays the stored response on retry; concurrent /continue calls are serialized under a per-run lock. POST /v1/specs/run now holds the same idempotency lock as /runs (409 idempotency_key_in_progress on contention).
  • Invalid specs on sync POST /v1/runs return 400 with diagnostics where they previously surfaced as an opaque 500. 413 (payload_too_large) is now published in the OpenAPI error responses.

Persistence schemas: no migration. No new persisted fields; rollback to 0.7.x after running 0.8.0 is persistence-safe (the new request/spec fields are request-scoped or compile-scoped only).

0.10.0

Theme: performance & capacity / 1.0 runway. A correctness/security/ hardening cut. Additive-only; no migration; SDK floor held at kneo-agent>=2.2.0,<3.0.0. 0.10.0 is a normal additive minor — not the 1.0 cut.

Intentional behavior changes (each corrects provably-wrong shipped behavior per the contract-stability policy; act if you keyed on the old behavior):

  • tool-stage redact/warn guardrails now actually enforce. Before 0.10.0 a declared tool-stage guardrail validated, satisfied the production require_guardrails gate, and deployed — but was never wired into the runtime, so it never ran. redact/warn now execute in the tool-call chain. Action: if a deployment declared a tool-stage redact/warn guardrail, it was unprotected until now (disclosed on fix) — re-review it and confirm the now-live behavior is what you want (e.g. a redact action will now actually redact tool output).
  • Raising tool-stage guardrail actions are now rejected at /specs/validate (E_GUARDRAIL_ACTION_UNSUPPORTED). A tool-stage guardrail with the default block (or escalate/human_review/retry/revise) cannot abort the run yet — the SDK bridge executor's per-tool-failure contract converts the raised violation into a recoverable result, so it would fail open. Rather than ship that, such specs now fail validation. Action: for tool-stage guardrails use redact/warn, or move a blocking check to the input/output stage (those enforce block correctly). True tool-stage block-enforcement is planned for 0.11.0. Note: a tool-stage guardrail with no explicit action defaults to block, so add action: redact (or warn) to such specs.
  • Guardrails with a non-middleware mode are now rejected at /specs/validate (E_GUARDRAIL_MODE_UNSUPPORTED). Only the middleware attachment is wired; other modes (runtime/tool/workflow) were silently dropped. Action: remove mode (it defaults to middleware) or set it to middleware.
  • workflow-stage guardrails are now rejected at /specs/validate (E_GUARDRAIL_STAGE_UNSUPPORTED). No runtime hook enforces them yet, so a spec declaring one previously validated green and silently did nothing. Action: remove workflow-stage guardrail blocks (or move the control to a tool/input/output stage); such specs will now fail validation.
  • A kind: workflow step containing a human-approval step is rejected at /specs/validate (E_STEP_WORKFLOW_NESTED_HUMAN). Such specs used to validate and then complete the run with the unapproved output. Action: lift the human-approval step to the top-level workflow (the supported pattern — it blocks and resumes correctly).
  • Secret redaction now covers pluralized credential keys (api_keys, refresh_tokens, KNEO_SERV_API_KEYS). Single-segment usage counters (input_tokens, max_tokens) are unaffected. Action: none expected; if you scraped a redacted log/trace/audit field expecting a plural credential key to appear in the clear, it no longer will.
  • Release packaging: the container image's public tags (:X.Y.Z / :X.Y / :latest) and the GitHub Release are now gated behind the Trivy CVE scan and the coverage/postgres lanes (the release → scan → gated ship split). No operator action; relevant only if you build the image from this repo's workflow.

Persistence schemas: no migration. Terminal-write atomicity, the persistent idle-poll worker, and checkpoint-prune liveness are behavior-internal; no new persisted fields. Rollback to 0.9.x is persistence-safe.

0.11.0

0.11.0 is a breaking, 1.0-runway cut: it ships the two held 1.0-register /v1 contract changes, plus guardrail-enforcement that turns some previously-rejected specs into accepted-and-enforced ones.

Breaking — /v1 contract:

  • Async run-create returns 202 Accepted (was 200). POST /runs / POST /v1/runs with async_mode=true now returns 202; synchronous creates (async_mode=false) still return 200. The response body is unchanged. Action: if your client asserts status_code == 200 on async create, accept 202 (or treat 2xx as success); keep polling GET /runs/{run_id} exactly as before. Idempotent replays preserve the 202.
  • Unknown query parameters are rejected with 422. Any query-string parameter a route does not declare now returns 422 {"error": "unknown_query_parameters", "unknown": [...]} on the authenticated /v1 (and root) surface; through 0.10.x they were silently ignored. /healthz, /readyz, /metrics are exempt. Action: remove stray/misspelled query params from API calls; a typo that was previously a silent no-op now errors (which is the point — it surfaces the bug). Request bodies already rejected unknown fields, so this only changes query strings.

Behavior — guardrail enforcement (specs rejected at 0.10.0 now validate):

  • Tool-stage guardrails with a raising action (block/escalate/etc.) are now enforced — a violation aborts the run (sync → 422, async → failed) instead of failing open. E_GUARDRAIL_ACTION_UNSUPPORTED is no longer raised at /specs/validate.
  • workflow-stage guardrails are now accepted and enforced per step (each step's output is checked; block aborts, redact/revise rewrite). E_GUARDRAIL_STAGE_UNSUPPORTED is no longer raised. Action: if you relied on these being rejected as a lint, note they now run — audit any tool/workflow-stage guardrail blocks you had declared "for later."
  • Guardrails now also apply to streaming runs (Agent.stream): input guardrails run before the stream; an output revise buffers and rewrites the caller-received text (so a stream with an output guardrail yields the revised result as one chunk rather than token-by-token).

Persistence schemas: no migration — all changes are API-surface or behavior-internal; no new persisted fields. Rollback to 0.10.x is persistence-safe (but clients depending on the new 202/422 contract must roll back too).

Downstream: kneo_client (and anything pinning the /v1 contract) needs a coordinated uptake for the 202 + reject-unknown-query-params changes — see its TODO-0.8.0.

0.12.0

0.12.0 is an additive, production-ready minor (the GA candidate). No breaking /v1 change ships in this cut; the one deliberate break (spec-path confinement default-on) is staged here as a deprecation warning and lands at 1.0.0.

Behavior change — POST /specs/run honors async_mode:

  • Through 0.11.x, POST /specs/run silently ignored async_mode and always ran the spec inline, returning 200. It now mirrors POST /runs: with async_mode=true it dispatches to the worker queue and returns 202 Accepted with the queued run_id (poll GET /runs/{run_id}); the synchronous default (async_mode=false) still returns 200; idempotent replay preserves the original status. Action: a client that sent async_mode=true to /specs/run and relied on getting a completed run back at 200 will now get 202 + a queued id — switch to polling (as /runs callers already do). Clients that only used the synchronous default are unaffected.

Deprecation (becomes a default-on break at 1.0.0) — spec-path confinement:

  • spec_path and overlays are caller-supplied filesystem paths the service reads at compile time. 0.12.0 adds an opt-in KNEO_SERV_SPEC_ROOT env var: set it to an allow-listed root and any path resolving outside it (absolute, ..-traversal, symlink escape) is rejected 422 spec_path_confined. While KNEO_SERV_SPEC_ROOT is unset, behavior is unchanged except that an absolute path now logs a DeprecationWarning. At 1.0.0 confinement becomes default-on and absolute / out-of-root paths are rejected by default. Action: set KNEO_SERV_SPEC_ROOT to the directory that holds your specs now — this both closes the path-disclosure surface today and adopts the GA behavior ahead of the 1.0.0 flip. (Held 1.0-register change; see ../dev/contract_stability.md.)

Also in this cut (no action needed): human-approval (kind=human) gates now pause + resume in every workflow shape (graph, handoff, group-chat, concurrent — previously sequential only); the kneo spec explain CLI command; an enforced seeded backup/restore release gate; and internal correctness/security fixes (overlay path-confinement, a tool-policy fail-open close, file-store retention-race hardening). See the CHANGELOG.

Persistence schemas: no migration — additive only; rollback to 0.11.x is persistence-safe. (A run blocked inside a graph/orchestration workflow on 0.12.0 cannot be resumed after a rollback to 0.11.x, which lacks the continuation support — drain in-flight blocked runs before rolling back.)

1.0.0

BREAKING — spec-path confinement is default-on. Through 0.12.x, KNEO_SERV_SPEC_ROOT was opt-in: with it unset, an absolute or out-of-root spec_path/overlays was accepted and only logged a DeprecationWarning. At 1.0.0 the default flips to reject: a caller-supplied path that resolves outside the confinement root is refused with 422 spec_path_confined. When KNEO_SERV_SPEC_ROOT is unset, the confinement root is the process working directory.

This break now also covers skills[].source — a declared skill bundle's filesystem path, which through 0.12.x was read unconfined (the sibling path that bypassed the spec_path/overlays confinement and left an authenticated arbitrary-file-read oracle open). At 1.0.0 an out-of-root skill source is rejected like any other spec read, and a ../~ traversal in a skill source is rejected at spec validation.

Action. Pick one:

  • Set KNEO_SERV_SPEC_ROOT to the directory that holds your specs, overlays, and skill bundles (the recommended posture — an explicit allow-listed root). Everything you load by spec_path / overlays / skills[].source must resolve inside it.
  • Or keep specs under the service's working directory and leave KNEO_SERV_SPEC_ROOT unset (the working directory is the default root).

If you deploy with out-of-tree spec or skill paths (e.g. absolute paths to a shared bundle directory), move them under the root or add the root to KNEO_SERV_SPEC_ROOT before upgrading — otherwise those requests begin returning 422 spec_path_confined. Inline specs (spec in the request body) and per-run skill overlays are unaffected; only caller-supplied filesystem paths are confined.

Local CLI is operator-trusted. Spec-path confinement applies to the service's reads of caller-supplied paths (the /v1 surface, run, resume). The local kneo CLI reads the operator's own filesystem directly and is not confined to KNEO_SERV_SPEC_ROOT — a local operator already owns the filesystem, so kneo spec validate /any/path.yaml keeps working from any directory. (When the CLI targets a remote service with --service-url, it sends the resolved spec inline; the service applies its own confinement.)

kneo spec validate now exits 1 on an invalid spec (was exit 0, with the diagnostics only in the body). It now works as a CI gate, consistent with kneo spec lint; --json still prints valid + diagnostics. Update any pipeline that relied on exit 0 for an invalid spec.

Persistence schemas: no migration — these are request-validation + CLI changes only.

1.1.0

A normal additive minor on the 1.x line — no /v1 contract break. It does tighten spec validation under the reject-don't-drop policy: controls that were previously accepted and then silently dropped, ignored, or crashed at runtime now fail fast at /v1/specs/validate (and at run-create). All are validation behavior, not /v1 shape changes; a spec that was already correct is unaffected.

GET /v1/runs/{id}/policy-report now requires the runs:read scope (was specs:read), aligning it with every other per-run read (get / recovery / replay / trace / checkpoints / graph). The operator, service, and viewer roles already carry both scopes and are unaffected; the reviewer role (runs:read, no specs:read) gains access. Action: a custom API key that read this endpoint with only specs:read must add runs:read.

Unknown spec keys are now rejected (extra='forbid'). A typo'd or stray key anywhere in a spec block (tols:, systme_prompt:, an unknown sub-block) was silently discarded — so an agent could ship missing the tool/prompt/governance the author intended. Every v1 spec block now rejects unknown keys with an E_SCHEMA validation error. Blocks that are intentionally open carry a dedicated field for loose values — put provider-specific inference params under model.extra, not as unknown keys on model:. Action: run kneo spec validate (or kneo spec lint) over your specs before upgrading; fix or remove any flagged stray keys. A common one: a runtime selector must use runtime_preferences.preferred_mode (+ allowed_modes), not a bare runtime: key (which never took effect and is now rejected).

Guardrail action is validated against its stage. An action the stage's runtime does not honor now fails validation (E_GUARDRAIL_ACTION_UNSUPPORTED) instead of crashing (HTTP 500) or silently degrading at runtime. Specifically: redact is rejected on the input/output stages (it is a tool/workflow action), and revise is rejected on input (it only applies on output/tool/ workflow). Supported per stage: input — warn/block/retry/escalate/human_review; output — those + revise; tool/workflow — those + redact. action still defaults to block. Action: for output PII handling use block (fail-closed) or revise; redaction belongs on the tool/workflow stage.

Participant fallback is rejected. A concurrent / handoff / group-chat participant declaring on_error: fallback or fallback_ref now fails validation (E_PARTICIPANT_FALLBACK_UNSUPPORTED) — orchestration never honored participant fallback, so it was a silent no-op. Action: remove the participant fallback, or use a sequential / graph workflow where fallback is enforced.

Tool domain gating is now enforced; binary capability flags are advisory. tools.permissions.allowed_domains (and the newly-wired denied_domains) are now enforced at tool-call time: a URL-shaped argument whose host is in denied_domains — or, when allowed_domains is set, is not in it — fails the call (422 sync / failed async). Action: if a tool legitimately calls a host, add it to allowed_domains before upgrading. The binary capability flags (allow_network / allow_filesystem_read|write / allow_shell) remain a declared static-governance posture — they drive the policy report and the deny_unrestricted_tools env-policy gate, but are not runtime-enforced against arbitrary in-process tool code (a real sandbox is a future 2.0 item; see capability_enforcement_design.md). Treat them as governance signals, not a runtime sandbox.

Run spec is frozen at create. A run now captures its resolved base + overlay spec inputs at creation and compiles every execute/resume from that snapshot. Editing, moving, or deleting the source spec/overlay files after a run is created — or restarting the service against a changed tree — no longer changes or breaks that run (it ran against a possibly-mutated file before). The reload-on-resume behavior is gone: editing a spec file and resuming will not apply the edit to an existing run; start a new run instead. Runs created before 1.1.0 keep the old recompile-from-source behavior (no snapshot). The snapshot now also captures the declared skill bundles' content (instructions + metadata) at create, so a run's skill prompts are durable too — editing, moving, or deleting a skill bundle after a run is created no longer changes that run. Remaining limit: tool implementation code (Python import paths) and a skill bundle's tool callables are code, not data, and are still resolved at build — a tool whose code changes under a restart can still change behavior (the package-version pin / fingerprint mitigations remain the deferred audit/drift work).

Resuming an expired human task is now refused. POST /v1/human-tasks/{id}/resume returns 409 human_task_expired when the task's deadline (expires_at) has already passed — previously a resume landing in the window between expiry and the next prune_expired_human_tasks sweep could complete a task the deadline said was over, and whether the resume or the sweep won was a race. The deadline is now authoritative: the configured on_timeout policy (fail / continue) governs an expired task. Exception: on_timeout: escalate tasks remain resumable past their deadline — escalation deliberately keeps the run blocked for a manual late resume. Action: none for well-behaved reviewers; clients that resumed past the deadline must handle the 409 (the task timed out — inspect the run's terminal state).

Persistence schemas: a single automatic, backward-compatible SQLite/ PostgreSQL migration (v4) adds a nullable runs.session_id column (W9 #6) plus filter indexes; it applies on first connect with no operator action and leaves existing rows intact. The remaining changes are additive run-metadata + validation/gating behavior.

1.2.0

A normal additive minor on the 1.x line — no /v1 contract break. Theme: sibling-parity & contract-fidelity hardening. It does tighten a few validate-/config-/runtime behaviors under the reject-don't-drop policy; controls previously accepted-then-dropped, silently-ignored, or a prod no-op now fail fast or take effect. A deployment that was already correctly configured is unaffected.

allow_filesystem_read now requires human approval like allow_filesystem_write. An agent whose tool policy sets allow_filesystem_read is now flagged as a privileged surface (W_TOOL_FILESYSTEM_READ_ALLOWED) and, with no human-review step, trips W_HUMAN_APPROVAL_MISSING — matching the write flag and the policy report, which already counted read-any-file as unrestricted. Action: add a human approval step to such specs (or document the exemption), as you already do for write-capable agents.

A mistyped role/scope — or a stray : — in KNEO_SERV_API_KEYS now fails at startup. A token that is neither a known role nor a scope some role grants (e.g. reviewr, runs:reed, or a : inside the key value, which is the field delimiter) previously parsed as an inert explicit scope, silently giving the key fewer privileges than intended. It now raises a RuntimeError at config parse. Action: if a key fails to load, fix the typo; keys themselves must not contain a :.

Retention status overrides must be terminal. KNEO_SERV_RETENTION_RUN_STATUSES / _QUEUE_STATUSES (and programmatic RetentionPolicy) now reject any non-terminal status — a value like running would have let a prune pass delete live in-flight records. Action: set only terminal statuses (completed/failed/cancelled/timed_out/expired for runs; completed/failed for the queue).

redact_tool_results is now enforced. The tools.permissions.redact_tool_results policy flag was only honored by an unwired middleware — a no-op in production. It now runs on the live tool path and scrubs secrets from tool output via redact_data (per its docstring), not a blanket marker. Action: none required; if you set this flag expecting it to take effect, it now does (secret values in tool results are scrubbed).

Unknown query parameters: 403 now precedes 422. An authenticated but under-scoped caller sending an unrecognized query param now gets 403 (missing scope) before the 422 (unknown param), so the error no longer discloses which params a route accepts to a caller not authorized for it. Action: none.

New additive surface (no action; opt-in): GET /v1/runs?q= bounded content search; a first-class usage field on the run status; a typed GET /v1/security/credentials inventory with a health status; POST /v1/policies/environment/{env}/preview; and dropped/complete on GET /v1/runs/{id}/trace.

SDK floor: unchanged (kneo-agent>=2.2.0,<3.0.0). 1.2.0 consumes no new SDK surface.

Rolling back

Schema-forward migrations make in-place downgrade unsafe; the only supported rollback path is restore from the pre-upgrade backup, then re-install the previous version.

For the full step-by-step procedure — stop, restore, re-install, restart, verify with the deployment smoke — see backup_and_recovery.md § Rolling back after a failed upgrade.

Keep the pre-upgrade backup until you have verified the new version through at least one business cycle.

Reporting upgrade issues

Capture the same context listed in troubleshooting.md § What to capture before opening a bug, plus:

  • Source version (pip show kneo-serv before the upgrade).
  • Target version (after the upgrade).
  • Migration log lines from the first start on the new version.
  • The exact env file or compose .env (with secrets redacted).