Performance and capacity¶

This page is about platform overhead and capacity planning — how much throughput a kneo-serv deployment sustains, where the latency goes, and which knobs move the numbers. It is not about model quality or provider latency: those dominate real run wall-time and are outside the service's control.

The guidance here is anchored to a repeatable bench harness (scripts/bench) so you can reproduce the numbers on your own hardware rather than trusting a table you can't audit. Do that before you size a production deployment — the illustrative numbers below come from a single modest machine and are meant to show shape and ratios, not to be copied into a capacity plan.

What determines run throughput¶

A run's wall-time is the sum of:

Provider/model latency — the LLM call(s). Usually 100 ms–10 s+ per step and the dominant term for real workloads. The service does not control this; tune it with KNEO_SERV_PROVIDER_TIMEOUT_SECONDS / KNEO_SERV_PROVIDER_RETRIES (see environment.md).
Platform overhead — compile-from-spec, queue dispatch, worker lease, state save, checkpoint append. This is what the service can control and what the bench harness isolates by running an echo agent with no provider I/O.
Persistence latency — the per-save cost of the state store. SQLite is a single-writer embedded file; PostgreSQL supports concurrent writers. This is the single biggest capacity lever (see below).

When you read "throughput" below it means runs/second of platform overhead — the ceiling you hit if the model were instantaneous. Real throughput is min(platform_ceiling, provider_ceiling), and for most deployments the provider is the binding constraint.

The bench harness¶

scripts/bench drives the real PlatformManager with a deterministic echo agent and reports throughput, latency percentiles, and peak RSS. Run it as a module:

# 300 runs, 8 concurrent workers, SQLite store, synchronous execute path
python -m scripts.bench --total-runs 300 --concurrency 8 --store sqlite

# Machine-readable line for sweep aggregation
python -m scripts.bench --total-runs 300 --concurrency 8 --store sqlite --json

Key options:

Option	Default	Meaning
`--total-runs`	200	Measured runs (after warmup).
`--concurrency`	8	Worker threads (sync mode) / dispatch fan-out.
`--store`	`sqlite`	`sqlite`, `memory`, or `postgres`.
`--mode`	`sync`	`sync` (thread pool of `execute_run`) or `queue` (dispatch + worker drain).
`--agent-delay`	0.0	Simulated provider latency per run (seconds).
`--postgres-dsn`	—	Required for `--store postgres`.

sync mode measures the execution + persistence path with clean per-run latency. queue mode measures end-to-end durable-queue throughput, so its per-run latency includes queue wait — useful for understanding worker-drain behaviour, not for comparing per-run cost.

To benchmark PostgreSQL (the production-representative path):

python -m scripts.bench --store postgres \
  --postgres-dsn "postgresql://kneo:kneo@localhost:5432/kneo" \
  --total-runs 300 --concurrency 8 --json

The harness is also exercised by pytest -m bench as a smoke check so it does not bit-rot; that lane asserts the harness runs and returns sane metrics, it is not a performance gate.

Reference profile and measured numbers¶

A reference baseline, not a production sizing table

Measured on a dedicated bare-metal host (2026-06-18): AMD Ryzen Threadripper PRO 3975WX (64 logical cores), 126 GiB RAM, Linux 6.17 (x86_64), CPython 3.12.3; SQLite on local disk and PostgreSQL 16.14; echo agent, zero provider delay — so these are the platform-overhead ceiling (the rate you'd hit if the model were instantaneous), not your real throughput. Single run, --total-runs 500, 50-run warmup, 256-byte payload. They show ratios and shape, not a sizing table — re-run the harness on your own hardware and store before sizing (the release procedure that automates this is bench_soak_runbook.md, and the operator how-to is dev/release_soak.md).

Store / mode	Concurrency	Throughput (runs/s)	p50	p95	p99	Peak RSS
sqlite / sync	1	51	10.7 ms	11.6 ms	12.4 ms	56 MiB
sqlite / sync	4	51	54.5 ms	71.7 ms	84.0 ms	56 MiB
sqlite / sync	8	50	105.9 ms	170.4 ms	230.3 ms	57 MiB
sqlite / sync	16	51	197.3 ms	415.9 ms	634.9 ms	58 MiB
postgres / sync	1	79	8.1 ms	8.6 ms	8.9 ms	57 MiB
postgres / sync	8	77	72.3 ms	99.6 ms	139.0 ms	58 MiB
postgres / sync	16	76	149.9 ms	220.5 ms	324.6 ms	58 MiB
postgres / sync	32	75	260.4 ms	626.5 ms	804.5 ms	57 MiB
postgres / queue	8	44	5039 ms	5830 ms	6233 ms	57 MiB

With a simulated 0.5 s provider delay (postgres / sync, 8 concurrent) throughput is 15 runs/s at p50 513 ms — the ~13 ms of platform overhead is dwarfed by model latency, which is the point: real throughput is provider-bound.

What this profile shows — and what should hold directionally on any hardware:

SQLite throughput is flat as concurrency rises (~51 runs/s from 1 → 16 here) while per-run latency scales linearly (p50 10.7 → 197 ms) — SQLite is a single writer, so concurrent runs serialize on the write lock. More workers do not buy more write throughput on SQLite; they buy queueing. This is the headline capacity fact. (The absolute SQLite ceiling is fsync/disk-bound and varies by hardware — it is lower here than on a laptop with a faster single-thread fsync — but the flat-with-concurrency shape is invariant.)
PostgreSQL holds ~75–79 runs/s flat from 1 → 32 concurrent (sync): per-run latency scales linearly (more in-flight) but throughput does not collapse, and on this host PostgreSQL out-throughputs SQLite even at concurrency 1. This is the measured evidence for the "move to PostgreSQL for write-concurrent deployments" guidance below — the concurrent-writer path scales where the single-writer one does not.
Queue mode latency is dominated by queue wait (p50 ~5 s here): the background pool drains FIFO at the default one worker per process, so the ~44 runs/s is the single-worker drain rate, not per-run cost. Add workers / processes to raise it.
Peak RSS is flat (~56–59 MiB) across every shape — memory is not the first constraint at these volumes; the write path is.

Sustained-load soak (resource stability)¶

A production-class, bare-metal sustained run on the host above (AMD Threadripper PRO 3975WX, 64 cores, 126 GiB, PostgreSQL 16.14) — 1 hour, 16 workers, multi-worker queue drain — held steady end to end (2026-06-18):

158,745 runs dispatched, 0 errors; backlog bounded (peak queued 59) and drained to zero on stop.
RSS 61.3 → 61.8 MiB (+0.8%) over the full hour (peak 61.8 MiB), inside a tightened 10% tolerance — no leak in the worker pool, the cancellation-Event map, the MCP session host, or event loops under sustained load.
Thread count bounded at 32 — the persistent idle-poll worker (0.10.0) neither spawns nor leaks threads.

Run in the same pass against the same PostgreSQL: queue-depth backpressure (load-sheds QueueFullError, terminalizes the rejected run failed{queue_full} with no phantom row, drops its cancellation Event) and the reliability test lanes — postgres_integration (CAS terminal writes, cross-process double-claim / idempotency, prune liveness), operability/durable-queue, and MCP transports — all passed. The procedure is in bench_soak_runbook.md; run it with dev/release_soak.md.

Minimum sizing (a starting point)¶

The numbers above are reference baselines, not a guarantee for your workload — but they do bound the service process's own footprint, which is the part you can size confidently:

Memory — peak RSS is flat at ~57–62 MiB across every bench shape and held at 61.3 → 61.8 MiB over the 1-hour / 158,745-run soak (no leak). Memory is not the bottleneck. Start with 512 MiB for the service container (≈8× the measured peak) and only revisit if your own profile shows otherwise.
CPU — a run's wall-clock is dominated by the model provider's network latency, not service CPU; platform overhead per run is sub-millisecond-to-low-ms (see the table above). Start with 1 vCPU for a single-team deployment; add cores/processes (not just threads) when you need write-concurrent throughput on PostgreSQL.
Workers — one process runs KNEO_SERV_WORKER_CONCURRENCY worker threads (the soak ran 16). Raise it for more in-flight runs, bounded by your provider's rate limits; for write-concurrent scale beyond one process, run multiple service processes against shared PostgreSQL (see deployment.md § Workers, scaling).
Persistence — SQLite suits a single node at low write concurrency (throughput is flat ~51 runs/s — single writer); use PostgreSQL for concurrent runs / multiple workers, and size it for your retention window and connection count. This sizing covers the service process only — budget your PostgreSQL (and any provider-side resources) separately.

This is a floor to deploy against, then measure on your own hardware with the bench harness before committing capacity (see the reference-profile warning above).

Choosing a store for capacity¶

	SQLite	PostgreSQL
Writers	Single (serialized)	Concurrent
Concurrency scaling	Throughput flat-to-down	Scales with connections/cores
Durability	File `fsync`	WAL + replication (your responsibility)
When	Single-node, low write concurrency, simplest ops	Concurrent runs, multi-worker, production

If your bench shows the SQLite write lock is your ceiling and you need more concurrent run throughput, move to PostgreSQL — that is the supported path for write-concurrent deployments. See tutorial_postgres_deployment.md for the guided setup. (The PostgreSQL store-contract + multi-connection concurrency suite already runs as a default CI lane on every PR — promoted in 0.6.0; see ga_notes.md.)

PostgreSQL sizing notes¶

Connection pooling lives at the psycopg layer; size the pool to your worker concurrency plus headroom for the API request path. Start from your bench: run --store postgres at the concurrency you intend to deploy, watch p95/p99, and provision database CPU and max_connections so the store is not the binding constraint. Replication and cross-region failover are the deployer's responsibility and an explicit non-goal.

Capacity tuning knobs¶

These environment variables move the platform-overhead and storage-growth terms. Full semantics in environment.md; the deployment-oriented subset and defaults also appear in tutorial_postgres_deployment.md § 8.

The Default column is the code default (no env set); the Tune when column carries a suggested production value where it differs.

Variable	Default	Tune when
`KNEO_SERV_PROVIDER_TIMEOUT_SECONDS`	unset (no timeout)	Provider tail latency needs bounding — `120` is a reasonable start.
`KNEO_SERV_PROVIDER_RETRIES`	`0`	Provider has a documented transient error rate — `2` is a reasonable start.
`KNEO_SERV_MAX_BODY_BYTES`	1 MiB	Larger inline specs or override payloads.
`KNEO_SERV_MAX_INPUT_CHARS`	20000	Run inputs larger than the default.
`KNEO_SERV_RETENTION_RUNS_DAYS`	unset	Cap run-history storage growth.
`KNEO_SERV_RETENTION_CHECKPOINTS_DAYS`	unset	Cap checkpoint-history storage growth.
`KNEO_SERV_RETENTION_QUEUE_DAYS`	unset	Cap queue-record storage growth.
`KNEO_SERV_CHECKPOINT_COMPRESS_BYTES`	64 KiB	Many large checkpoints; lower to compress more aggressively.
`KNEO_SERV_CHECKPOINT_MAX_BYTES`	—	Bound a single checkpoint payload.

Checkpoint payload growth¶

Branch-level checkpointing (concurrent and group-chat workflows) emits one checkpoint per step occurrence, so checkpoint volume grows with workflow breadth × rounds, not just run count. Each payload over KNEO_SERV_CHECKPOINT_COMPRESS_BYTES is compressed; KNEO_SERV_CHECKPOINT_MAX_BYTES bounds a single payload. For long-lived deployments set KNEO_SERV_RETENTION_CHECKPOINTS_DAYS so checkpoint history is pruned. The step_iteration_counts and execution_context metadata the run carries is bounded by step count and does not grow unboundedly within a run.

Worker concurrency and queue-lease tuning¶

Durable-queue runs are leased FIFO by the worker pool. By default there is one worker thread per process (KNEO_SERV_WORKER_CONCURRENCY=1), so queue throughput is a single drain rate — the queue-mode bench above shows the resulting per-run queue wait. Raise KNEO_SERV_WORKER_CONCURRENCY for provider-bound workloads: the threads overlap on provider I/O (the LLM call) while the shared store connection is held only briefly, so N workers drain roughly N× faster until the store write path saturates. For store-bound workloads on SQLite the single-writer lock caps the gain (see the table above) — move to PostgreSQL, where multiple worker processes claim safely via FOR UPDATE SKIP LOCKED, for write-concurrent horizontal scale.

KNEO_SERV_WORKER_LEASE_SECONDS (default 300) sets how long a claimed run is leased; a worker that dies mid-run has its run re-claimed once the lease expires (bounded by the KNEO_SERV_QUEUE_MAX_ATTEMPTS dead-letter cap). Set KNEO_SERV_MAX_QUEUE_DEPTH to shed load with 503 once the backlog reaches a ceiling instead of growing the queue unboundedly.

Watch the backlog via the kneo_runs_queued / kneo_runs_running / kneo_worker_count gauges on the Prometheus /metrics endpoint, and dispatch/lease latency via the kneo.queue.dispatch and kneo.worker.lease OpenTelemetry spans. Rising kneo_runs_queued with workers busy means the run path — not the API — is your constraint: add worker concurrency or move to PostgreSQL.

A capacity-planning recipe¶

Pick the store you will deploy (PostgreSQL for any concurrent-write load).
Run the bench at your intended concurrency with --agent-delay set to your provider's typical per-step latency, so the numbers reflect real run shape: python -m scripts.bench --store postgres --postgres-dsn ... --concurrency 16 --agent-delay 1.0 --json.
Read throughput and p95/p99. If the store is the ceiling, scale database resources or reduce write concurrency; if the provider is the ceiling, the platform has headroom.
Record the hardware profile alongside the numbers — a throughput figure without a profile is not reproducible and will mislead the next operator.
Set retention env vars so storage growth is bounded for the run/checkpoint volume you measured.