Agent outage — 12 September 2026

Impact and timeline

Production .24 stopped completing Clapilot chat requests. WhatsApp stopped replying and its settings status remained on “loading”. Telegram and Signal polling also stalled. The app and agent containers remained running, so container uptime did not reveal the outage.

All times below are UTC (add two hours for Europe/Berlin):

• 11:17:26: affected agent process started with image revision 6c8180a3c23ca207f1d2a711cde920b6b4a5549b.
• 12:05:26: latest completed chat observed before the blockage. A chat created at 12:05:43 remained running. This bounds the observed onset; the exact first failed inbound message is not known.
• Around 14:00: diagnosis found all ten agent database connections idle in PostgreSQL for almost two hours, with the same context-graph SELECT as their last query. /metrics returned HTTP 200 in 24 ms, while both /health and WhatsApp status exceeded a 12-second diagnostic deadline.
• Fresh database connections succeeded. DNS resolved PostgreSQL, the app, WhatsApp, and Telegram. This incident was not the earlier app-to-agent DNS failure.
• Around 14:02: only the affected agent container was restarted. Its existing registry image and WhatsApp credentials were retained.
• 14:02:30: app-container → runtime health returned HTTP 200 in 42 ms; WhatsApp status returned in 9 ms with linked, running, and connected true and no error.
• 14:02:47: an isolated production chat diagnostic completed through the native chat execution path using claude-default/claude-opus-5, returning OK in 7.4 seconds. Its persisted run was completed.

The observed loss of chat completions lasted approximately 1 hour 57 minutes. No stored-message loss was established. Receipt/replay of every WhatsApp message sent during the outage was not verified; a connected socket is not proof that those messages were delivered or answered.

Root cause

maybeCompactContextGraph checked out a shared PostgreSQL client for its entire operation. After reading context items it called summarizeForContext without returning that client. Provider preparation forcibly reloads provider configuration through the same shared pool, requiring another client. Ten concurrent compactions can therefore hold all ten clients while waiting for an eleventh. PostgreSQL sees idle clients, not a SQL lock cycle, so its deadlock detector cannot resolve this application-level circular wait.

Channel polling, WhatsApp status refresh, health checks, and foreground agent work all use this pool. They queued behind the blocked compactions. This explains why the event loop and metrics endpoint remained responsive while database-dependent work stopped.

Contributing gaps:

• Pool acquisition had no timeout.
• Lossless context summarization had no end-to-end deadline covering provider setup.
• Compaction retained clients across network/model work and could run concurrently without an admission bound.
• Summary/link/context replacement writes were not one transaction and did not reject stale snapshots.
• WhatsApp status GET had no whole-response deadline. The main settings panel silently ignored HTTP errors.
• Recovery monitoring treated an alive process too readily as a usable agent.

The exact scheduler interleaving was not captured from the deadlocked process. Production database evidence plus the deployed code establishes the circular-wait mechanism; the regression reproduces its resource pattern against real PostgreSQL.

Corrective changes

1. Release clients before inference. Snapshot reads use short pool queries. Leaf and condensed model calls hold no database client.
2. Commit atomically after inference. A short transaction uses the same per-session advisory-lock key as message appenders, verifies that every source item is still present, then inserts summary/provenance links and replaces context items together. A busy lock or stale snapshot skips the write. Original source messages remain stored.
3. Bound optional work. Each memory module admits at most two compactions and one per session. Excess work is skipped for a later turn without removing source context. Provider setup/inference has an aborting deadline, capped at 90 seconds, including a hard race so an uncooperative provider cannot retain admission indefinitely. Existing source-excerpt fallback is logged and retains provenance; late provider results cannot commit another summary.
4. Fail pool acquisition visibly. The agent defaults to a five-second acquisition/connection timeout. CLAPILOT_AGENT_DB_CONNECTION_TIMEOUT_MS supports a bounded 100–30000 ms override. Health queries also have a two-second query deadline and report pool total/idle/waiting/max counts on success and failure. An idle-connection error is logged instead of becoming an unhandled pool error.
5. End status loading. Primary and specialist WhatsApp status GETs have an eight-second whole-response deadline. Browser status requests have a twelve-second deadline and show an existing localized error. QR scan POST waits keep their separate longer budget. Interrupted bodies propagate as failures instead of returning a false empty success.

The native iOS/macOS clients have no corresponding WhatsApp QR/status settings surface in the inspected source. Their chats use the repaired shared runtime; no native UI change is required.

Regression and release verification

services/clapilot-agent/src/memory/context-compaction.integration.test.mjs uses an isolated schema, the real context-table migration, and a two-client PostgreSQL pool. It covers twelve concurrent compactions whose provider setup queries the same pool, foreground health during stalled inference, admission bounds, stale competing snapshots, transaction rollback, abort/late completion, leaf and condensed summaries, concurrent message append, and exhausted-pool health recovery. No external model or production credentials are used. The twelve-compaction case was also run against the pre-fix memory module: it failed with zero providers reaching their DB-backed setup and pool-acquisition timeouts. The fixed module passed all eight integration cases.

The dedicated Agent Compaction Pool Regression CI job provisions an ephemeral PostgreSQL 16 service and supplies the explicit integration-test URL. The ordinary unit suite must continue to run without database credentials. WhatsApp API tests cover connected status, failure before headers, interrupted body, and preservation of QR wait budgets.

Release gates: targeted regression tests, lint, production build, local source-built Docker verification, scoped commit to main, successful registry-image publish, then Watchtower deployment. Verify the deployed revision, health/pool counters, WhatsApp connected status, and a completed native chat response. An image push or container restart alone is not completion proof.

Permanent repair deployed

The repair commit is 668908de5b5db6f5b9a360068cdac3c31acf3eb1. The registry-image publish completed successfully. Watchtower replaced all three production runtime containers at 14:19:51–14:19:52 UTC, reporting updated=3, failed=0; both the web and agent containers were inspected and matched this revision.

At 14:20:23 UTC, the production app-container → agent probe returned health HTTP 200 in 35 ms, with pool total=5, idle=2, waiting=0, max=10. WhatsApp status returned HTTP 200 in 7 ms, linked/running/connected, zero reconnect attempts and no last error. A fresh native chat diagnostic on this revision returned OK from claude-default/claude-opus-5 in 3.7 seconds; its database run completed successfully. No external test message was sent on the user's behalf.

Recovery follow-up also verified that the pre-outage stuck chat was marked failed at 14:02:27 UTC and that a WhatsApp run completed at 14:03:26 UTC after the initial restart. This establishes processing recovery, not delivery/read confirmation for every message sent during the outage.

Validation: 772 local tests passed (764 related memory/settings/API/session tests and eight real-PostgreSQL regressions). Lint reported zero errors and 18 existing warnings. The production build and local Docker startup passed, with local health reporting zero queued pool requests. The separate host build compiled/typechecked/generated pages but hit the existing transient .git/refs/codex/turn-diffs file-copy race during packaging; the successful Docker build used a stable source snapshot. The dedicated Agent Compaction Pool Regression job passed in CI; the broader Unit Tests job was still running at this verification point and is not counted as passed here.

Runbook if symptoms recur

Compare /metrics with authenticated /health and /api/agent-runtime/transport-status?probe=1. Inspect pool counts and pg_stat_activity for the agent's container IP, including query age, transaction state, and last query. Test a fresh database connection and DNS separately. Do not kill arbitrary PostgreSQL sessions or relink WhatsApp merely because status is hanging. Preserve diagnostic evidence before restarting the affected agent. After recovery, verify a persisted completed chat run and fresh channel delivery; do not assume messages sent during the outage were replayed.