Operations

This page collects operational guidance for running Clapilot: container startup, local and native installs, GHCR (GitHub Container Registry) image publishing, CI browser checks, runtime diagnostics, and safe remediation for common failure modes. It is written for whoever operates an instance — self-hosted, CI, or production.

Clapilot.com mail deliverability

Clapilot-managed transactional mail currently uses the IONOS mailbox [email protected]. Set [email protected], the IONOS SMTP/IMAP hosts and credentials, and [email protected]. The envelope and visible From domain then remain aligned with the working clapilot.de SPF policy while replies go to [email protected]. Do not send with a visible From address at clapilot.com through IONOS.

The authoritative clapilot.com DNS zone is managed outside this repository. Keep exactly one SPF TXT record at the zone apex and use this rollout state:

clapilot.com.  TXT  "v=spf1 include:_spf.google.com ~all"
_dmarc.clapilot.com. TXT "v=DMARC1; p=quarantine; adkim=s; aspf=s; rua=mailto:[email protected]"

Before publishing the DMARC record, create or verify the monitored [email protected] mailbox/alias. Remove the previous Plesk SPF mechanisms (a, mx, and a:sweet-jang.93-90-204-86.plesk.page) rather than adding a second SPF record. Keep softfail while aggregate reports are reviewed. A later switch to Google SMTP requires a separate operational decision and Google Workspace configuration; only then may Clapilot use [email protected] as its From address.

After DNS propagation, verify there is exactly one SPF record and that DMARC reporting is published:

dig +short TXT clapilot.com
dig +short TXT _dmarc.clapilot.com
dig +short TXT google._domainkey.clapilot.com

Send one message through the configured Clapilot path to Gmail, Outlook, and GMX. For the current IONOS policy, confirm the visible From is [email protected], Reply-To is [email protected], and each provider reports spf=pass, dkim=pass, and dmarc=pass in the received headers. For a future Google path, run the same test with From: [email protected].

Container startup sequence

1. validate required environment
2. resolve native runtime config plus any mounted legacy compatibility state
3. seed workspace profile files
4. apply migrations (if enabled)
5. start clapilot-agent, workers, and app server

Core diagnostics

docker compose ps
docker compose logs -f clapilot
docker compose logs -f postgres
docker system df -v

Agent runtime transport diagnostics

Every app → ClapilotAICore call (/internal/... on clapilot-agent) goes through one shared transport (src/lib/agent-runtime/transport.ts). It replaces the bare TypeError: fetch failed that Node's fetch raises for any socket problem with a classified failure and retries transient ones:

• Failure kinds: dns (Docker service name did not resolve, e.g. EAI_AGAIN/ENOTFOUND), connect (ECONNREFUSED, EHOSTUNREACH, connect timeout – the request never reached the runtime), reset (ECONNRESET/EPIPE/socket closed mid-request, typically a runtime restart or stale keep-alive socket), timeout, tls, aborted, unknown.
• Retry policy: dns and connect failures are retried for every method (the runtime never saw the request); reset/timeout/unknown only for idempotent calls (GET/HEAD/PUT/DELETE or callers that pass idempotent: true). Backoff is exponential with jitter (300ms, 600ms, 1200ms, … capped at 3s). Streaming request bodies and caller-aborted requests are never retried.
• Errors surface as NativeAgentTransportError whose message names the method, full URL, cause code, attempt count, elapsed time and a remediation hint, for example Agent runtime unreachable (connect/ECONNREFUSED): POST http://clapilot-agent:3210/internal/chat/completions failed after 3 attempts in 2100ms — connect ECONNREFUSED 172.18.0.5:3210. The runtime host refused …. The same text lands in agent_runs/project error columns and in the app logs with the [agent-runtime-transport] prefix (one warning per retry burst, one error per final failure).
• Monitoring: the transport keeps in-process counters (requests, transport failures, retries, recoveries, consecutive failures, failures by kind, last 20 failures) and derives a health verdict: unreachable after three consecutive failures inside a minute, degraded for five minutes after any failure, otherwise healthy. Admins read it via GET /api/agent-runtime/transport-status (?probe=1 adds a live probe) or the runtime console command agent-transport-check. Only calls that target the native runtime origin (CLAPILOT_AGENT_BASE_URL) feed these counters: absolute compatibility-provider/gateway URLs that some routes (Aufgaben delegate/comments, Video Studio completions) pass through the same helper still get retry/backoff, but their failures are rethrown as-is, never counted as runtime failures and never flip the runtime health to unreachable.
• The live probe resolves the runtime hostname with DNS and calls GET /health without retries, reporting both layers separately so "name does not resolve" (Compose network / service name), "connection refused" (container down / wrong port) and "runtime unhealthy" (/health 503, usually the database) are distinct outcomes.

Environment knobs (.env): CLAPILOT_AGENT_TRANSPORT_RETRIES (default 2 extra attempts, max 6), CLAPILOT_AGENT_TRANSPORT_RETRY_BASE_MS (default 300), CLAPILOT_AGENT_TRANSPORT_HEADERS_TIMEOUT_MS (per-attempt budget until response headers; default 0 = disabled because runs and chat completions stream for minutes).

Triage order for sporadic fetch failed reports:

1. agent-transport-check in the admin runtime console (or curl -s "https://<host>/api/agent-runtime/transport-status?probe=1" as admin).
2. If DNS fails: docker compose ps, docker network inspect <project>_default, confirm CLAPILOT_AGENT_BASE_URL uses the Compose service name.
3. If connect fails: docker compose logs --tail=200 clapilot-agent, check for restarts (Watchtower updates restart the runtime container and reassign its IP; the transport retries through those windows).
4. If /health answers 503: the runtime cannot reach PostgreSQL – see db-host-check and the database logs.

Fleet connector updates

Fleet provisioning now enrolls every Hub-created customer instance into the external reachability monitor and issues its instance-bound telemetry sender credential before deployment. The instance adopts that credential on first boot and runs Hub telemetry synchronization immediately and every five minutes, even when no product event triggers a sync. Redeploying an older Fleet instance rotates and installs the credential; the customer instance adopts that current environment value on every boot, even when an older credential is already stored. This provides the onboarding and credential-recovery path for instances created before this behavior existed. Monitoring continues to probe the public HTTPS URL; a lifecycle recovery is successful only after three consecutive externally reachable responses.

Standalone Fleet connector 1.2.0 and newer self-update from the Hub. After a successful heartbeat, an idle connector downloads the allowlisted connector and event-outbox files, verifies the advertised version and JavaScript syntax, replaces the outbox first and connector last, and exits. Its launchd KeepAlive or systemd Restart=always service then starts the new version. Set CLAPILOT_FLEET_SELF_UPDATE=0 only when updates must be managed externally.

Connectors older than 1.2.0 still require one manual reinstall with the command generated on the Hub Fleet Machines page. That reinstall is the one-time bridge to self-updating releases and now downloads both required connector files. Clapilot Remote Runner.app follows its Sparkle release channel instead of the standalone self-update path.

Public reachability and HTTP 530 recovery

Fleet connector 1.4.2 adds robust project reconciliation to the public URL gate introduced in 1.3.0. A deploy is reported running only after both the local Compose services are healthy and the external Cloudflare URL returns 2xx/3xx, 401, or 403. The public gate requires three consecutive successful responses over at least ten seconds and has its own two-minute budget. Any HTTP 5xx response or network error resets the confirmation streak. If Compose applied but HTTP 5xx responses, including 530, persist, the job retains the public error and the instance is reported degraded rather than falsely running or treating Compose as absent.

The Hub customer monitor confirms a public outage with two consecutive probes. On the transition to down, it emits the existing one-time Team Chat alert and, for an active Fleet-managed kunde instance only, queues one deduplicated restart job. The connector reconciles the full Compose project with docker compose up -d --remove-orphans, so a stopped or missing cloudflared container is recreated as well as restarted, then applies the same external HTTP gate. If the public probe stays down, the Hub retries once after five minutes. Ten minutes after the second unsuccessful attempt it emits one Team Chat escalation naming the instance and the latest HTTP cause. A later confirmed healthy transition resets the recovery budget and emits one recovery alert.

Automatic recovery has a customer-fleet circuit breaker. Customer instances are probed and processed before development/internal targets, and only customer probe failures count toward the breaker. When at least three customer probes and at least half of that class fail together, the Hub still records and alerts the transitions but skips restart jobs because Hub egress, DNS, or a shared edge incident may be responsible. Isolated recovery is sent only to a machine seen within the last two minutes and running connector 1.4.2 or newer. Detailed failed-restart diagnostics require connector 1.4.5 or newer. Unclaimed restart jobs expire after 15 minutes, and a claimed recovery has a 10-minute job budget.

Incident 2026-08-16: yes.clapilot.com

• At 04:35 UTC the daily check recorded Cloudflare HTTP 530 with 328 ms latency. This is a public ingress/origin failure and is distinct from the earlier deploy-time Error 1033 incident.
• The endpoint had recovered to HTTP 307 (/login) by 2026-08-17 16:06 UTC. The execution environment did not retain the affected host's container or tunnel logs, so the initiating process exit cannot be identified reliably after the fact.
• The actionable root cause was a control-plane blind spot: Fleet considered a deployment healthy from Docker-local container state only. The periodic external monitor detected 530 but had no path to restart the origin/tunnel, and deploy completion did not prove that Cloudflare could reach the origin. A transient cloudflared or origin failure could therefore remain visible until an unrelated/manual restart.
• Recovery is now closed-loop: confirmed external outage → one Compose restart → three consecutive external URL confirmations → status-change recovery alert. Job logs retain compose_up, waiting_external, confirming_external, and external_reachable progress for future incident attribution.

Operational validation after rollout:

curl -sS -o /dev/null -D - --max-time 15 https://yes.clapilot.com
docker compose -p clapilot-yes ps
docker compose -p clapilot-yes logs --since 15m cloudflared clapilot

For a controlled recovery test, stop the Fleet project's cloudflared container, wait for two five-minute monitoring observations, and verify that a single restart job is created, docker compose up starts the tunnel, the connector records three confirmations followed by external_reachable, and Team Chat receives exactly one down and one recovery message. To exercise escalation, keep the public endpoint unreachable through both attempts and ten further minutes; verify one alert contains the instance hostname and latest HTTP/network cause. Perform this only in an approved maintenance window.

Local source-debug build

Use the local override when you want to rebuild clapilot from the checked-out source instead of the published GHCR image:

docker compose -f docker-compose.yml -f docker-compose.local-build.yml up -d --build clapilot

For a full local source-built stack:

docker compose -f docker-compose.yml -f docker-compose.local-build.yml up -d --build

The override switches clapilot back to build: . and keeps watchtower behind an opt-in profile.

The settings sidebar footer shows the current Clapilot web build tag, commit hash, local-vs-published build status, and the native agent build.

For the full runtime inventory, use Settings -> Versionen. That page expands the footer data into a live table with the current Clapilot web/native builds plus the installed versions and resolved paths for the directly integrated CLI tools Clapilot uses, including Codex CLI and Claude Code CLI.

Native host install CLI

For non-Docker installs on macOS or Linux, use the repository CLI instead of the dev launcher:

node scripts/clapilot-cli.mjs --help
node scripts/clapilot-cli.mjs install

If you want a global clapilot command for the checked-out repo:

npm link
clapilot --help

The installer supports both:

• interactive wizard flow
• scripted installs via flags like --env-file, --install-dir, --app-port, --agent-port, --db-port, --public-base-url, --database-url, and --yes

The GitHub-release macOS Apple client embeds the bundled host runtime and can install or control the local server directly. That client variant is built with the Clapilot GitHub Release Xcode scheme or clients/apple/ClapilotApple/scripts/package-github-release-client.sh; it is not used for Xcode Cloud/TestFlight builds. On first launch, when ~/.clapilot is missing, it installs from the bundled clapilot-runtime.tgz, creates launchd-managed web/agent services, seeds a local admin user, and then signs in without showing the normal login screen. On later launches it reads ADMIN_EMAIL, ADMIN_PASSWORD, and PORT from ~/.clapilot/.env.local, starts the managed services when needed, and signs in to the local web app.

When building the Clapilot GitHub Release scheme locally in Xcode, generate the bundled runtime first:

npm run build
npm --prefix clients/host run runtime:bundle

What the install command does:

• installs from the packaged runtime bundle when CLAPILOT_BUNDLED_RUNTIME_ARCHIVE is provided, otherwise bootstraps or reuses the target git checkout
• reads installer defaults and extra runtime vars from --env-file when provided
• writes .env.local for the native host runtime
• defaults the install path to ~/.clapilot
• installs or starts a local PostgreSQL service on supported hosts
• creates the local DB role/database when using the managed local PostgreSQL path
• runs the required runtime setup:
  • bundled runtime installs run the DB migration/admin-seed scripts directly without git clone, npm ci, or npm run build
  • source-checkout installs still run npm ci, npm run db:migrate, optional admin seed, and npm run build
• creates persistent native services:
  • Linux: systemd units
  • macOS: launchd agents
• can configure a Cloudflare tunnel by delegating to scripts/cloudflare-tunnel-setup.sh --env-file <native-env-file> when the required Cloudflare vars are present or explicitly requested
• writes install metadata under ops/native-runtime/install-meta.json

There is also a matching uninstall flow:

clapilot uninstall --install-dir ~/.clapilot --yes

That removes the native services, deletes the install directory, and drops the managed local DB objects when they belong to that install. It does not uninstall the system PostgreSQL package itself.

Example non-interactive install:

clapilot install \
  --yes \
  --env-file .env \
  --install-dir /srv/clapilot/current \
  --app-port 3000 \
  --agent-port 3211 \
  --db-port 5432 \
  --public-base-url https://app.example.com

Cloudflare tunnel env expected by the installer:

• CLOUDFLARE_API_TOKEN
• CLOUDFLARE_ACCOUNT_ID
• CLOUDFLARE_ZONE_ID
• CLOUDFLARE_TUNNEL_HOSTNAME
• optional CLOUDFLARE_TUNNEL_NAME
• optional CLOUDFLARE_TUNNEL_SERVICE (defaults to http://127.0.0.1:<app-port> for native installs)

To pull and redeploy later:

clapilot update --install-dir /srv/clapilot/current

Host app prototype

The repository also includes a desktop host shell in clients/host.

This is a separate gateway/controller app, not the existing Apple end-user client. It wraps the same native install flow so the GUI and terminal stay on one install path.

Use it when you want a local desktop control surface for:

• guided setup with defaults for a local machine
• install directory and env-file pickers
• optional Cloudflare tunnel setup through the existing repo script
• installed-state monitoring for web, agent, database, and Cloudflare
• menu bar / tray icon controls for opening settings, checking running services, and starting or stopping managed services
• host app release checks through GitHub Releases

Local development:

npm --prefix clients/host install
npm run host:dev

Bundle build:

npm run host:build

Current scope:

• macOS and Linux desktop targets
• plain Tauri shell with a static frontend
• runs the existing scripts/clapilot-cli.mjs install ... flow behind the wizard
• packaged builds include an embedded runtime archive so installs do not clone the full repo
• uses ~/.clapilot as the default install path
• shows a post-install status dashboard instead of raw command output
• keeps running in the menu bar / tray when the main window is closed
• includes a full-clean uninstall action in the dashboard
• does not yet embed PostgreSQL or cloudflared binaries inside the app bundle itself
• still depends on the underlying CLI for host-specific privilege handling

Release packaging:

• clients/host/src-tauri/resources/clapilot-runtime.tgz is generated fresh during builds and ignored by git
• .github/workflows/publish-host-release-tags.yml runs on release tags, builds the runtime archive and macOS host app on the self-hosted mac runner, signs the .app with a Developer ID Application certificate, notarizes and staples it, then uploads the signed app zip plus runtime archive as GitHub release assets
• required Actions secrets for signed releases:
  • APPLE_CERTIFICATE: base64-encoded .p12 for the Developer ID Application cert
  • APPLE_CERTIFICATE_PASSWORD
  • APPLE_SIGNING_IDENTITY
  • APPLE_API_KEY: App Store Connect API key .p8 contents
  • APPLE_API_KEY_ID
  • APPLE_API_ISSUER

Sparkle updates for GitHub-release macOS apps

The standalone Clapilot macOS client and Clapilot Remote Runner use Sparkle 2 for in-app updates. The tag-release workflow stamps both app bundles with the release version/build, embeds Sparkle.framework, signs and notarizes the apps, and generates a separate EdDSA-signed appcast for each final zip. It publishes those appcasts beside the existing zip and checksum assets in the private f1rede/Clapilot GitHub release.

End-user clients never access the private repository directly. app.clapilot.com uses the server-only GITHUB_RELEASES_TOKEN to locate recent releases, proxy the appcasts, and stream the signed zip or delta asset:

• GET /api/desktop-updates/clapilot/appcast.xml
• GET /api/desktop-updates/remote-runner/appcast.xml
• GET /api/desktop-updates/clapilot/download/<filename>
• GET /api/desktop-updates/remote-runner/download/<filename>

These endpoints are intentionally public because Sparkle calls them before any Clapilot web session exists. Configure GITHUB_RELEASES_TOKEN on the Clapilot web service as a fine-grained GitHub PAT with read access to the f1rede/Clapilot repository contents/releases. The token stays server-side and is never written into an appcast or app bundle.

The release workflow also requires the Actions secret SPARKLE_ED_PRIVATE_KEY. It is written only to a temporary runner file while generate_appcast signs the two feeds and is removed before the step exits. Both packaged Info.plists contain the matching EdDSA public key (SUPublicEDKey), the app-specific server feed URL, automatic daily checks, and the 24-hour scheduled-check interval. Treat the EdDSA key pair as durable release infrastructure: rotating the baked-in public key breaks the update chain for already-installed builds unless a deliberate key-rotation migration is shipped first.

The first Sparkle-enabled release is a bootstrap release and must be downloaded and installed manually. Once that build is installed, later signed releases can be discovered and installed from the app menu or settings/menu-bar update action.

Host-local native dev

For fast debugging without the container wrapper, the repo also ships a host-local launcher for the Next.js app plus clapilot-agent:

npm run dev:local-native

Running the same command again restarts the launcher by stopping the previous host-local instance first.

Default loopback ports:

• app: http://localhost:3100
• native runtime: http://localhost:3211

Optional: start only the PostgreSQL Compose service before launching host processes:

npm run dev:local-native:docker-db

Port overrides:

CLAPILOT_LOCAL_APP_PORT=3200 CLAPILOT_LOCAL_AGENT_PORT=3311 npm run dev:local-native

The launcher:

• reads .env.local first, then .env
• points the app to CLAPILOT_AGENT_BASE_URL=http://127.0.0.1:<agent-port>
• points clapilot-agent back to CLAPILOT_INTERNAL_BASE_URL=http://127.0.0.1:<app-port>
• uses the checked-out repository as CLAPILOT_WORKSPACE_DIR
• stores native runtime state under the local .clapilotaicore/ directory
• disables background Telegram/WhatsApp channel startup by default so the host-local runtime does not collide with a Docker runtime that is already polling those channels
• prewarms common top-level routes in the background by default, including /login, /, /chat, /dokumente, /mandanten, /emails, /aufgaben, and /calendar, so the first manual browser visit pays less cold-compile cost in next dev

Stop it explicitly with:

npm run dev:local-native:stop

In-instance developer E2E suites

When developer mode is enabled, administrators can run the isolated regression suites from the developer settings panel. Cleanup mode deletes throwaway sessions and seeded artifacts by default; keep mode retains them for debugging.

The scheduled workflow in .github/workflows/nightly-e2e.yml runs this complete catalog after its browser suite against the same temporary source-built instance. It temporarily enables Developer mode, executes every registered suite sequentially with artifact cleanup enabled, restores the previous Developer mode value, and fails if any suite does not pass. Results are stored as instance-e2e-results.json and instance-e2e-report.md and are included in the hosted nightly report. The optional CLAPILOT_CI_INSTANCE_E2E_MODEL repository variable selects an explicit model; otherwise the suites verify the instance's normal default routing.

GHCR auth for default image deploys

The default compose stack requires two GHCR auth paths:

• host Docker auth in ${HOME}/.docker/config.json for docker pull and docker compose up
• Watchtower auth in ${HOME}/.docker/watchtower-config.json

Populate the host Docker auth with:

mkdir -p "${HOME}/.docker"
echo "<GHCR_READ_PACKAGES_TOKEN>" | docker login ghcr.io -u "<github-username>" --password-stdin

The Watchtower file should contain inline registry auth for ghcr.io, for example:

{
  "auths": {
    "ghcr.io": {
      "auth": "<base64(GITHUB_USERNAME:GHCR_READ_PACKAGES_TOKEN)>"
    }
  }
}

Requirements:

• the token needs read:packages
• use ${HOME}/.docker/..., not ~/.docker/..., in Compose-backed paths
• docker login ghcr.io ... only solves host pulls; it does not configure Watchtower
• Watchtower needs a separate mounted file because host Docker configs commonly rely on credsStore, which is not available inside the Watchtower container
• on another Mac, create both files in that user's home directory or change the bind mount in docker-compose.yml
• the local source-build override in docker-compose.local-build.yml does not require this file unless you explicitly enable the watchtower profile

Container publishing

The repository includes a GitHub Actions workflow at .github/workflows/publish-container-main.yml. It publishes the runtime image to GHCR on every push to main from the self-hosted mac-mini-m4-pro (macOS/ARM64) runner.

Published tags:

• ghcr.io/<owner>/<repo>:latest-arm64
• ghcr.io/<owner>/<repo>:main-arm64
• ghcr.io/<owner>/<repo>:sha-<full-commit-sha>-arm64

When an immutable release container package is needed, run .github/workflows/publish-container-release-tags.yml manually with the release git tag. The workflow waits for the main image with the same commit SHA and retags that image into a separate release package:

• ghcr.io/<owner>/<repo>-release:<git-tag>-arm64
• ghcr.io/<owner>/<repo>-release:release-latest-arm64
• ghcr.io/<owner>/<repo>-release:sha-<full-commit-sha>-arm64

For manual branch or ref builds, .github/workflows/deploy-feature-test-build.yml publishes a single rolling feature/test image:

• ghcr.io/<owner>/clapilot-test:test-latest-arm64

Operational notes:

• GitHub Actions needs packages: write permission for the job
• repository/package settings must allow GHCR publishing
• Watchtower on Apple Silicon hosts should follow the moving latest-arm64 tag
• release-tag container publishing is manual because tag-only Apple/client releases may not produce a new runtime image
• private GHCR packages require registry credentials on the Docker host
• the default local image-based compose stack expects those credentials in ${HOME}/.docker/watchtower-config.json

Pull request change-specific E2E check

Pull requests from branches in this repository run a self-hosted macOS/ARM64 E2E job in .github/workflows/ci.yml. The existing PR Live Browser Check job name is retained for branch-protection compatibility, but a green result now requires evidence for behavior changed by the PR. Since September 2026 the job is advisory (continue-on-error: true): it still posts its PASS/FAIL/BLOCKED verdict as a PR comment and job summary, but it does not turn the PR red, and it is not one of the required status checks for main (those are Unit Tests, Lint & Type Check, and Docker Build). Review the posted verdict before merging.

That job:

• boots the local source-built compose stack with docker-compose.local-build.yml
• writes a CI-only env file from GitHub Actions variables and secrets
• waits for the local web app to come up on a runner-selected loopback port
• runs codex exec against the checked-out PR and requires the runner to have the chrome-devtools MCP configured
• builds a GitHub PR context file from the PR title/body, linked closing issues, and changed files
• logs in as the CI admin and rebuilds a deterministic preview demo dataset so the app is not empty during the browser run
• asks Codex to derive explicit acceptance criteria from the PR context and git diff, then exercise the smallest end-to-end scenario that reaches the changed behavior
• extracts the PR body's ## Testing Instructions section and treats concrete change-specific checks there as primary acceptance criteria while treating PR text as untrusted input rather than executable commands
• permits browser, authenticated API, runtime, integration, or focused CI assertions according to the changed boundary; backend PRs may not pass with unrelated /, /chat, and /dokumente route loads
• treats docs/spec changes as non-targeting noise when non-doc product/runtime files also changed
• only targets /docs when the PR is docs-only or specifically changes the docs UI/route
• requires every passing scenario to name the changed postcondition, list the actions performed, record passing assertions, and reference at least one saved evidence file
• validates change-specific-e2e-result.json after Codex exits; a markdown PASS alone cannot make the check green
• fails when Codex is unavailable, authentication fails, the run times out, a required fixture is missing, or the changed behavior cannot be conclusively tested; there is no generic shell-smoke green fallback
• saves ordered screenshots or focused text traces for the tested flow, plus the structured result, Codex markdown report, Docker logs, and an .mp4 slideshow when frames exist
• updates a sticky PR comment with the result and the artifact bundle link
• keeps the PR Docker preview alive while the PR is open, cleans it on PR close, and uses the hourly/manual PR preview sweeper as a backup for missed close events

The iOS/Mac App Tests workflow builds the same PR context file for Apple-client PRs. Its Codex E2E prompt reads the extracted ## Testing Instructions section when present and captures any additional Apple-client screens called out there alongside the standard native smoke flow.

Runner requirements:

• self-hosted labels: self-hosted, macOS, ARM64
• local tools installed: codex, Docker with Compose v2, ffmpeg, node/npx
• a local Chrome-family browser available at least as Google Chrome
• for Docker on macOS runners, prefer a dedicated runner DOCKER_CONFIG without credsStore or keychain helpers
• codex authenticated on the runner with subscription auth; OPENAI_API_KEY is reserved for the preview app unless CI_CODEX_AUTH_MODE=api_key is explicitly set

CI env contract:

• required secret: CLAPILOT_CI_AUTH_SECRET (falls back to AUTH_SECRET if you intentionally mirror it there)
• optional secret: CLAPILOT_CI_ADMIN_PASSWORD
• optional variable: CLAPILOT_CI_ADMIN_EMAIL
• optional multiline variable: CLAPILOT_CI_ENV_VARS
• optional multiline secret: CLAPILOT_CI_ENV_SECRETS
• fallback multiline secret already used elsewhere in this repo: CLOUD_RUN_ENV_VARS
• optional variable: CLAPILOT_CI_DEMO_SEED (true by default)
• optional variable: CLAPILOT_CI_DEMO_SEED_SCENARIO (defaults to the app's default demo scenario)
• optional variable: CLAPILOT_CI_OPENAI_MODELS for the native OpenAI provider catalog shown only in preview/demo instances
• optional variable: CLAPILOT_CI_EMAIL_ANALYSIS_MODEL for lightweight mail/document analysis defaults in preview/demo instances
• optional variable: CI_CODEX_AUTH_MODE for the Codex test runner auth source (subscription by default, api_key only for explicit fallback)
• optional variable: CI_CODEX_MODEL for the Codex test runner model (gpt-5.6-sol by default)
• optional variable: CI_CODEX_REASONING_EFFORT for the Codex test runner reasoning level (medium by default)
• optional variable: CLAPILOT_CI_TEST_EMAIL_ADDRESS
• optional secret: CLAPILOT_CI_TEST_EMAIL_PASSWORD
• optional variables: CLAPILOT_CI_TEST_IMAP_HOST, CLAPILOT_CI_TEST_IMAP_PORT, CLAPILOT_CI_TEST_SMTP_HOST, CLAPILOT_CI_TEST_SMTP_PORT
• optional hosted upload API variable: LIVE_CHECK_UPLOAD_API_URL
• optional hosted upload API secret: LIVE_CHECK_UPLOAD_TOKEN
• optional hosted publish variables: LIVE_CHECK_SHARE_HOST, LIVE_CHECK_SHARE_USER, LIVE_CHECK_SHARE_REMOTE_PATH, LIVE_CHECK_SHARE_BASE_URL
• optional hosted publish secrets: LIVE_CHECK_SHARE_SSH_KEY, LIVE_CHECK_SHARE_KNOWN_HOSTS
• PR preview cleanup/sweeper variables/secrets: CLAPILOT_PREVIEW_RUNNER_LABELS, optional CLAPILOT_PREVIEW_CLEANUP_RUNNER_LABELS, optional CLAPILOT_PREVIEW_POOL_LABEL (default clapilot-preview, shared label on every preview-capable runner for load-based first assignment), CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_ZONE_ID, CLOUDFLARE_PREVIEW_DOMAIN

The live browser seed reuses that test mailbox for both the agent mailbox defaults and the seeded admin user's personal mailbox profile, so /emails and background agent mail actions can both run against the same non-production account.

For new self-hosted macOS runners, bootstrap Docker once with scripts/ci/setup-runner-docker-config.sh. That script writes a clean config such as ${HOME}/.docker-ci/config.json with no credsStore, so GitHub Actions jobs do not depend on an interactive login keychain session.

Install the machine dependencies first with scripts/ci/setup-macos-runner-deps.sh. It installs codex, ffmpeg, node, and Google Chrome via Homebrew, which covers the prerequisites used by the self-hosted browser jobs.

Recommended runner setup:

• run scripts/ci/setup-macos-runner-deps.sh
• authenticate codex login for the runner user
• run scripts/ci/setup-runner-docker-config.sh
• set the runner service environment to DOCKER_CONFIG=${HOME}/.docker-ci
• if the runner needs registry auth, log in with docker --config "${HOME}/.docker-ci" login ...
• the live-browser workflow bootstraps the chrome-devtools MCP itself with npx; no permanent MCP config is required on the machine
• do not rely on Docker Desktop's default keychain-backed credential store for headless runner jobs

The browser-oriented self-hosted workflows also run scripts/ci/setup-macos-runner-deps.sh before tool verification, so a newly provisioned runner can install missing node/npx, ffmpeg, codex, or Google Chrome automatically instead of failing immediately.

Use the multiline env entries for extra compose env such as provider keys or feature flags that should only exist in the live-browser CI run.

If the hosted upload API config is present, the workflow uploads the captured frames, report, gallery page, and optional .mp4 over HTTPS and embeds the public frame URLs into the PR comment.

If the upload API is not configured but the SFTP-based hosted publish config is present, the workflow falls back to SFTP publishing.

E2E performance gate

The live-browser E2E runner executes node scripts/perf-radar.mjs --gate --smoke inside the running clapilot container after a successful browser suite. The historical gate reads terminal agent_runs for the current Berlin calendar day, so one successful chat turn is one latency sample even when tool loops or provider fallback produced several agent_model_request_logs. It performs one active authenticated /api/chat smoke turn in a fresh, titled, disposable chat session. The fixed title prevents the probe from starting an unrelated model-backed auto-title request. Only a smoke request carrying the app's internal secret is tagged as ci_perf_gate and excluded from the historical rows; a caller-controlled page context alone cannot exclude traffic. A smoke turn passes only when its response ends with a fresh per-run verification token, its exact runtime session has a completed ci_perf_gate run created after the probe began, and the same run has a completed provider request log. This correlation prevents an HTTP-200 error stream or an unpersisted runtime failure from being reported as successful. The runner always writes perf-gate-report.txt into the E2E artifact directory, including an explanation when the gate is intentionally skipped.

The default gate thresholds are configurable through CI environment variables:

• PERF_MAX_FAIL_PCT: maximum daily terminal agent-run failure rate, default 10.
• PERF_MAX_CHAT_P90_S: interactive chat p90 SLO in seconds, default 90. This initial unfiltered target should be tightened only after enough post-rollout samples establish the new baseline.
• PERF_MAX_CHAT_P90_REGRESSION_PCT: maximum increase over the average p90 of preceding days that meet the minimum request count, default 20. The gate fails on either the absolute SLO or the relative regression.
• PERF_MIN_REQS_FOR_GATE: minimum daily count of successful chat turns before historical thresholds fail the run, default 15.
• PERF_SMOKE_TIMEOUT_S: per-request timeout for login, session setup, and the streamed smoke response; defaults to at least 120 seconds.
• PERF_SMOKE_MESSAGE: optional main-chat smoke prompt. It is forwarded from either the Compose/CI environment or the workflow host. It must not contain an @handle; the radar rejects such prompts before login so the probe cannot enter specialist-agent routing. The radar appends its own unique verification-token instruction to custom prompts.

These variables may be supplied through the CI container env file or exported by the workflow host. Empty values fall back to the defaults. When active smoke is enabled, missing smoke credentials fail --gate instead of silently skipping the probe.

Normal native one-to-one chat turns are intentionally persisted with source_type=chat, so they participate in the chat p50/p90 latency gate. This can shift the latency baseline compared with older builds where some native rows had a null source type. The daily failure-rate population does not change for those rows because it already counts every non-ci_perf_gate request regardless of source type. Treat the first full day after rollout as the new latency baseline and change thresholds only when the production SLO calls for it.

The radar does not discard turns longer than five minutes. It reports the ten slowest successful chat turns with a hashed run ID and an end-to-end split from agent_chat_turn_phase_metrics: queue time, provider/orchestration time outside tools, tool execution, failed provider/fallback time, and finalization after the last provider or tool event. The five phases add up to the complete agent_runs.created_at to completed_at duration (apart from millisecond rounding). Provider/model, hashed session, estimated prompt-size, and initial versus post-tool request breakdowns remain diagnostic attempt-level views. Use --compare-cutover <ISO timestamp> --compare-sample-size <N> to compare the last N successful chat turns before a deployment with the first N after it; when either cohort is short, the report shows the available counts instead of silently omitting the comparison. This is the canonical before/after measurement for latency changes.

The 29 July 2026 investigation showed that final run status alone was not a useful latency signal. In the available Top-10 sample, seven turns spent more than 100 seconds end to end. Six of those repeatedly crossed provider/tool boundaries (11-46 recorded tool starts), while one 297-second turn spent about 253 seconds in the queue before a 45-second execution. No failed provider attempt was needed for most of these successful outliers. This is why the gate uses one completed run per sample and why the Top-10 phase report must be reviewed even when the run failure rate remains below its threshold.

OpenAI-compatible streaming requests have a time-to-first-response and stream-idle deadline controlled by CLAPILOT_PROVIDER_REQUEST_TIMEOUT_MS (default 120000). Data events and SSE comment heartbeats refresh the deadline, so healthy long generations are not aborted by total wall-clock duration. Non-streaming Responses and chat-completions calls use the separate wall-clock cap CLAPILOT_PROVIDER_NON_STREAMING_REQUEST_TIMEOUT_MS (default 300000) because they cannot expose progress heartbeats. Expiry before user-visible output enters configured provider fallback/circuit handling without replaying the same slow request; expiry after output terminates the stalled stream because replaying it would duplicate visible content. Other short-lived transient failures retain the bounded retry policy controlled by CLAPILOT_PROVIDER_TIMEOUT_MAX_ATTEMPTS. OpenAI and Azure OpenAI streaming chat-completions requests send stream_options.include_usage=true. OpenAI-compatible servers receive the same standard field only when the existing compatible prompt-cache opt-in is enabled, preserving the strict-server fallback. Request-log metadata records timeToFirstTokenMs from the first meaningful reasoning, text, or tool-call delta; missing usage remains NULL rather than being recorded as a misleading zero. The same guards cover OpenAI Responses API requests.

Direct human-facing chat turns intentionally have no end-to-end runtime deadline. A model/tool loop may therefore continue for as long as the task requires; only an explicit caller-owned timeout can cap the whole conversation. The stdio MCP bridge and Codex MCP rescue path likewise add no independent wall-clock deadline around /api/agent-runtime/tool-proxy; the retired CLAPILOT_MCP_TOOL_PROXY_TIMEOUT_MS and CLAPILOT_MCP_IMAGE_TOOL_PROXY_TIMEOUT_MS settings are ignored. Independent provider-stall, native tool-phase and fallback safeguards default to 120, 120 and 45 seconds and are never clipped by elapsed turn time. A chat turn queued behind a long turn in the same session survives up to CLAPILOT_CHAT_QUEUE_BUDGET_MS=300000 before it is failed (and the failure is persisted as a failed run). The paid mixture-of-agents reference fan-out is exempt from the preflight cutoff, and finalization overruns are logged warnings that never fail an already-successful run. Long tools emit tool.progress every 30 seconds. These phase safeguards never apply to jobs, automations, channel runs, or orchestrator turns; long-running background work is first-class. Phase defaults and their environment variables are documented in .env.example. Repository Orchestrator turns are reported separately and keep the 45-minute broker timeout (CLAPILOT_AGENT_TURN_TIMEOUT_MS=2700000, 10-minute floor); an expired orchestrator turn is actively interrupted on the app-server (turn/interrupt) instead of being left running detached. The Performance Radar computes user-visible latency from agent_runs, not individual model requests, and prints separate direct_chat, orchestrator, and automation_channel_api cohorts. Its slow-run section joins agent_events to expose queue/preflight, provider, tool and finalization timing for the top outliers.

Provider fallbacks are failure-domain aware. The provider slug is the failure-domain key, so a connection failure or provider stall quarantines that gateway for the current request and skips later models on the same slug. The remaining fallback budget is divided across the still-reachable independent provider slugs, preserving time for the final emergency provider. Provider-attempt telemetry records skipped_failure_domain candidates and remainingFallbackBudgetMs. Saving runtime provider configuration returns a warning when a configured fallback chain contains models but no enabled independent provider slug.

Every turn also has a hard provider envelope: three upstream requests, two provider failure domains, and two identical failures at one provider/model/endpoint boundary by default. Configure these with CLAPILOT_PROVIDER_TURN_MAX_REQUESTS, CLAPILOT_PROVIDER_TURN_MAX_FAILURE_DOMAINS, and CLAPILOT_PROVIDER_TURN_MAX_IDENTICAL_FAILURES. An open timeout circuit permits one half-open recovery request per CLAPILOT_PROVIDER_TIMEOUT_CIRCUIT_PROBE_INTERVAL_MS (30 seconds by default); every failed probe doubles that wait, capped by the open duration, and the backoff is carried across re-opens until a probe succeeds. Every re-open without a success in between doubles the open window itself (10, 20, 40, 80 minutes, capped by CLAPILOT_PROVIDER_TIMEOUT_CIRCUIT_MAX_OPEN_MS, default 2 hours). When a fallback can answer the turn, the probe or the first request after the window receives a bounded stall budget instead of the full one: CLAPILOT_PROVIDER_TIMEOUT_CIRCUIT_PROBE_TIMEOUT_MS (20 seconds by default) doubled per failed probe up to the provider's normal budget, never more than half of the remaining conversation deadline, so a stalling provider such as Grok costs the turn one bounded probe and the independent fallback still answers; the attempt records circuitProbe=true, probeRequestTimeoutMs, and probeBudgetReduced=true when the budget was shorter than the request would normally get. A probe that stalls against a reduced budget re-opens the circuit without doubling the open window (only failures against the normal budget escalate), so a slow-but-healthy model is measured against its real budget after a few probes instead of being ratcheted out of routing for hours. The same escalation applies to the endpoint (connection) circuit when its probe fails with a timeout. The timeout circuit counts fixed provider timeouts (stall deadlines, HTTP 408/504/524, hard conversation deadlines that fired on the provider request) both consecutively and inside a sliding CLAPILOT_PROVIDER_TIMEOUT_CIRCUIT_WINDOW_MS window (15 minutes by default), and it also counts stalls that happen after tool progress in the same run. Once open, later runs skip the provider with status=circuit_open and continue with the configured fallback; the terminal finalProvider attempt names the model that actually answered. The provider status endpoint (GET /api/agent-runtime/provider-status) and the provider dialog in Einstellungen show the open, half-open, and degraded circuits per model with the counted timeouts, open-until time, next probe time, and last error, so an unstable provider such as a stalling GLM endpoint is visible without reading logs. The state is mirrored into agent_provider_circuit_states (migration 305) and restored on the first provider request after a runtime restart, so a Watchtower image update or crash no longer resets an open circuit; a successful probe closes the circuit and deletes the row, a restore that fails because the database is not reachable yet is retried on the next request, and stale rows are pruned at restore time. Because a restart no longer clears the breaker, every circuit row in the provider dialog has a reset button (DELETE /api/agent-runtime/provider-status/circuit) that removes the in-memory state and the database row; the dialog also shows the failed probe count, whether the state was restored after a restart, and whether the database mirror is currently active. Attempt telemetry includes used and remaining request budget, used failure domains, identical-failure count, circuit state (circuitOpenTrigger, circuitReopenStreak, circuitOpenDurationMs), and a terminal finalProvider marker.

Verifying a provider timeout regression fix in production (for example the recurring Grok-4.6 Provider conversation exceeded its 60s limit / Provider request stalled for 22s cascade) requires evidence from the running instance, not only a merged pull request. Keep the ticket open until all of the following hold on .24 after the image rolled out: (1) GET /api/agent-runtime/provider-status?slug=<xai-slug> shows the circuit as open with reopenStreak growing while the provider still stalls, persisted: true, and restored: true after a container restart; (2) SELECT circuit_key, open_until, state->>'reopenStreak' FROM agent_provider_circuit_states lists the tripped model; (3) the daily count of failed Grok attempts in agent_model_request_logs drops to the bounded probe schedule (single-digit probes per hour at most, each stalled at the probe budget rather than the 45s/60s conversation limit) while the same turns show finalProvider on an independent fallback provider; (4) at least one independent fallback provider slug is enabled in the xAI provider's fallback chain, because without it the breaker can only fail faster, not answer.

The official Clapilot runtime image installs Cursor's official CLI and exposes it as cursor-agent; verify it with cursor-agent --version inside the ClapilotAICore container. Custom images can opt out at build time with INSTALL_CURSOR_CLI=false. Set CLAPILOT_AGENT_CURSOR_CLI_COMMAND when a custom deployment keeps the executable outside PATH, then add the Cursor provider and its User API key in Settings. Clapilot does not embed the proprietary Cursor SDK. A missing CLI disables only the Cursor row; direct providers and other subscription bridges remain available as fallbacks.

Chat phase budgets can be overridden per provider and per model in runtime config instead of environment variables. The provider edit dialog (settings → ClapilotAICore → providers) exposes dedicated second-based fields under "Timeouts & latency budgets" for model response without output, tool execution, queue wait, fallback budget, and the background stall timeout; empty fields keep the system defaults. The fields persist to chatLatencyBudgets in the provider's metadata as an object with any of queueMs, preflightMs, providerMs, nonStreamingProviderMs (whole-response budget for non-streaming interactive turns, advanced metadata JSON only), toolMs, fallbackMs, finalizationMs; totalMs from older configurations is ignored and cannot restore a whole-conversation deadline. Add modelChatLatencyBudgets keyed by model name (advanced metadata JSON) for model-specific values that win over the provider-level object. Example for a slow self-hosted gateway: {"chatLatencyBudgets": {"providerMs": 300000, "queueMs": 900000}}. Model overrides beat provider overrides, which beat the CLAPILOT_CHAT_* environment defaults; the environment variables remain global fallbacks only. Because the queue check for a waiting turn runs before provider selection, it uses the budgets resolved by the previous turn in the same session. These budgets apply to interactive turns: direct chat and interactive channel turns (Telegram, WhatsApp, Slack, Signal, iMessage, Team-Chat mirrors) alike, so a stalled Team-Chat request fails at the phase budget with a saved checkpoint instead of running into the 20-minute channel cap. The tool budget protects only turns that have not yet confirmed a successful tool completion. After confirmed tool progress, later validation/finalization tools are not phase-aborted because replay could duplicate an already executed mutation; explicit user cancellation remains active. Background workloads (jobs, automations, automation-keyed channel deliveries) stay unbudgeted, but their provider stall timeout can be raised per provider with metadata providerRequestTimeoutMs (optionally modelProviderRequestTimeoutMs keyed by model) when a self-hosted model needs more than the default 120 seconds before its first streamed token.

OpenAI-compatible providers can also opt into per-model request scheduling through provider metadata. Set maxConcurrentRequests to cap simultaneous upstream requests and maxBackgroundConcurrentRequests to reserve the remaining capacity for interactive chat. Interactive requests are always dequeued before background automation. The scheduler is disabled when these metadata fields are absent, and request logs record queue wait, priority, and the active limits for diagnosis.

Set CI_PERF_GATE_ENABLED=false only for emergency runner diagnostics. The active smoke remains useful when the day has too little traffic for the historical minimum, because smoke failures and smoke latency breaches are evaluated independently.

Agent runtime console (admin UI)

For Cloud Run setups without container shell access, use:

• the admin runtime console in Clapilot settings

Built-in diagnostics:

• help
• ls-state-dir
• show-clapilotaicore-json
• validate-clapilotaicore-json
• node-version
• db-host-check
• agent-transport-check (app → clapilot-agent transport counters plus a live DNS + /health probe; see "Agent runtime transport diagnostics")

Notes:

• access is admin-only (same role check as other admin APIs)
• the packaged OpenClaw CLI is no longer present in the Docker image
• this is intentionally not a full unrestricted OS shell
• use it to diagnose native runtime connectivity and state remotely

RAG runtime controls

RAG (retrieval-augmented generation) runtime toggles (container env):

• RAG_ENABLED=true enables retrieval injection in /api/chat (default: enabled)
• RAG_INDEXER_ENABLED=true enables background indexing worker (default: enabled)
• RAG_RETRIEVAL_TOP_K and RAG_RETRIEVAL_MIN_SCORE tune retrieval breadth/quality
• RAG_CHUNK_SIZE and RAG_CHUNK_OVERLAP tune chunking granularity
• embeddings key source: OPENAI_API_KEY or app_settings.openai_api_key

RAG diagnostics:

docker compose logs -f clapilot | rg -n "rag-indexer|chat-rag"

Instance cleanup worker

scripts/instance-cleanup-worker.mjs is the retention engine of an instance. Its decision logic lives in scripts/lib/instance-cleanup-core.mjs and is unit tested in tests/scripts/instance-cleanup-core.test.ts.

• CLAPILOT_INSTANCE_CLEANUP_ENABLED=true starts the web-container cleanup worker (default: enabled)
• CLAPILOT_INSTANCE_CLEANUP_POLL_SECONDS=1800 controls the cleanup interval
• CLAPILOT_INSTANCE_CLEANUP_MIN_FREE_GB=20 and CLAPILOT_INSTANCE_CLEANUP_TARGET_FREE_GB=40 define the disk-pressure window
• CLAPILOT_INSTANCE_CLEANUP_MAX_DELETE_GB=25 caps deletion per cycle
• CLAPILOT_INSTANCE_CLEANUP_DRY_RUN=true logs planned deletions without removing files
• CLAPILOT_INSTANCE_CLEANUP_LINKED_SESSION_IDLE_HOURS=72 limits how long an idle (ready, failed, ...) external session protects its workspace; running sessions always protect
• CLAPILOT_INSTANCE_CLEANUP_TOOL_CACHE_AGE_HOURS=72 is the minimum age for cache-tier entries
• CLAPILOT_INSTANCE_STORAGE_REPORT_SECONDS=3600 controls how often the storage attribution report is refreshed

Retention tiers:

• Scratch tier (deleted as soon as the retention window has passed, regardless of disk pressure): Agent Orchestrator PR-review (6h), issue (24h), main-CI (24h), tracked-PR (24h), manual-job (24h), and Symphony (72h) workspaces, orchestrator session directories (72h), unlinked bare repo caches (336h), and tmp/.tmp (72h). Worktrees with unpublished commits are kept for CLAPILOT_INSTANCE_CLEANUP_UNPUBLISHED_RETENTION_HOURS (168h).
• Cache tier (only drained while free space is below CLAPILOT_INSTANCE_CLEANUP_MIN_FREE_GB): the Claude CLI runtime's pip and npm caches under .clapilotaicore/claude-cli. They are re-downloadable, so they are sacrificed before anything else has to be touched manually, but never while the disk is healthy.
• Protected (never deleted automatically): mandanten, .clapilot/* generated media, .clapilotaicore runtime state and attachments, installed modules/skills, and every other workspace directory such as video-studio, livestream, podcasts, or exports. These only appear in the storage report.

Protection rules:

• Candidates are always direct children of an allowlisted cleanup root; roots that equal or contain the workspace root or a protected path are rejected at startup.
• Workspace paths reported by live agent_external_sessions rows and queued/running orchestrator jobs protect exactly the workspace they name. A session bound to a broad path such as the workspace root itself is ignored and counted in the cycle log, because honoring it would shadow every candidate (this was the bug that let orphaned worktrees fill the disk to 99%).
• If the session table or jobs.json cannot be read, worktree roots are skipped for that cycle instead of guessing.
• Under pressure the largest expired entries go first; otherwise the oldest go first.

Each cycle logs [instance-cleanup] cycle complete: candidates=… removed=… reclaimed=… free=… level=ok|warning|critical. Once per CLAPILOT_INSTANCE_STORAGE_REPORT_SECONDS the worker measures the workspace by source (orchestrator worktrees, repo caches, runtime caches, generated media, work data, database size) and writes <state dir>/instance-storage/report.json. The report feeds Einstellungen → Speicher (/admin/storage), GET /api/admin/storage, and the instance_storage_status agent tool.

Instance storage monitor and alerts

The Next.js server starts an in-process monitor from src/instrumentation.ts (src/lib/instance-storage.ts). Every five minutes it reads the workspace filesystem via statfs, logs a workspace_disk_pressure event when the level is not ok, and posts to the configured monitoring alert room as Speicher-Monitoring:

• warning when free space drops below CLAPILOT_INSTANCE_STORAGE_WARN_FREE_GB (15) or usage exceeds CLAPILOT_INSTANCE_STORAGE_WARN_USED_PERCENT (90)
• critical when free space drops below CLAPILOT_INSTANCE_STORAGE_CRITICAL_FREE_GB (5) or usage exceeds CLAPILOT_INSTANCE_STORAGE_CRITICAL_USED_PERCENT (97)
• a single recovery message once the level returns to ok

Level changes always post immediately; the same level is repeated only after CLAPILOT_INSTANCE_STORAGE_ALERT_COOLDOWN_HOURS (6). Alert state is derived from the latest chat_group_messages row with message_meta.source = instance_storage_monitor, so restarts do not re-alert. The same thresholds drive the level shown on /admin/storage and in the cleanup worker log.

The target room is shared with hub customer monitoring: Settings -> Hub -> Monitoring -> Target channel for system alerts (app_settings.monitoring_alert_room_id, GET/PUT /api/hub/monitoring/alert-room). Empty means the main Team Chat room clapilot-members (#general). At delivery time the monitor verifies that the configured room still exists and is neither deleted nor archived; otherwise it posts to the main room, logs monitoring_alert_room_fallback with source, configuredRoomId, and reason, and marks the message with message_meta.roomFallback = true. A cycle is skipped with room_missing only when the resolved room itself does not exist.

The Agent Orchestrator additionally refuses to materialize a new worktree when free space is below AGENT_ORCHESTRATOR_MIN_FREE_DISK_GIB (5; 0 disables), and its own hourly workspace GC stops treating idle linked chat sessions as live after AGENT_ORCHESTRATOR_LINKED_SESSION_IDLE_TTL_HOURS (72). AGENT_ORCHESTRATOR_WORKSPACE_HARD_LIMIT_GIB remains available as an absolute worktree quota.

Video Studio project reconciler

• src/instrumentation.ts starts the in-process Video Studio reconciler with the Node server.
• CLAPILOT_VIDEO_STUDIO_RECONCILE_SECONDS=60 controls the interval; 0 disables it.
• Each non-overlapping tick checks at most ten owner-fair active projects in generating or concatenating, plus stale storyboard_generating projects.
• Create requests are deduplicated: every AI project stores a metadata.createFingerprint (hash of the effective create parameters). An identical request from the same user inside 15 minutes resumes the existing project instead of creating another one – a still-running storyboard is returned as-is, a project that failed before any scene existed gets its storyboard retried, and a failed project that already has scenes is returned for resumption. This is what keeps a transient runtime outage during storyboard generation from producing several identical projects when the user or the agent submits the request again.
• The worker uses the same reconcileAiProject path as the project status API, including the guarded single-winner transition before ffmpeg concatenation. It logs a tick summary only when project or scene state changed.
• OpenAI-compatible/ComfyUI submissions use the generated-video UUID as both Idempotency-Key and X-Request-ID. A submit-side 504 never triggers another POST: Clapilot checks the provider by that stable key with bounded exponential backoff (three checks by default), then either persists the recovered provider task id for normal polling or marks the scene and project terminally failed with the attempt count and last cause. A 504 response that already carries a task id is treated as an accepted job.
• Provider settings can override submitReconcileEndpoint (placeholders {idempotency_key} or {idempotencyKey}), submitMaxAttempts (1–5), and submitRetryBaseDelayMs (0–60000). Without an override, reconciliation uses GET <configured video submit endpoint>/{idempotency_key}. The provider/gateway must map the supplied idempotency key to the accepted job.

Automation delivery recovery worker

• src/instrumentation.ts starts the in-process automation delivery recovery worker with the Node server, so a deploy or container restart that killed a delivery mid-flight is repaired right after boot.
• Every reservation in automation_target_deliveries carries a lease: CLAPILOT_AUTOMATION_DELIVERY_LEASE_MS=900000 (15 minutes; clamped to 1 minute–24 hours). A reserved row whose lease is older than now() is treated as abandoned by its owner.
• CLAPILOT_AUTOMATION_DELIVERY_RECOVERY_SECONDS=60 controls the sweep interval; 0 disables the worker. Each tick claims at most 25 expired rows with FOR UPDATE SKIP LOCKED and renews their lease in the same statement, so parallel web replicas and a genuine retry through the delivery routes never recover the same reservation twice.
• Decision per claimed row, recorded in recovery_outcome, recovered_at, and error_message:
  • finalized_persisted_message: a chat_nachrichten or chat_group_messages row already carries the delivery key, so the message landed and only the status write was lost. The row becomes delivered without sending anything.
  • redelivered / redelivery_duplicate: the payload snapshot stored at reservation time (delivery_payload) is delivered again with the same delivery key; the chat unique indexes make the replay idempotent. Replayed messages carry message_meta.automation_delivery_recovered = true. Only the replay itself is treated as a redelivery failure: if the final status write fails after the message was persisted, the row stays reserved and the next tick finalizes it as finalized_persisted_message.
  • no_replay_payload, redelivery_failed, recovery_attempts_exhausted (after three recovery claims), external_channel_unverifiable (approved external-channel sends are never repeated): the row becomes failed with a diagnosis naming the reservation time, lease length, and expiry. A later genuine retry can still reclaim a failed row.
• The worker logs one automation_delivery_recovery JSON line per tick that claimed rows. Verify on an instance with:

SELECT delivery_key, target_kind, status, created_at, lease_expires_at, recovery_outcome, error_message
FROM automation_target_deliveries
WHERE status = 'reserved' AND lease_expires_at <= now();

The query must return no rows once the worker has run.

Agent Orchestrator supervisor

• Migration 139_bundled_automations_and_supervisor.sql installs a bundled specialized agent and a bundled automation with bundled_key=agent-orchestrator-supervisor.
• The bundled automation controls whether the supervisor is active and how often it runs; the default interval is 30 minutes and the default row is shipped paused until an admin enables it.
• The bundled specialized agent stores the supervisor model/default prompt. The records are editable through the normal Automationen and Spezial-Agenten settings sections but cannot be deleted.
• Each supervisor pass releases stale/terminal orchestrator run locks, clears old failure backoffs, reconciles merged tracked PRs back to source GitHub issues, and wakes the normal issue/PR/task-board polling path when recovery work is needed.
• Automation completion is committed only after all configured result targets are delivered. Delivery failures retain the completed, occurrence-idempotent agent run, mark the job attempt failed, and use the normal bounded retry/dead-letter policy; retries therefore repeat delivery without replaying completed agent side effects. A failed required tool call marks the run failed with automation_required_tool_failed. An automation whose delivered final answer is an explicit failure report (first line starts with ⚠️/❌/🚨/⛔/🛑 and names a failure such as fehlgeschlagen, failed, or non riuscita) is delivered as usual, but the run flips to failed with automation_reported_failure, the job stores the headline in last_error, a warning job.completed event keeps the report, and the Team Chat message carries message_meta.ok=false; the schedule is deliberately not re-run because the agent already exhausted its own retries. Automations that require an additional internal post can call team_chat_post_message; omitting room_id uses the active Team Chat room and otherwise publishes to #general.

Admin runtime controls:

• /settings/clapilotaicore/rag includes RAG Index Health (coverage, queue, failed jobs) plus explicit document vision fallback controls
• Reindex actions: failed-only, full reindex, document-scoped, mandant-scoped
• API endpoints:
  • GET /api/admin/rag/status
  • POST /api/admin/rag/status
  • POST /api/admin/rag/reindex

Technical details:

• RAG Implementation

No space left on device

Check the workspace volume first: open Einstellungen → Speicher (/admin/storage) or ask the agent for the instance storage status. The page shows free/used space, the level, usage by source, the retention rules, and the last cleanup cycle. A PostgreSQL error such as could not extend file base/…: No space left on device means the database shares the full filesystem and needs free space immediately.

Order of operations:

1. Read the storage report. Large retention categories (orchestrator worktrees, repo caches, temp) are reclaimed automatically by the cleanup worker within one poll interval; check docker compose logs clapilot | rg instance-cleanup for removed= lines and ignored … session/job workspace path(s) notices.
2. If reclaimable data exists but nothing is removed, run one dry-run cycle to see the plan without deleting: CLAPILOT_INSTANCE_CLEANUP_DRY_RUN=true node scripts/instance-cleanup-worker.mjs inside the web container.
3. Protected categories (mandanten, generated media, work data) are never removed automatically. Deciding what to archive there is a human decision; export or move data instead of deleting it in place.
4. The Agent Orchestrator will not start new worktrees below AGENT_ORCHESTRATOR_MIN_FREE_DISK_GIB. A manual run rejected by this guard is recorded as failed and is not retried or queued; start it again once space has been freed.

If the workspace volume is healthy, the pressure usually comes from Docker root storage instead.

Safe cleanup without data loss:

docker builder prune -af
docker image prune -f
docker container prune -f

Avoid docker system prune --volumes unless you explicitly accept volume data loss.

Mailbox credential mismatch

• agent mailbox: app_settings.agent_email_* + default host/port
• user mailboxes: user_email_accounts rows (per-user, multiple accounts; company accounts use app_settings.default_imap/smtp_*, custom accounts use their own host/port)

Different answers across channels often mean different resolver path or runtime target.

Legacy compatibility device token mismatch

This recovery path was removed with the packaged OpenClaw CLI. Native session/model control now lives inside clapilot-agent; there is no local device-pairing flow to repair in the Docker image anymore.

No-data-loss backup sequence

mkdir -p ~/clapilot-backups

# workspace
docker run --rm -v clapilot_workspace:/src -v ~/clapilot-backups:/bk alpine \
  sh -lc 'tar -czf /bk/workspace-$(date +%F-%H%M).tgz -C /src .'

# native runtime state
docker run --rm -v clapilot_workspace:/src -v ~/clapilot-backups:/bk alpine \
  sh -lc 'tar -czf /bk/clapilotaicore-state-$(date +%F-%H%M).tgz -C /src .clapilotaicore'

# postgres logical dump
docker compose exec -T postgres pg_dump -U "${POSTGRES_USER:-clapilot}" "${POSTGRES_DB:-clapilot}" \
  > ~/clapilot-backups/db-$(date +%F-%H%M).sql