Operations

This page collects operational guidance for running Clapilot: container startup, local and native installs, GHCR (GitHub Container Registry) image publishing, CI browser checks, runtime diagnostics, and safe remediation for common failure modes. It is written for whoever operates an instance — self-hosted, CI, or production.

Container startup sequence

1. validate required environment
2. resolve native runtime config plus any mounted legacy compatibility state
3. seed workspace profile files
4. apply migrations (if enabled)
5. start clapilot-agent, workers, and app server

Core diagnostics

docker compose ps
docker compose logs -f clapilot
docker compose logs -f postgres
docker system df -v

Fleet connector updates

Standalone Fleet connector 1.2.0 and newer self-update from the Hub. After a successful heartbeat, an idle connector downloads the allowlisted connector and event-outbox files, verifies the advertised version and JavaScript syntax, replaces the outbox first and connector last, and exits. Its launchd KeepAlive or systemd Restart=always service then starts the new version. Set CLAPILOT_FLEET_SELF_UPDATE=0 only when updates must be managed externally.

Connectors older than 1.2.0 still require one manual reinstall with the command generated on the Hub Fleet Machines page. That reinstall is the one-time bridge to self-updating releases and now downloads both required connector files. Clapilot Remote Runner.app follows its Sparkle release channel instead of the standalone self-update path.

Local source-debug build

Use the local override when you want to rebuild clapilot from the checked-out source instead of the published GHCR image:

docker compose -f docker-compose.yml -f docker-compose.local-build.yml up -d --build clapilot

For a full local source-built stack:

docker compose -f docker-compose.yml -f docker-compose.local-build.yml up -d --build

The override switches clapilot back to build: . and keeps watchtower behind an opt-in profile.

The settings sidebar footer shows the current Clapilot web build tag, commit hash, local-vs-published build status, and the native agent build.

For the full runtime inventory, use Settings -> Versionen. That page expands the footer data into a live table with the current Clapilot web/native builds plus the installed versions and resolved paths for the directly integrated CLI tools Clapilot uses, including Codex CLI and Claude Code CLI.

Native host install CLI

For non-Docker installs on macOS or Linux, use the repository CLI instead of the dev launcher:

node scripts/clapilot-cli.mjs --help
node scripts/clapilot-cli.mjs install

If you want a global clapilot command for the checked-out repo:

npm link
clapilot --help

The installer supports both:

• interactive wizard flow
• scripted installs via flags like --env-file, --install-dir, --app-port, --agent-port, --db-port, --public-base-url, --database-url, and --yes

The GitHub-release macOS Apple client embeds the bundled host runtime and can install or control the local server directly. That client variant is built with the Clapilot GitHub Release Xcode scheme or clients/apple/ClapilotApple/scripts/package-github-release-client.sh; it is not used for Xcode Cloud/TestFlight builds. On first launch, when ~/.clapilot is missing, it installs from the bundled clapilot-runtime.tgz, creates launchd-managed web/agent services, seeds a local admin user, and then signs in without showing the normal login screen. On later launches it reads ADMIN_EMAIL, ADMIN_PASSWORD, and PORT from ~/.clapilot/.env.local, starts the managed services when needed, and signs in to the local web app.

When building the Clapilot GitHub Release scheme locally in Xcode, generate the bundled runtime first:

npm run build
npm --prefix clients/host run runtime:bundle

What the install command does:

• installs from the packaged runtime bundle when CLAPILOT_BUNDLED_RUNTIME_ARCHIVE is provided, otherwise bootstraps or reuses the target git checkout
• reads installer defaults and extra runtime vars from --env-file when provided
• writes .env.local for the native host runtime
• defaults the install path to ~/.clapilot
• installs or starts a local PostgreSQL service on supported hosts
• creates the local DB role/database when using the managed local PostgreSQL path
• runs the required runtime setup:
  • bundled runtime installs run the DB migration/admin-seed scripts directly without git clone, npm ci, or npm run build
  • source-checkout installs still run npm ci, npm run db:migrate, optional admin seed, and npm run build
• creates persistent native services:
  • Linux: systemd units
  • macOS: launchd agents
• can configure a Cloudflare tunnel by delegating to scripts/cloudflare-tunnel-setup.sh --env-file <native-env-file> when the required Cloudflare vars are present or explicitly requested
• writes install metadata under ops/native-runtime/install-meta.json

There is also a matching uninstall flow:

clapilot uninstall --install-dir ~/.clapilot --yes

That removes the native services, deletes the install directory, and drops the managed local DB objects when they belong to that install. It does not uninstall the system PostgreSQL package itself.

Example non-interactive install:

clapilot install \
  --yes \
  --env-file .env \
  --install-dir /srv/clapilot/current \
  --app-port 3000 \
  --agent-port 3211 \
  --db-port 5432 \
  --public-base-url https://app.example.com

Cloudflare tunnel env expected by the installer:

• CLOUDFLARE_API_TOKEN
• CLOUDFLARE_ACCOUNT_ID
• CLOUDFLARE_ZONE_ID
• CLOUDFLARE_TUNNEL_HOSTNAME
• optional CLOUDFLARE_TUNNEL_NAME
• optional CLOUDFLARE_TUNNEL_SERVICE (defaults to http://127.0.0.1:<app-port> for native installs)

To pull and redeploy later:

clapilot update --install-dir /srv/clapilot/current

Host app prototype

The repository also includes a desktop host shell in clients/host.

This is a separate gateway/controller app, not the existing Apple end-user client. It wraps the same native install flow so the GUI and terminal stay on one install path.

Use it when you want a local desktop control surface for:

• guided setup with defaults for a local machine
• install directory and env-file pickers
• optional Cloudflare tunnel setup through the existing repo script
• installed-state monitoring for web, agent, database, and Cloudflare
• menu bar / tray icon controls for opening settings, checking running services, and starting or stopping managed services
• host app release checks through GitHub Releases

Local development:

npm --prefix clients/host install
npm run host:dev

Bundle build:

npm run host:build

Current scope:

• macOS and Linux desktop targets
• plain Tauri shell with a static frontend
• runs the existing scripts/clapilot-cli.mjs install ... flow behind the wizard
• packaged builds include an embedded runtime archive so installs do not clone the full repo
• uses ~/.clapilot as the default install path
• shows a post-install status dashboard instead of raw command output
• keeps running in the menu bar / tray when the main window is closed
• includes a full-clean uninstall action in the dashboard
• does not yet embed PostgreSQL or cloudflared binaries inside the app bundle itself
• still depends on the underlying CLI for host-specific privilege handling

Release packaging:

• clients/host/src-tauri/resources/clapilot-runtime.tgz is generated fresh during builds and ignored by git
• .github/workflows/publish-host-release-tags.yml runs on release tags, builds the runtime archive and macOS host app on the self-hosted mac runner, signs the .app with a Developer ID Application certificate, notarizes and staples it, then uploads the signed app zip plus runtime archive as GitHub release assets
• required Actions secrets for signed releases:
  • APPLE_CERTIFICATE: base64-encoded .p12 for the Developer ID Application cert
  • APPLE_CERTIFICATE_PASSWORD
  • APPLE_SIGNING_IDENTITY
  • APPLE_API_KEY: App Store Connect API key .p8 contents
  • APPLE_API_KEY_ID
  • APPLE_API_ISSUER

Sparkle updates for GitHub-release macOS apps

The standalone Clapilot macOS client and Clapilot Remote Runner use Sparkle 2 for in-app updates. The tag-release workflow stamps both app bundles with the release version/build, embeds Sparkle.framework, signs and notarizes the apps, and generates a separate EdDSA-signed appcast for each final zip. It publishes those appcasts beside the existing zip and checksum assets in the private f1rede/Clapilot GitHub release.

End-user clients never access the private repository directly. app.clapilot.com uses the server-only GITHUB_RELEASES_TOKEN to locate recent releases, proxy the appcasts, and stream the signed zip or delta asset:

• GET /api/desktop-updates/clapilot/appcast.xml
• GET /api/desktop-updates/remote-runner/appcast.xml
• GET /api/desktop-updates/clapilot/download/<filename>
• GET /api/desktop-updates/remote-runner/download/<filename>

These endpoints are intentionally public because Sparkle calls them before any Clapilot web session exists. Configure GITHUB_RELEASES_TOKEN on the Clapilot web service as a fine-grained GitHub PAT with read access to the f1rede/Clapilot repository contents/releases. The token stays server-side and is never written into an appcast or app bundle.

The release workflow also requires the Actions secret SPARKLE_ED_PRIVATE_KEY. It is written only to a temporary runner file while generate_appcast signs the two feeds and is removed before the step exits. Both packaged Info.plists contain the matching EdDSA public key (SUPublicEDKey), the app-specific server feed URL, automatic daily checks, and the 24-hour scheduled-check interval. Treat the EdDSA key pair as durable release infrastructure: rotating the baked-in public key breaks the update chain for already-installed builds unless a deliberate key-rotation migration is shipped first.

The first Sparkle-enabled release is a bootstrap release and must be downloaded and installed manually. Once that build is installed, later signed releases can be discovered and installed from the app menu or settings/menu-bar update action.

Host-local native dev

For fast debugging without the container wrapper, the repo also ships a host-local launcher for the Next.js app plus clapilot-agent:

npm run dev:local-native

Running the same command again restarts the launcher by stopping the previous host-local instance first.

Default loopback ports:

• app: http://localhost:3100
• native runtime: http://localhost:3211

Optional: start only the PostgreSQL Compose service before launching host processes:

npm run dev:local-native:docker-db

Port overrides:

CLAPILOT_LOCAL_APP_PORT=3200 CLAPILOT_LOCAL_AGENT_PORT=3311 npm run dev:local-native

The launcher:

• reads .env.local first, then .env
• points the app to CLAPILOT_AGENT_BASE_URL=http://127.0.0.1:<agent-port>
• points clapilot-agent back to CLAPILOT_INTERNAL_BASE_URL=http://127.0.0.1:<app-port>
• uses the checked-out repository as CLAPILOT_WORKSPACE_DIR
• stores native runtime state under the local .clapilotaicore/ directory
• disables background Telegram/WhatsApp channel startup by default so the host-local runtime does not collide with a Docker runtime that is already polling those channels
• prewarms common top-level routes in the background by default, including /login, /, /chat, /dokumente, /mandanten, /emails, /aufgaben, and /calendar, so the first manual browser visit pays less cold-compile cost in next dev

Stop it explicitly with:

npm run dev:local-native:stop

In-instance developer E2E suites

When developer mode is enabled, administrators can run the isolated regression suites from the developer settings panel. Cleanup mode deletes throwaway sessions and seeded artifacts by default; keep mode retains them for debugging.

The scheduled workflow in .github/workflows/nightly-e2e.yml runs this complete catalog after its browser suite against the same temporary source-built instance. It temporarily enables Developer mode, executes every registered suite sequentially with artifact cleanup enabled, restores the previous Developer mode value, and fails if any suite does not pass. Results are stored as instance-e2e-results.json and instance-e2e-report.md and are included in the hosted nightly report. The optional CLAPILOT_CI_INSTANCE_E2E_MODEL repository variable selects an explicit model; otherwise the suites verify the instance's normal default routing.

GHCR auth for default image deploys

The default compose stack requires two GHCR auth paths:

• host Docker auth in ${HOME}/.docker/config.json for docker pull and docker compose up
• Watchtower auth in ${HOME}/.docker/watchtower-config.json

Populate the host Docker auth with:

mkdir -p "${HOME}/.docker"
echo "<GHCR_READ_PACKAGES_TOKEN>" | docker login ghcr.io -u "<github-username>" --password-stdin

The Watchtower file should contain inline registry auth for ghcr.io, for example:

{
  "auths": {
    "ghcr.io": {
      "auth": "<base64(GITHUB_USERNAME:GHCR_READ_PACKAGES_TOKEN)>"
    }
  }
}

Requirements:

• the token needs read:packages
• use ${HOME}/.docker/..., not ~/.docker/..., in Compose-backed paths
• docker login ghcr.io ... only solves host pulls; it does not configure Watchtower
• Watchtower needs a separate mounted file because host Docker configs commonly rely on credsStore, which is not available inside the Watchtower container
• on another Mac, create both files in that user's home directory or change the bind mount in docker-compose.yml
• the local source-build override in docker-compose.local-build.yml does not require this file unless you explicitly enable the watchtower profile

Container publishing

The repository includes a GitHub Actions workflow at .github/workflows/publish-container-main.yml. It publishes the runtime image to GHCR on every push to main from the self-hosted mac-mini-m4-pro (macOS/ARM64) runner.

Published tags:

• ghcr.io/<owner>/<repo>:latest-arm64
• ghcr.io/<owner>/<repo>:main-arm64
• ghcr.io/<owner>/<repo>:sha-<full-commit-sha>-arm64

When an immutable release container package is needed, run .github/workflows/publish-container-release-tags.yml manually with the release git tag. The workflow waits for the main image with the same commit SHA and retags that image into a separate release package:

• ghcr.io/<owner>/<repo>-release:<git-tag>-arm64
• ghcr.io/<owner>/<repo>-release:release-latest-arm64
• ghcr.io/<owner>/<repo>-release:sha-<full-commit-sha>-arm64

For manual branch or ref builds, .github/workflows/deploy-feature-test-build.yml publishes a single rolling feature/test image:

• ghcr.io/<owner>/clapilot-test:test-latest-arm64

Operational notes:

• GitHub Actions needs packages: write permission for the job
• repository/package settings must allow GHCR publishing
• Watchtower on Apple Silicon hosts should follow the moving latest-arm64 tag
• release-tag container publishing is manual because tag-only Apple/client releases may not produce a new runtime image
• private GHCR packages require registry credentials on the Docker host
• the default local image-based compose stack expects those credentials in ${HOME}/.docker/watchtower-config.json

Pull request live browser check

Pull requests from branches in this repository now also run a self-hosted macOS/ARM64 browser-smoke job in .github/workflows/ci.yml.

That job:

• boots the local source-built compose stack with docker-compose.local-build.yml
• writes a CI-only env file from GitHub Actions variables and secrets
• waits for the local web app to come up on a runner-selected loopback port
• runs codex exec against the checked-out PR and requires the runner to have the chrome-devtools MCP configured
• builds a GitHub PR context file from the PR title/body, linked closing issues, and changed files
• logs in as the CI admin and rebuilds a deterministic preview demo dataset so the app is not empty during the browser run
• asks Codex to use that PR context plus the git diff to choose the most relevant feature or bugfix flow to smoke-test
• extracts the PR body's ## Testing Instructions section and treats concrete browser-relevant checks there as the primary smoke-test target
• treats docs/spec changes as non-targeting noise when non-doc product/runtime files also changed
• only targets /docs when the PR is docs-only or specifically changes the docs UI/route
• saves as many ordered screenshots as needed for the tested flow, plus a Codex markdown report, Docker logs, and an .mp4 slideshow artifact built from the captured frames
• updates a sticky PR comment with the result and the artifact bundle link
• keeps the PR Docker preview alive while the PR is open, cleans it on PR close, and uses the hourly/manual PR preview sweeper as a backup for missed close events

The iOS/Mac App Tests workflow builds the same PR context file for Apple-client PRs. Its Codex E2E prompt reads the extracted ## Testing Instructions section when present and captures any additional Apple-client screens called out there alongside the standard native smoke flow.

Runner requirements:

• self-hosted labels: self-hosted, macOS, ARM64
• local tools installed: codex, Docker with Compose v2, ffmpeg, node/npx
• a local Chrome-family browser available at least as Google Chrome
• for Docker on macOS runners, prefer a dedicated runner DOCKER_CONFIG without credsStore or keychain helpers
• codex authenticated on the runner with subscription auth; OPENAI_API_KEY is reserved for the preview app unless CI_CODEX_AUTH_MODE=api_key is explicitly set

CI env contract:

• required secret: CLAPILOT_CI_AUTH_SECRET (falls back to AUTH_SECRET if you intentionally mirror it there)
• optional secret: CLAPILOT_CI_ADMIN_PASSWORD
• optional variable: CLAPILOT_CI_ADMIN_EMAIL
• optional multiline variable: CLAPILOT_CI_ENV_VARS
• optional multiline secret: CLAPILOT_CI_ENV_SECRETS
• fallback multiline secret already used elsewhere in this repo: CLOUD_RUN_ENV_VARS
• optional variable: CLAPILOT_CI_DEMO_SEED (true by default)
• optional variable: CLAPILOT_CI_DEMO_SEED_SCENARIO (defaults to the app's default demo scenario)
• optional variable: CLAPILOT_CI_OPENAI_MODELS for the native OpenAI provider catalog shown only in preview/demo instances
• optional variable: CLAPILOT_CI_EMAIL_ANALYSIS_MODEL for lightweight mail/document analysis defaults in preview/demo instances
• optional variable: CI_CODEX_AUTH_MODE for the Codex test runner auth source (subscription by default, api_key only for explicit fallback)
• optional variable: CI_CODEX_MODEL for the Codex test runner model (gpt-5.6-sol by default)
• optional variable: CI_CODEX_REASONING_EFFORT for the Codex test runner reasoning level (medium by default)
• optional variable: CLAPILOT_CI_TEST_EMAIL_ADDRESS
• optional secret: CLAPILOT_CI_TEST_EMAIL_PASSWORD
• optional variables: CLAPILOT_CI_TEST_IMAP_HOST, CLAPILOT_CI_TEST_IMAP_PORT, CLAPILOT_CI_TEST_SMTP_HOST, CLAPILOT_CI_TEST_SMTP_PORT
• optional hosted upload API variable: LIVE_CHECK_UPLOAD_API_URL
• optional hosted upload API secret: LIVE_CHECK_UPLOAD_TOKEN
• optional hosted publish variables: LIVE_CHECK_SHARE_HOST, LIVE_CHECK_SHARE_USER, LIVE_CHECK_SHARE_REMOTE_PATH, LIVE_CHECK_SHARE_BASE_URL
• optional hosted publish secrets: LIVE_CHECK_SHARE_SSH_KEY, LIVE_CHECK_SHARE_KNOWN_HOSTS
• PR preview cleanup/sweeper variables/secrets: CLAPILOT_PREVIEW_RUNNER_LABELS, optional CLAPILOT_PREVIEW_CLEANUP_RUNNER_LABELS, CLOUDFLARE_API_TOKEN, CLOUDFLARE_ACCOUNT_ID, CLOUDFLARE_ZONE_ID, CLOUDFLARE_PREVIEW_DOMAIN

The live browser seed reuses that test mailbox for both the agent mailbox defaults and the seeded admin user's personal mailbox profile, so /emails and background agent mail actions can both run against the same non-production account.

For new self-hosted macOS runners, bootstrap Docker once with scripts/ci/setup-runner-docker-config.sh. That script writes a clean config such as ${HOME}/.docker-ci/config.json with no credsStore, so GitHub Actions jobs do not depend on an interactive login keychain session.

Install the machine dependencies first with scripts/ci/setup-macos-runner-deps.sh. It installs codex, ffmpeg, node, and Google Chrome via Homebrew, which covers the prerequisites used by the self-hosted browser jobs.

Recommended runner setup:

• run scripts/ci/setup-macos-runner-deps.sh
• authenticate codex login for the runner user
• run scripts/ci/setup-runner-docker-config.sh
• set the runner service environment to DOCKER_CONFIG=${HOME}/.docker-ci
• if the runner needs registry auth, log in with docker --config "${HOME}/.docker-ci" login ...
• the live-browser workflow bootstraps the chrome-devtools MCP itself with npx; no permanent MCP config is required on the machine
• do not rely on Docker Desktop's default keychain-backed credential store for headless runner jobs

The browser-oriented self-hosted workflows also run scripts/ci/setup-macos-runner-deps.sh before tool verification, so a newly provisioned runner can install missing node/npx, ffmpeg, codex, or Google Chrome automatically instead of failing immediately.

Use the multiline env entries for extra compose env such as provider keys or feature flags that should only exist in the live-browser CI run.

If the hosted upload API config is present, the workflow uploads the captured frames, report, gallery page, and optional .mp4 over HTTPS and embeds the public frame URLs into the PR comment.

If the upload API is not configured but the SFTP-based hosted publish config is present, the workflow falls back to SFTP publishing.

E2E performance gate

The live-browser E2E runner executes node scripts/perf-radar.mjs --gate --smoke inside the running clapilot container after a successful browser suite. The historical gate reads terminal agent_runs for the current Berlin calendar day, so one successful chat turn is one latency sample even when tool loops or provider fallback produced several agent_model_request_logs. It performs one active authenticated /api/chat smoke turn in a fresh, titled, disposable chat session. The fixed title prevents the probe from starting an unrelated model-backed auto-title request. Only a smoke request carrying the app's internal secret is tagged as ci_perf_gate and excluded from the historical rows; a caller-controlled page context alone cannot exclude traffic. A smoke turn passes only when its response ends with a fresh per-run verification token, its exact runtime session has a completed ci_perf_gate run created after the probe began, and the same run has a completed provider request log. This correlation prevents an HTTP-200 error stream or an unpersisted runtime failure from being reported as successful. The runner always writes perf-gate-report.txt into the E2E artifact directory, including an explanation when the gate is intentionally skipped.

The default gate thresholds are configurable through CI environment variables:

• PERF_MAX_FAIL_PCT: maximum daily terminal agent-run failure rate, default 10.
• PERF_MAX_CHAT_P90_S: interactive chat p90 SLO in seconds, default 90. This initial unfiltered target should be tightened only after enough post-rollout samples establish the new baseline.
• PERF_MAX_CHAT_P90_REGRESSION_PCT: maximum increase over the average p90 of preceding days that meet the minimum request count, default 20. The gate fails on either the absolute SLO or the relative regression.
• PERF_MIN_REQS_FOR_GATE: minimum daily count of successful chat turns before historical thresholds fail the run, default 15.
• PERF_SMOKE_TIMEOUT_S: per-request timeout for login, session setup, and the streamed smoke response; defaults to at least 120 seconds.
• PERF_SMOKE_MESSAGE: optional main-chat smoke prompt. It is forwarded from either the Compose/CI environment or the workflow host. It must not contain an @handle; the radar rejects such prompts before login so the probe cannot enter specialist-agent routing. The radar appends its own unique verification-token instruction to custom prompts.

These variables may be supplied through the CI container env file or exported by the workflow host. Empty values fall back to the defaults. When active smoke is enabled, missing smoke credentials fail --gate instead of silently skipping the probe.

Normal native one-to-one chat turns are intentionally persisted with source_type=chat, so they participate in the chat p50/p90 latency gate. This can shift the latency baseline compared with older builds where some native rows had a null source type. The daily failure-rate population does not change for those rows because it already counts every non-ci_perf_gate request regardless of source type. Treat the first full day after rollout as the new latency baseline and change thresholds only when the production SLO calls for it.

The radar does not discard turns longer than five minutes. It reports the ten slowest successful chat turns with a hashed run ID and an end-to-end split from agent_chat_turn_phase_metrics: queue time, provider/orchestration time outside tools, tool execution, failed provider/fallback time, and finalization after the last provider or tool event. The five phases add up to the complete agent_runs.created_at to completed_at duration (apart from millisecond rounding). Provider/model, hashed session, estimated prompt-size, and initial versus post-tool request breakdowns remain diagnostic attempt-level views. Use --compare-cutover <ISO timestamp> --compare-sample-size <N> to compare the last N successful chat turns before a deployment with the first N after it; when either cohort is short, the report shows the available counts instead of silently omitting the comparison. This is the canonical before/after measurement for latency changes.

The 29 July 2026 investigation showed that final run status alone was not a useful latency signal. In the available Top-10 sample, seven turns spent more than 100 seconds end to end. Six of those repeatedly crossed provider/tool boundaries (11-46 recorded tool starts), while one 297-second turn spent about 253 seconds in the queue before a 45-second execution. No failed provider attempt was needed for most of these successful outliers. This is why the gate uses one completed run per sample and why the Top-10 phase report must be reviewed even when the run failure rate remains below its threshold.

OpenAI-compatible streaming requests have a time-to-first-response and stream-idle deadline controlled by CLAPILOT_PROVIDER_REQUEST_TIMEOUT_MS (default 120000). Data events and SSE comment heartbeats refresh the deadline, so healthy long generations are not aborted by total wall-clock duration. Non-streaming Responses and chat-completions calls use the separate wall-clock cap CLAPILOT_PROVIDER_NON_STREAMING_REQUEST_TIMEOUT_MS (default 300000) because they cannot expose progress heartbeats. Expiry before user-visible output enters configured provider fallback/circuit handling without replaying the same slow request; expiry after output terminates the stalled stream because replaying it would duplicate visible content. Other short-lived transient failures retain the bounded retry policy controlled by CLAPILOT_PROVIDER_TIMEOUT_MAX_ATTEMPTS. OpenAI and Azure OpenAI streaming chat-completions requests send stream_options.include_usage=true. OpenAI-compatible servers receive the same standard field only when the existing compatible prompt-cache opt-in is enabled, preserving the strict-server fallback. Request-log metadata records timeToFirstTokenMs from the first meaningful reasoning, text, or tool-call delta; missing usage remains NULL rather than being recorded as a misleading zero. The same guards cover OpenAI Responses API requests.

Direct human-facing chat turns intentionally have no end-to-end runtime deadline. A model/tool loop may therefore continue for as long as the task requires; only an explicit caller-owned timeout can cap the whole conversation. The stdio MCP bridge and Codex MCP rescue path likewise add no independent wall-clock deadline around /api/agent-runtime/tool-proxy; the retired CLAPILOT_MCP_TOOL_PROXY_TIMEOUT_MS and CLAPILOT_MCP_IMAGE_TOOL_PROXY_TIMEOUT_MS settings are ignored. Independent provider-stall, native tool-phase and fallback safeguards default to 120, 120 and 45 seconds and are never clipped by elapsed turn time. A chat turn queued behind a long turn in the same session survives up to CLAPILOT_CHAT_QUEUE_BUDGET_MS=300000 before it is failed (and the failure is persisted as a failed run). The paid mixture-of-agents reference fan-out is exempt from the preflight cutoff, and finalization overruns are logged warnings that never fail an already-successful run. Long tools emit tool.progress every 30 seconds. These phase safeguards never apply to jobs, automations, channel runs, or orchestrator turns; long-running background work is first-class. Phase defaults and their environment variables are documented in .env.example. Repository Orchestrator turns are reported separately and keep the 45-minute broker timeout (CLAPILOT_AGENT_TURN_TIMEOUT_MS=2700000, 10-minute floor); an expired orchestrator turn is actively interrupted on the app-server (turn/interrupt) instead of being left running detached. The Performance Radar computes user-visible latency from agent_runs, not individual model requests, and prints separate direct_chat, orchestrator, and automation_channel_api cohorts. Its slow-run section joins agent_events to expose queue/preflight, provider, tool and finalization timing for the top outliers.

Provider fallbacks are failure-domain aware. The provider slug is the failure-domain key, so a connection failure or provider stall quarantines that gateway for the current request and skips later models on the same slug. The remaining fallback budget is divided across the still-reachable independent provider slugs, preserving time for the final emergency provider. Provider-attempt telemetry records skipped_failure_domain candidates and remainingFallbackBudgetMs. Saving runtime provider configuration returns a warning when a configured fallback chain contains models but no enabled independent provider slug.

Chat phase budgets can be overridden per provider and per model in runtime config instead of environment variables. The provider edit dialog (settings → ClapilotAICore → providers) exposes dedicated second-based fields under "Timeouts & latency budgets" for model response without output, tool execution, queue wait, fallback budget, and the background stall timeout; empty fields keep the system defaults. The fields persist to chatLatencyBudgets in the provider's metadata as an object with any of queueMs, preflightMs, providerMs, toolMs, fallbackMs, finalizationMs; totalMs from older configurations is ignored and cannot restore a whole-conversation deadline. Add modelChatLatencyBudgets keyed by model name (advanced metadata JSON) for model-specific values that win over the provider-level object. Example for a slow self-hosted gateway: {"chatLatencyBudgets": {"providerMs": 300000, "queueMs": 900000}}. Model overrides beat provider overrides, which beat the CLAPILOT_CHAT_* environment defaults; the environment variables remain global fallbacks only. Because the queue check for a waiting turn runs before provider selection, it uses the budgets resolved by the previous turn in the same session. Background workloads (jobs, automations, channels) stay unbudgeted, but their provider stall timeout can be raised per provider with metadata providerRequestTimeoutMs (optionally modelProviderRequestTimeoutMs keyed by model) when a self-hosted model needs more than the default 120 seconds before its first streamed token.

OpenAI-compatible providers can also opt into per-model request scheduling through provider metadata. Set maxConcurrentRequests to cap simultaneous upstream requests and maxBackgroundConcurrentRequests to reserve the remaining capacity for interactive chat. Interactive requests are always dequeued before background automation. The scheduler is disabled when these metadata fields are absent, and request logs record queue wait, priority, and the active limits for diagnosis.

Set CI_PERF_GATE_ENABLED=false only for emergency runner diagnostics. The active smoke remains useful when the day has too little traffic for the historical minimum, because smoke failures and smoke latency breaches are evaluated independently.

Agent runtime console (admin UI)

For Cloud Run setups without container shell access, use:

• the admin runtime console in Clapilot settings

Built-in diagnostics:

• help
• ls-state-dir
• show-clapilotaicore-json
• validate-clapilotaicore-json
• node-version
• db-host-check

Notes:

• access is admin-only (same role check as other admin APIs)
• the packaged OpenClaw CLI is no longer present in the Docker image
• this is intentionally not a full unrestricted OS shell
• use it to diagnose native runtime connectivity and state remotely

RAG runtime controls

RAG (retrieval-augmented generation) runtime toggles (container env):

• RAG_ENABLED=true enables retrieval injection in /api/chat (default: enabled)
• RAG_INDEXER_ENABLED=true enables background indexing worker (default: enabled)
• RAG_RETRIEVAL_TOP_K and RAG_RETRIEVAL_MIN_SCORE tune retrieval breadth/quality
• RAG_CHUNK_SIZE and RAG_CHUNK_OVERLAP tune chunking granularity
• embeddings key source: OPENAI_API_KEY or app_settings.openai_api_key

RAG diagnostics:

docker compose logs -f clapilot | rg -n "rag-indexer|chat-rag"

Instance cleanup worker

• CLAPILOT_INSTANCE_CLEANUP_ENABLED=true starts the web-container cleanup worker (default: enabled)
• CLAPILOT_INSTANCE_CLEANUP_POLL_SECONDS=1800 controls the cleanup interval
• CLAPILOT_INSTANCE_CLEANUP_MIN_FREE_GB=20 and CLAPILOT_INSTANCE_CLEANUP_TARGET_FREE_GB=40 define the disk-pressure window
• CLAPILOT_INSTANCE_CLEANUP_MAX_DELETE_GB=25 caps deletion per cycle
• CLAPILOT_INSTANCE_CLEANUP_DRY_RUN=true logs planned deletions without removing files
• The worker only deletes allowlisted instance scratch paths such as old Agent Orchestrator PR-review, issue, tracked-PR, manual-job, and workspace temp directories. It preserves active agent_external_sessions, queued/running orchestrator jobs, mandanten, and .clapilotaicore.

Video Studio project reconciler

• src/instrumentation.ts starts the in-process Video Studio reconciler with the Node server.
• CLAPILOT_VIDEO_STUDIO_RECONCILE_SECONDS=60 controls the interval; 0 disables it.
• Each non-overlapping tick checks at most ten owner-fair active projects in generating or concatenating, plus stale storyboard_generating projects.
• The worker uses the same reconcileAiProject path as the project status API, including the guarded single-winner transition before ffmpeg concatenation. It logs a tick summary only when project or scene state changed.

Agent Orchestrator supervisor

• Migration 139_bundled_automations_and_supervisor.sql installs a bundled specialized agent and a bundled automation with bundled_key=agent-orchestrator-supervisor.
• The bundled automation controls whether the supervisor is active and how often it runs; the default interval is 30 minutes and the default row is shipped paused until an admin enables it.
• The bundled specialized agent stores the supervisor model/default prompt. The records are editable through the normal Automationen and Spezial-Agenten settings sections but cannot be deleted.
• Each supervisor pass releases stale/terminal orchestrator run locks, clears old failure backoffs, reconciles merged tracked PRs back to source GitHub issues, and wakes the normal issue/PR/task-board polling path when recovery work is needed.

Admin runtime controls:

• /settings/clapilotaicore/rag includes RAG Index Health (coverage, queue, failed jobs) plus explicit document vision fallback controls
• Reindex actions: failed-only, full reindex, document-scoped, mandant-scoped
• API endpoints:
  • GET /api/admin/rag/status
  • POST /api/admin/rag/status
  • POST /api/admin/rag/reindex

Technical details:

• RAG Implementation

No space left on device

Usually this means Docker root storage is full, not the entire host disk.

Safe cleanup without data loss:

docker builder prune -af
docker image prune -f
docker container prune -f

Avoid docker system prune --volumes unless you explicitly accept volume data loss.

Mailbox credential mismatch

• agent mailbox: app_settings.agent_email_* + default host/port
• user mailbox: user_profiles.kanzlei_email(+password)

Different answers across channels often mean different resolver path or runtime target.

Legacy compatibility device token mismatch

This recovery path was removed with the packaged OpenClaw CLI. Native session/model control now lives inside clapilot-agent; there is no local device-pairing flow to repair in the Docker image anymore.

No-data-loss backup sequence

mkdir -p ~/clapilot-backups

# workspace
docker run --rm -v clapilot_workspace:/src -v ~/clapilot-backups:/bk alpine \
  sh -lc 'tar -czf /bk/workspace-$(date +%F-%H%M).tgz -C /src .'

# native runtime state
docker run --rm -v clapilot_workspace:/src -v ~/clapilot-backups:/bk alpine \
  sh -lc 'tar -czf /bk/clapilotaicore-state-$(date +%F-%H%M).tgz -C /src .clapilotaicore'

# postgres logical dump
docker compose exec -T postgres pg_dump -U "${POSTGRES_USER:-clapilot}" "${POSTGRES_DB:-clapilot}" \
  > ~/clapilot-backups/db-$(date +%F-%H%M).sql