Agent Orchestrator

Regression tasks whose description contains an exact session.*.failed production signature are not marked erledigt immediately after an implementation run. Symphony moves them to warten and persistently waits for the linked pull request to be merged. The production observation window starts at that merge timestamp, lasts 24 hours by default (SYMPHONY_REGRESSION_OBSERVATION_MS), and survives orchestrator restarts through the durable job record. Jobs with pending observations are protected from the 12-hour job TTL cleanup. Symphony only completes the task when the signature stays absent for the full post-merge window. A recurrence or an unmerged PR closure returns the task to offen.

What it does

Agent Orchestrator (agent-orchestrator) runs coding agents against Git repositories and monitors them from one place. It starts detached coding-agent jobs (codex, claude, clapilot-code), hosts interactive agent sessions backed by the native ClapilotAICore broker, and keeps a small compatibility fallback for older job paths. Clapilot-code is Clapilot's own in-process coding loop; Codex and Claude remain external harnesses. GitHub and GitLab repositories share the same repository picker and automation matrix. The runtime reviews pull requests or merge requests, implements issues on a fresh branch, replies to mentions, follows review feedback, and fixes default-branch CI failures.

How to open / enable it

• Agent Orchestrator is a Developer-mode module: the web module and the Settings -> Agent Orchestrator surface are hidden while Developer mode is disabled (src/lib/module-store/developer-mode-modules.ts).
• With Developer mode enabled, open it from the module menu; the route is /modules/agent-orchestrator. GitHub automations and runtime settings live under Settings -> Agent Orchestrator (src/app/(app)/(einstellungen)/settings/agent-orchestrator/page.tsx).
• Manifest: bundled-modules/agent-orchestrator/module.json (slug agent-orchestrator, entry index.html, renderer react, icon bot). The React UI is src/components/modules/agent-orchestrator-module.tsx, hosted by src/app/(app)/modules/[slug]/page.tsx; bundled frontend assets remain in bundled-modules/agent-orchestrator/ but the app renders the module directly in the main React tree.
• Auth for the module API: web users authenticate with the clapilot_session cookie; legacy compatibility skills/scripts use a short-lived machine Bearer token from POST /api/auth/agent/system-token; remote runners use Hub HMAC auth with the shared CLAPILOT_HUB_SHARED_SECRET.

Key workflows

Start a detached coding job

Use jobs when the user wants a detached run such as "go implement this", "prepare a PR", or "review this repo".

• + New Agent opens a compact chat-style composer. With Local selected it creates an interactive workspace session; its footer contains attachment, repository, icon-prefixed model, optional speech-to-text, and send controls. The model menu combines available Codex, Claude, and Clapilot Code models and infers the harness from the selected model.
• When at least one online remote runner is available, the composer adds a runner selector between repository and model (web and Apple clients). Auto lets any online runner claim the work, while choosing a named runner pins it to that machine. A remote selection limits the model menu to the harnesses the eligible runners advertise (Codex, plus Clapilot Code on capable runners), disables attachments, and creates a decoupled coding job instead of an interactive session; the queued job remains pinned even if that runner later goes offline.
• The Apple client's Neuer Agent surface mirrors the same session-first contract and single-row footer. It hides the generic composer emoji and keyboard-hint controls so repository, runner, and model selection remain visible without a second selector row. On macOS and iPad the Neuer Agent composer opens inline in the right detail pane (like the web module's right pane) instead of a separate sheet or window; iPhone keeps the full-screen create view.
• Detached and remote jobs remain supported for the web New Agent composer, automations, GitHub observers, remote runners, API/tool calls, existing activity rows, and job follow-ups.
• Codex-backed detached jobs keep an internal linked orchestrator session, so the same background run can be resumed later with a follow-up prompt from the web module, the native Apple client, or agent tools. Plain CLI jobs without a linked session, including Claude CLI jobs, can also receive follow-up prompts; those run in the original job workspace and append their output to the same job log. Remote Codex jobs requeue follow-up prompts for the remote runner and reuse the remote job workspace.
• Follow-ups can be sent while a job is still running. Session-backed jobs queue the turn inside the linked session runtime; remote and plain CLI jobs store the prompt in a per-job pending queue (max 20, persisted in jobs.json) and return 202 with turn.status: "queued". The prompt appears in the transcript immediately, serialized jobs expose the pending count as queuedFollowUps, and queued entries run in order once the current run finishes with succeeded or failed (a user-initiated stop pauses the queue until the next explicit follow-up). The web composer stays enabled while running and shows a subtle "runs after the current run" indicator under the input.

Model selection

• provider=clapilot-code runs through Clapilot's in-process coding loop instead of the Codex app-server or Claude CLI. It reuses the native embedded_pi adapter internally. The web creation UI shows a model picker backed only by usable non-subscription entries in the configured Provider & Modelle catalog; session creation fails clearly when none are available instead of silently falling back.
• pi, embedded_pi, embedded-pi, and clapilot_code remain accepted input aliases for clapilot-code. New jobs, session metadata, and automation settings use the canonical clapilot-code provider value.
• The web Neuer Agent panel shows model pickers for claude and codex too, backed by GET /coding-models. The picker preselects the model a job would run with by default (runtime config override, then the provider catalog default). Options merge the configured agent_provider_configs catalog with the current CLI-native model lineup, so models the CLI accepts stay selectable even when the Clapilot catalog only lists a subset.
• The current Codex lineup includes the pre-release gpt-6-astra, gpt-5.6-sol, gpt-5.6-terra, gpt-5.6-luna, gpt-5.5, gpt-5.4, gpt-5.4-mini, and the subscription-only gpt-5.3-codex-spark. Spark is a current selectable model and is not normalized through the legacy GPT-5.3 fallback rule.
• Detached Claude jobs always resolve to a concrete model: an explicit request wins, then the Settings -> Agent Orchestrator Claude runtime model, then the configured Anthropic provider catalog default. The current built-in fallback is claude-opus-5; the resolved ref (for example claude-default/claude-opus-5) is stored on the job and shown on activity cards, matching interactive sessions.
• Claude CLI jobs run with --output-format stream-json --verbose --include-partial-messages, so the job log streams live assistant text and tool activity while the job runs, and the job's model field is corrected to the model the CLI actually reports in its init event. Claude tool use is stored as the same structured job-log items as Codex and Clapilot Code (command_execution, file_change, web_search, mcp_tool_call, dynamic_tool_call); tool results complete the matching item with output and failed/completed status, and partial assistant text streams to live SSE viewers as transient item.updated events while only the finished message is persisted. Older persisted [tool] … text lines keep rendering through the legacy fallback.
• The job follow-up composer (web terminal panel and Apple job detail) has a model picker limited to the running job's provider catalog, preselected to the model the job last ran with. The chosen model is sent as model on POST /jobs/{id}/follow-up; detached CLI follow-ups re-run the CLI with that model, linked-session follow-ups forward it as a per-turn model.
• The Codex app-server bridge resolves the Codex service tier for each model. Codex-auth OpenAI providers can set a per-model Codex-Tier (Global, Fast, Flex, Aus) in ClapilotAICore settings; these values are stored as provider metadata under modelServiceTiers. The model picker reads Codex app-server model/list and uses the returned speed-tier metadata for the per-model tier menu. Models left on Global use CLAPILOT_AGENT_CODEX_SERVICE_TIER when it is set; otherwise the tier stays unset and Codex chooses.
• Agent Orchestrator runtime settings accept max and ultra reasoning in addition to the older effort values. Codex currently advertises max for the GPT-5.6 family and ultra for Sol and Terra; unsupported model/effort combinations remain subject to the Codex runtime's own validation.

Run interactive sessions

Use sessions when the user wants to stay in the same thread/workspace and continue iteratively:

• resumable repo conversations
• thread-bound Telegram/Slack/WhatsApp coding sessions
• live turn + terminal streaming
• fork/archive lifecycle

Interactive session adapters remain codex_app_server, acp_agent, and embedded_pi. provider=auto resolves to Codex unless an explicit adapter override is provided, provider=clapilot-code defaults to the internal adapter=embedded_pi, and an explicit provider=claude defaults to adapter=acp_agent. The acp_agent adapter remains scaffolded for local ACP-compatible sidecars/agents and is still experimental. The web + New Agent panel offers interactive execution for the production-ready Codex and Clapilot-code paths.

Monitor work in the module

The React module shows detached jobs, interactive sessions, and native runtime sessions from ClapilotAICore so admins can inspect normal Assistant chats, Clapilot-code coding sessions, and bridge runtime activity without switching to the dedicated Sessions diagnostics page.

The activity rail uses a canonical read-only thread projection over the existing job, orchestrator-session, native runtime-session, and remote-session records. The projection classifies each source as chat, coding, scheduled, system, or remote, preserves its original session/chat/job/external identifiers, and advertises capabilities such as conversation inspection, reply, diagnostics, workspace files, diffs, terminal, approvals, checkpoints, and Git actions. It does not migrate or replace stored sessions.

Normal web, group, and channel chats remain first-class entries in Sessions. Their persisted chatTranscript stays the authoritative conversation, while model runs, tool events, memory/context diagnostics, raw events, and metadata remain available in the detail inspector. Coding-only workspace capabilities are enabled only for repository/workspace-backed projections, so later Changes, Files, Terminal, Preview, and checkpoint panels cannot replace or hide ordinary chat history.

The Apple client applies the same projection on iPhone, iPad, and macOS. Selecting a native runtime chat loads its full persisted transcript instead of stopping at the list summary; supported chats can continue through the existing runtime-session turn endpoint, while fork, close, terminal, diff, checkpoint, and Git actions remain limited to session types that actually support them.

• The left rail is a dense, searchable activity navigator with counted filters for Coding Agents, Sessions, Alle, Remote Agents, and Analytics. Each row keeps category, provider, repository/session identity, title, latest output, resolved model, status, latest tool use, and update time scannable without competing with the selected-thread detail pane.
• Selected-thread headers lead with the human-readable session/channel title and retain the technical session id as secondary metadata. This keeps normal chat inspection understandable without removing diagnostic identity.
• Detached-job details use the same compact identity hierarchy as session details: task title, repository/model context, provider mark, status dot, and an optional stop action. A segmented detail control separates Conversation, Activity, Terminal, Changes, and Files; changed files and diffs never render inline in chat history. Do not add a Terminal/Prompt switch, a second "Current Terminal" title, explanatory subtitle, status badge, or always-visible job/workspace metadata row above it. The original task remains visible as a user message.
• Coding Jobs shows detached coding jobs plus repository-backed interactive and runtime sessions, so coding work stays grouped with repository context.
• Session Jobs bundles chat- and runtime-backed session work: normal Assistant sessions (runtimePath=native), native Clapilot-code coding sessions (runtimePath=embedded_pi), and native sessions whose active execution harness resolves to the Claude CLI Bridge. Hidden follow-up sessions that belong to detached coding jobs are intentionally excluded; they stay attached to their owning Coding Job detail view.
• Alle mixes session-backed work and detached coding jobs in one rail.
• Session overview requests are summary-only by default: they carry compact status/model/activity previews but omit per-session event and chat-history arrays. The web module refreshes visible overview lists every 30 seconds, skips hidden-tab refreshes, and suppresses overlapping requests. Full history is loaded only for the selected session.
• Remote Agents switches the rail to a remote-machine dropdown, then lists that machine's local Codex CLI and clapilot-code sessions from the latest runner heartbeat. Clapilot Code rows carry a harness label. Selecting a remote session opens its metadata and transcript once the runner has answered the detail request in a later heartbeat. A composer at the bottom creates a decoupled resume job pinned to that machine, so replies queue even while the runner is stale or offline and continue through the matching CLI harness when a compatible runner reconnects.
• Analytics opens the internal Agent Costs & Yield dashboard for repo-bound Orchestrator runs. The first MVP derives run/yield data from existing jobs and interactive sessions, including repo, issue/PR context when detectable, provider/model, status category, duration, and simple waste hints such as failed runs or completed runs without a detected PR. Cost and token fields are explicitly marked as unavailable until reliable provider usage data exists; this analytics view does not render raw logs.

Detail-pane behavior:

• Session-backed details preserve each data mode without changing the underlying records. Conversation is the default and shows the chronological user/assistant dialog plus compact inline tool and command rows (command_execution, file_change, mcp_tool_call, dynamic_tool_call, web_search) rendered with the same collapsible tool treatment as Activity — raw terminal output, turn markers, and other meta/system noise stay out of Conversation. Activity isolates tool calls, terminal output, model runs, and lifecycle events, and Details exposes identifiers, runtime metadata, and diagnostics. The same conversation contract applies to detached jobs and session-backed jobs. Local repository-backed coding sessions additionally expose capability-gated Terminal, Changes, and Files modes. Terminal isolates command cards and raw terminal deltas already emitted by the agent runtime; Changes shows the current branch, Git status, staged diff, and working-tree diff; Files shows a bounded Git-ignore-aware relative inventory. Normal chats never receive these workspace modes.
• Workspace inspection is lazy-loaded from GET /sessions/{id}/workspace or GET /jobs/{id}/workspace. Each local job records its starting Git commit; Changes compares the current checkout against that baseline, so committed and pushed agent edits remain visible instead of appearing clean. Before an automation checkout is removed, the Orchestrator persists an immutable final workspace snapshot and serves it to both the owning job and linked session. Historical PR jobs created before snapshot support can recover a read-only changed-file view from the GitHub pull-request files API. Ownership and trusted-root checks still match workspace file references, and remote-runner workspaces remain unavailable. The Terminal inspector is read-only: it reuses the selected session's persisted and SSE-streamed command/terminal events and does not expose the unrestricted Developer-mode admin terminal or introduce an arbitrary-shell endpoint.
• Local coding sessions expose an Approvals mode on web and Apple. Codex app-server runs use approvalPolicy=on-request, approvalsReviewer=user, and workspace-write; command and file-change server requests remain pending in the native broker until the owner explicitly approves or declines them through GET/POST /sessions/{id}/approvals. The same mode can create bounded read-only workspace checkpoints through GET/POST /sessions/{id}/checkpoints; checkpoint data is stored in Clapilot runtime state outside the repository and never mutates or rolls back the worktree. Preview, checkpoint restore, and mutating Git actions remain unavailable until their separate guarded contracts are implemented.
• The Changes inspector exposes guarded Git actions through POST /sessions/{id}/git-actions and the equivalent job endpoint. Stage/unstage accepts one validated repository-relative path, commit accepts only a bounded message and requires staged changes, and push is fixed to a non-force origin push of the current branch. Every action rechecks the owner/trusted workspace plus the branch and HEAD the user reviewed; stale requests are rejected. Web uses a two-click push confirmation, and Apple mirrors the same controls. Destructive discard/reset, force push, arbitrary remotes/refspecs, and checkpoint restore are intentionally unsupported.
• Wrapper preferences persist without changing chat semantics: web retains the activity filter and resizable detail width in local storage, while Apple retains the activity filter in AppStorage. Selecting any other session still resets the detail mode to Conversation, so a previously inspected Terminal/Changes/Approvals pane can never hide a normal chat transcript. Web exposes Cmd/Ctrl+K for activity search, Cmd/Ctrl+Shift+N for a new agent, and Escape to clear search; macOS mirrors the search and new-agent shortcuts.
• Runtime conversations render oldest-to-newest across providers. Reply composers are available only in Conversation, preventing prompts from being entered while inspecting diagnostics; detached jobs retain their existing prompt/terminal workflow.
• Interactive Codex sessions render a typed timeline instead of a flat log wall: assistant replies are unboxed markdown prose, user messages are compact trailing bubbles, and Codex tool/command lifecycle items are quiet expandable rows. Tool details expose command/cwd, duration, exit code, output, and raw payload only on demand; unmatched low-level output is collapsed behind a bounded Terminal output disclosure.
• Multi-turn sessions remain one continuous chronological conversation. Turn IDs, event counts, provider avatars, and timestamps are not persistent section headers; timestamps stay secondary, while user prompts and assistant prose establish the turn boundaries naturally.
• Detached Codex jobs run codex exec --json. The job log preserves the legacy source and message fields and additionally carries typed eventType, itemType, itemId, turnId, itemStatus, item, and usage fields. Web and Apple render agent_message, command_execution, file_change, MCP/dynamic tool, and web-search items directly instead of classifying CLI stdout heuristically.
• Detached-job detail views use the same visual language on web and Apple: task and follow-up prompts appear as user messages, assistant text renders once as prose, the newest two tool actions stay visible, and older tool calls fold behind a +N previous tool calls disclosure. Routine runtime initialization/status chatter stays out of the primary transcript. The currently open job/session follows its SSE stream, preserves structured event fields across reconnect replay, and reconnects after transient transport failures. When a job is linked to a canonical session, the linked transcript owns user/assistant prose and job logs contribute only non-conversation activity, preventing duplicate answers.
• Repository-backed web timelines show a compact changed-files card with file count and diff totals; View diff opens the existing guarded Changes inspector. The composer remains anchored below the scroll content as a rounded floating surface, with attachments and the provider-scoped model selector inside its footer.
• Bridge runtime tool calls (tool.start / tool.end rows from agent_events, e.g. Claude CLI Bridge and Codex bridge runs) are reconstructed into the same expandable tool rows: calls are paired by toolCallId, labeled with the tool name plus an args-derived summary, and expose the executed command ($ …), result preview, and full raw JSON payload on demand. Older consecutive tool calls collapse while the newest actions remain visible. run.postprocess.* chatter is hidden unless it failed. The Apple client mirrors the same behavior.
• Clapilot-code entries use the same oldest-to-newest transcript direction as Codex and Claude detail panes, auto-scroll to the newest turn, and include a footer composer for follow-up prompts into the same runtime session. Raw runtime events and metadata stay hidden behind the provider icon in the pane header.
• Claude bridge entries expose attached model/tool runs plus the persisted Claude harness session id. The module footer can send follow-up prompts back into the same Claude bridge session; if the Claude CLI process is still running, the prompt is steered into the live stream, otherwise a new runtime turn resumes the persisted Claude session.
• The read-only native session detail pane shows readable timestamps on each model run and nests per-run tool usage events inside the corresponding turn/run card. When no classic chat_nachrichten transcript row exists for a native runtime session, the pane reconstructs the visible session transcript from the persisted run history instead of showing an empty transcript block. For native runtime runs that used a Mixture-of-Agents (MoA) preset, the run view also lists the reference model outputs (per-reference provider/model and response) alongside the aggregated answer.
• Interactive sessions continue directly from the terminal pane footer (no separate lower controls composer). Detached jobs expose a follow-up composer directly in the job detail pane: linked-session jobs continue the same hidden session, while plain CLI jobs continue in the same workspace and append output to the job log. The follow-up composers reuse the floating-chat input styling, minus voice/model controls, and support the same inline image/file attachments for first and follow-up turns. The new-agent composer carries repository and inferred harness/model selection plus optional speech-to-text in one footer row.
• User-owned local coding sessions and local job follow-ups support workspace file references in the composer. Type @ (or use the @ action), search the active working directory, and select a relative path. The request carries the selected paths as fileReferences[]; Codex, Claude CLI, and Clapilot-code are instructed to inspect those files relative to their active cwd without uploading or embedding the file contents. New-agent drafts have no checkout yet, legacy ownerless sessions are not claimed implicitly, and remote runner workspaces are not server-local, so the picker is intentionally unavailable in those cases.
• The web module keeps the selected interactive or linked-job session attached to GET /sessions/{id}/stream?replay=0 even while the session is ready or waiting for another turn. The initial SSE snapshot supplies recent history once; replay=0 prevents the same historical events from being emitted a second time during reconnects. New events then update the open detail pane without overview polling.
• Selected Clapilot-code and Claude bridge entries subscribe to the SSE stream of their linked external session and throttle runtime-detail reconciliation while events are arriving. Legacy runtime rows without a linked external stream, or rows whose persisted stream is no longer available after a runtime restart, use a 15-second selected-detail fallback poll instead of rapid SSE reconnects.
• The Apple client subscribes to the same SSE endpoints (GET /jobs/{id}/stream and GET /sessions/{id}/stream) for the selected activity item. Session list calls receive the new compact summary view by default, while streamed job logs and linked-session events remain the live detail path.

Run jobs on remote machines

• Detached jobs can set executionTarget: "remote" with provider codex, clapilot-code, or auto (which resolves to codex), and optionally remoteRunnerId to wait for a specific registered machine; otherwise any online runner that advertises the matching harness capability can claim the job. The + New Agent panel on web and the Apple client exposes local execution, automatic remote claiming, and pinned remote runners through its runner dropdown; the model menu offers Codex models plus, when a capable runner is online, Clapilot Code catalog models.
• Remote execution is pull-based: scripts/agent-orchestrator-codex-remote-runner.mjs runs on the remote machine, connects outbound to Clapilot with CLAPILOT_HUB_SHARED_SECRET, claims queued work, checks the repository out from a local shared bare cache into a per-job worktree, runs the assigned harness CLI (codex or clapilot-code), and reports logs/status back to the same job stream.
• Runner capability reporting: since codex-remote-runner/0.2.6 the Node runner always advertises codex (a missing Codex CLI fails loudly at spawn time, matching earlier versions) and additionally advertises clapilot-code when the CLI resolves on PATH or at ~/.clapilot-code/bin/clapilot-code.mjs. Fresh Clapilot Code jobs are only assigned to runners advertising the clapilot-code capability, so older runners are unaffected. The remote machine's clapilot-code must be logged in against an instance (clapilot-code login, or hub-driven provisioning below); job model selection uses this instance's Clapilot Code model catalog and is passed via clapilot-code exec --model. Codex stdout is parsed as --json JSONL into typed events. Clapilot Code jobs run with --output-format stream-json when the machine's CLI is 0.3.1 or newer, so assistant prose and tool calls render as the same typed conversation rows as Codex jobs; older CLIs keep emitting plain text, which is stored as raw job-log lines.
• Runner self-update (runner 0.2.8+): heartbeat/claim responses include a runnerUpdate: { version, source } payload while a self-update-capable runner is older than the hub's bundled script and idle. The runner syntax-checks the payload, verifies the declared version, atomically replaces its own script, and exits so its supervisor (fleet connector, launchd, systemd) restarts the new version; updates are never applied while jobs are running. Pre-0.2.8 runners cannot self-update — restarting the fleet connector (which re-downloads the bundled runner at startup, and self-updates itself when the hub advertises a newer connector version) upgrades them once.
• Remote provisioning (runner 0.2.7+): Settings -> Agent Orchestrator shows an Install Codex and an Install Clapilot Code button per registered machine, plus an auto-provision toggle. Actions are queued on the hub and executed by the runner on its next poll — the hub never connects to the machine. Install Clapilot Code ships the CLI source from the hub (no Developer mode needed), mints a dedicated, revocable inference:execute instance API key named remote-runner:<runnerId> (re-running rotates and revokes the previous key), and writes ~/.clapilot-code/config.json with the validated key and the first tool-capable catalog model. Install Codex installs the CLI via npm -g (falling back to npm --prefix ~/.local and brew) and writes the hub's stored Codex auth (codex_auth_json or OPENAI_API_KEY) into the machine's Codex home. When the auto-provision toggle is on, any connecting 0.2.7+ runner without the clapilot-code capability is provisioned once automatically; failed installs are never retried automatically and stay visible in the machine row. Install payloads (API keys, CLI source) travel only in runner-authenticated heartbeat/claim responses and are dropped from hub memory once the runner reports an outcome; browser-facing runner lists carry status, error, and detail only. The shell-tools kill switch blocks both queueing and delivery.
• The runner includes a read-only CLI-session snapshot in its heartbeat. It scans the remote machine's Codex home (CLAPILOT_REMOTE_RUNNER_CODEX_HOME, CODEX_HOME, or ~/.codex) for state_*.sqlite thread metadata and Codex CLI/Desktop history files (session_index.jsonl plus sessions/**/rollout-*.jsonl). It also scans ~/.clapilot-code/sessions/*.jsonl, or CLAPILOT_REMOTE_RUNNER_CLAPILOT_CODE_SESSIONS when configured. Both harnesses are exposed per machine through GET /remote-runners/{runnerId}/codex-sessions; selecting a session requests transcript detail, which the runner includes in a later heartbeat, so detail loading is asynchronous and outbound-only.
• Replying to a scanned Codex session requires the Node runner codex-remote-runner/0.2.3 or newer and runs codex exec resume with the explicit Codex home. Replying to a scanned Clapilot Code session requires codex-remote-runner/0.2.4 or newer and runs clapilot-code exec resume. Both paths use the scanned cwd when it still exists, without cloning or creating a repository worktree. The macOS Swift runner does not claim session-resume assignments.
• macOS machines can instead run the signed Clapilot Remote Runner menu-bar app. Its popup has an Overview / Connection / Log segmented control: Overview shows live worker stats, active sessions, and recent finished sessions (succeeded/failed/cancelled) with timing; Connection holds the Clapilot URL, Hub shared secret, runner id/label, workspace path, polling interval, slot count, and the Save/Test heartbeat/Quit actions; the secret is stored in Keychain. Start/stop and heartbeat status stay in the popup header and status bar. The macOS menu-bar runner advertises only the codex capability, so it never claims Clapilot Code jobs; use the Node runner on machines that should run both harnesses.
• Settings -> Agent Orchestrator lists registered remote machines with online/stale status, slots, capabilities, version, running jobs, and last heartbeat.
• The Clapilot server does not need SSH access or an inbound port on the remote machine. Treat connected runners as trusted, because repo jobs receive a GitHub token for cloning private repositories.

GitHub and GitLab automations

Configure named GitHub and GitLab accounts in Settings -> App Verbindungen, then select repositories in Settings -> Agent Orchestrator. GitLab connections accept GitLab.com or a self-managed instance URL. The settings page and the New Agent composer show one combined repository list; each row carries its GitHub or GitLab suffix icon and retains the source connection so same-named repositories do not collide. The picker caches the last loaded union locally, and a refresh button remains available.

The repository automation matrix selects a concrete model independently for PR review, issue implementation, and mention handling on every enabled repository. The picker reuses the same Codex, Claude, and Clapilot Code model catalog as New Agent; users select only the model, and the model's provider determines the coding harness. Main-CI failure fixes and Symphony coding tasks dispatched from that repository's mapped task board use the repository's issue-implementation model. Existing rows without model metadata remain backward-compatible with the former global provider defaults until the configuration is saved, at which point enabled rows persist concrete { provider, model } selections as prReviewModel, issueObserverModel, and mentionObserverModel inside agent_orchestrator_repo_automation_config.

For each repository you can independently enable:

• automatic pull-request or merge-request reviews
• automatic issue observer runs
• automatic mention replies
• default-branch CI failure fixes from GitHub or GitLab webhooks

Trigger mode is configured per repository as polling or webhook. Webhook rows show a repo-specific tokenized URL for the matching GitHub or GitLab project webhook; polling rows run from the interval loop. GitLab merge-request, issue, note, pipeline, and push hooks are normalized to the same scoped automation families as their GitHub equivalents.

Integrations and providers:

• Each repository retains its named GitHub or GitLab connection. Review scans, checkout, review submission, issue assignment, follow-up tracking, mentions, and CI inspection all use that same provider-specific connection.
• GitLab parity covers nested group paths, GitLab.com and self-managed base URLs, merge requests, issues, notes/discussions, approvals, pipeline jobs, emoji reactions, and project webhooks.
• Legacy github_token / github_pr_review_token values still exist as compatibility fallbacks, but the intended admin flow is named GitHub integrations.
• GitHub automations can select codex, claude, or local clapilot-code; the OpenClaw compatibility fallback is not used for review, implementation, or mention-reply submission.

PR review: scans non-draft open PRs, deduplicates by repo + PR number + head SHA, and derives a GitHub review decision from the generated findings:

• REQUEST_CHANGES when any finding is tagged critical or the review explicitly asks for changes
• APPROVE when the review reports no issues or only medium / minor findings
• COMMENT remains the fallback when the review output cannot be classified

Issue observer: assigns the issue to the configured GitHub identity when a coding run starts, comments that implementation has started, and closes the issue immediately after pickup so it cannot be picked up by another observer run. On success it persists the issue-to-PR link and comments the PR link back onto the issue. If another active implementation session already owns the same issue, the duplicate pickup is treated as already in progress and the issue stays closed. If implementation fails and no PR can be recovered, it reopens the issue and records the failure with the normal retry backoff. Codex-backed issue implementations default to interactive sessions so the same session can be resumed for later PR follow-up work. Issue observer settings include an Agent Goal toggle: when enabled and the implementation provider resolves to Codex or Claude, implementation and PR creation runs start with a /goal command derived from the issue task. PR creation prompts require PR bodies to include a closing issue reference, a leading ## Summary section written in plain language a non-technical reader can understand (what changed, why, and what behaves differently), plus a ## Testing Instructions section; custom issue-observer prompts still receive this mandatory PR body contract after template expansion.

Main CI failure fix: webhook-only. When enabled for a webhook-mode repository, completed failing workflow_run, check_run, or check_suite events on the repository default branch first pass an evidence gate. The failing SHA must still be the current default-branch head, and GitHub/GitLab must expose a completed failing job step or equivalent check output. Lost runners, incomplete steps on completed jobs, setup/cleanup failures, known infrastructure errors, stale heads, and evidence-free status deliveries are recorded without starting a coding run. Evidence-free check deliveries remain eligible for a later, richer workflow_run delivery.

An admitted run must reproduce the cited failure before changing files. If the failure is transient, already fixed, infrastructure-owned, or otherwise lacks a safe source-controlled repair, the run finishes with no_action and creates no PR. Repair agents receive no forge write credentials: they commit locally and return repair_ready with a proposed title/body. The orchestrator then verifies that the default branch has not advanced, the commit descends from the exact failing head, the checkout is clean, the diff is non-empty, the body starts with a plain-language ## Summary section and includes ## Testing Instructions, and the change is not only a JavaScript bundle-budget increase. Only after those checks pass does the orchestrator push the branch and create the PR/MR. A final forge read-back closes any invalid or raced publication before it can enter tracked follow-up. Valid repair PRs enter the same tracked follow-up loop as issue-observer PRs. Pull-request branch failures stay on the existing tracked-PR follow-up path.

Main-CI diagnostic sessions use a durable key derived from repository and failed head SHA. After a runtime restart, a workspace that still proves it is on the verified pre-turn commit may replay the diagnostic turn. A locally committed candidate interrupted before the module-owned publication gate remains protected and is not replayed by the write-disabled coding session. Pushed candidates and matching pull requests are reconciled where forge evidence is conclusive; unsupported repository identifiers, unexpected forge responses, and unavailable Git or forge evidence remain fail-closed.

Main-CI turns also keep a five-minute finalization reserve inside the global turn timeout. Three consecutive failed tool steps, five failed tool steps in total, or entry into that reserve asks the coding agent to stop mutations, inspect Git state, report its last validation, and emit the structured Main-CI result. The runtime persists metadata.mainCiAutofixCheckpoint before this steer and again before a hard-timeout interrupt. The checkpoint contains HEAD, status, a bounded binary diff, the last recognized test/build/lint command and result, failure counters, and explicit resume/discard instructions. Its disposition remains pending until the existing module-owned publication or cleanup path makes the outcome conclusive, so a timed-out workspace is never silently treated as valid or discarded.

Tracked PRs: PRs opened by orchestrator-managed coding runs or interactive sessions are tracked durably in the database until they merge or close.

• If an issue observer run fails after a linked PR was already opened, or a later manual/chat run opens a PR that closes the watched issue, the next issue scan recovers that PR into the same durable tracked-PR follow-up state instead of leaving it unmonitored.
• While a tracked PR remains open, the orchestrator watches for merge conflicts, failing GitHub checks, PR review comments, top-level PR review bodies, GitHub Actions CI report comments, and mention activity on the open PR branch. Failing check follow-up is not gated only by the PR thread updated_at timestamp, so edited GitHub Actions smoke/e2e reports still retrigger the tracked PR worker when the check run fails again.
• Every tracked-PR scan follow-up receives a new job-specific worktree. The source job and source session remain attribution/context references only for that path and are never used as its checkout, so a PR scan cannot replace or reset an earlier development workspace. Direct mention turns continue in their existing interactive session.
• CHANGES_REQUESTED reviews, merge conflicts, and critical comments trigger an immediate isolated follow-up run, while non-critical comments keep the guarded triage fallback. Triage can choose reply for direct questions, status requests, or clarification comments and post a GitHub PR conversation reply without changing code; it chooses follow_up for small safe code fixes (which can also post a result-specific PR conversation reply after updating the branch); ignore is reserved for noise or duplicate comments.
• Tracked PR status exposes recent follow-up trigger outcomes and failures for debugging (lastScanResults, recent handled triggers, recent failures).
• Active tracked-PR turns are checkpointed before agent execution. After a runtime restart the original trigger is restored once with the explicit requeued_after_restart status and dispatched with the same composite idempotency key. Review/conversation replies carry a hidden operation marker, so replay after a crash reuses an already posted GitHub comment or GitLab merge-request note instead of posting it twice.
• Detached/manual jobs that explicitly implement a GitHub issue share the same issue-level lock as the issue observer: a second job for the same repo issue is rejected while an implementation session is active or once an open PR already closes the issue.

Mention observer: watches configured issue and PR conversations for @<github-login> mentions of the GitHub identity behind the configured token, adds a lightweight reaction, and then either redirects the mention into the tracked PR session or starts a target-specific session/workspace for that PR or issue.

Runtime settings information

Symphony Runtime keeps primary metric values and runtime status visible. Secondary counts, polling and API-budget details, cache totals, and board/model/speed/reasoning explanations appear in adjacent info popovers. Field labels remain above their controls. Apple clients also expose automation count breakdowns and runtime field guidance through info buttons. Active errors, rate-limit cooldowns, and throttling notices remain visible.

Repository settings

Automation settings keep titles and field labels visible while descriptions, model guidance, Agent Goal help, and prompt hints live in adjacent info popovers. The repository header also places its four live automation counts inside the title info popover, keeping the collapsed header to one line. Remote Machines keeps its online status and refresh control visible; its description and automatic-provisioning explanation use info popovers, with the setup toggle remaining directly accessible on web and Apple clients. Web popovers open on hover, keyboard focus, or click; Apple clients support hover and tap/click. The three web automation cards align their controls and prompts without fixed-height description blocks.

Settings → Agent Orchestrator shows saved repository rows immediately, even before the GitHub/GitLab catalog is loaded or when a refresh returns only part of it. The catalog enriches existing rows without determining which rows are visible. Unconfigured repositories appear only in Add repository: open the dropdown, choose a repository, configure its board, trigger, and models, then save. Loading or refreshing the catalog does not add rows automatically. Web, iOS, and macOS use this same selection flow.

Adding a repository starts with all automations disabled. Saved rows remain configured even with every automation disabled; adding a row alone does not enable an observer. Repository discovery remains available to agents through agent_orchestrator_list_repos. Automation settings continue to use the existing admin-only config endpoint; the selection UI does not expand agent authority to change automation settings.

Symphony task-board dispatch

• Symphony's poll loop is controlled by a persisted enable/disable flag in app_settings.agent_orchestrator_symphony_enabled. POST /orchestrator/start enables and starts the loop, POST /orchestrator/stop disables it and clears queued retries. GET /status includes enabled; disabled loops do not dispatch aufgaben candidates. The UI control lives under Settings -> Agent Orchestrator, not in the user-facing module view.
• The Symphony task-board toggle only controls aufgaben coding task dispatch; GitHub automations continue to run from the per-repository automation matrix even when Symphony is disabled or no task board is selected.
• Dispatch observes the configured default task board in app_settings.agent_orchestrator_symphony_task_board_id plus any repo-specific boards selected in the Settings GitHub automation matrix. If Symphony is enabled without a default board or a repo board mapping, no task dispatch happens.
• Tasks inside repo-specific boards inherit the mapped GitHub repository automatically. Tasks inside the default board still need a concrete repo:owner/name reference in the task description so the agent can safely clone the repository.
• Repository coding tasks dispatched from a mapped board resolve the same concrete issueObserverModel used for issue implementation, including its implied Codex, Claude, or Clapilot Code harness. The job record stores the requested provider/model and the provider-reported effective model remains visible in Agent Activity. Rows without a concrete model retain the legacy global issue-observer provider fallback.
• Symphony dispatch uses local capacity first. Once the local concurrency or memory limit is reached, compatible Codex and Clapilot Code repository tasks reserve a slot on an online Fleet runner (0.3.1+), subject to the per-machine Orchestrator slots setting. Without compatible capacity, work remains queued. Claude and non-coding tasks remain local. Remote failures never bypass local admission for a fallback.
• Symphony only appends mandatory branch/commit/PR instructions for explicit repository coding tasks. General aufgaben records such as document or payroll review tasks without a repo stay in the non-repository task path and must not create GitHub PRs.
• Each Symphony attempt has its own persistent job workspace, preventing retries and parallel tasks from replacing one another's checkout.
• Symphony-created PR bodies must include both the Requested-by: Symphony task ... line and a Clapilot task: .../aufgaben/{id} link for traceability (using the configured public app URL, not internal container service URLs), plus a leading plain-language ## Summary section a non-technical reader can understand and a ## Testing Instructions section with concrete validation commands or manual checks. The coding job retains the originating task ID. After detecting the created PR, Symphony adds a system comment with its canonical URL, PR number, and open status to that task, localized from the task creator's persisted UI language (de, en, or it; private-board owner and German fallbacks). Separate PRs accumulate as separate comments; a typed UUID advisory lock and URL lookup run as separate statements in one transaction so retries and concurrent completion paths do not add the same canonical PR URL twice. Linking still runs when optional tracked-PR follow-up registration fails.
• Symphony tasks and repository automations share one resource-aware dispatch budget. A slot is reserved synchronously before asynchronous workspace preparation, preventing one poll from exceeding the configured limit. The safe default is one local run. The effective limit is also capped by total and currently free memory so coding workers cannot consume the runtime's safety reserve; GET /status exposes effectiveMaxConcurrentAgents and the capacity snapshot.
• Container bootstrap does not start Symphony by default. Set CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED=true only when a deployment should force the poll loop on during boot.

How the agent can drive it (tools)

Native ClapilotAICore tool contracts cover repository job orchestration and interactive session orchestration (services/clapilot-agent/src/sessions/index.mjs):

• agent_orchestrator_list_repos
• agent_orchestrator_start_job
• agent_orchestrator_list_jobs
• agent_orchestrator_get_job
• agent_orchestrator_follow_up_job
• agent_orchestrator_stop_job
• agent_orchestrator_start_session
• agent_orchestrator_list_sessions
• agent_orchestrator_get_session
• agent_orchestrator_send_turn
• agent_orchestrator_fork_session
• agent_orchestrator_close_session

Wiring notes:

• src/lib/agent-runtime/tool-proxy.ts forwards agent_orchestrator_* tool calls into this module API on behalf of the linked Clapilot user.
• Repository start tools forward the provider, named connection, instance URL, and clone URL returned by agent_orchestrator_list_repos; the module re-resolves the named connection and derives authenticated remote URLs server-side before using its token.
• services/clapilot-agent/src/orchestrator-sessions/index.mjs brokers interactive session lifecycle, Codex app-server integration, and event streaming into agent_events.
• Native/channel agent runs should use these tools for generic repo coding and PR-preparation work instead of legacy gateway execution paths.
• Website Canvas uses the same interactive session broker internally for iterative website edits, but keeps repo sync, preview, commit, and push ownership inside Website Canvas.

Configuration & limits

Repository checkout and cleanup:

• Server-side local automation keeps shared bare repository caches under .agent-orchestrator/repos and creates per-job/per-session Git worktrees under the existing job roots (manual-job-workspaces, github-automation-workspaces/*, and .agent-orchestrator/sessions).
• GitHub and GitLab tokens are supplied to Git fetches through temporary http.extraheader arguments instead of being persisted in remote URLs.
• Repository-backed Clapilot-code sessions keep the initiating Clapilot user id on initial and resumed turns, so user-scoped Agent Orchestrator tools remain available inside the linked coding session. Older sessions acquire and persist that binding when they are resumed from the signed-in module UI.
• Repository shells receive the selected credential only as ephemeral process environment: GitHub uses GH_TOKEN, GitLab uses GL_TOKEN and GITLAB_HOST, and Git push uses temporary process-local Git config. Named-connection and GitLab sessions use the embedded native runtime so credentials remain scoped to one session. Tokens are never written to the checkout remote URL, job log, or external-session metadata.
• Completed local automation worktrees are removed at job completion where possible, and the instance cleanup worker removes stale PR review, issue, main-CI, tracked PR, manual-job, and Symphony workspaces while protecting active work. Before either cleanup path removes a Git worktree, it checks whether HEAD contains commits absent from all remote-tracking refs. Such workspaces are retained temporarily with recovery metadata (workspace, branch, and commit), allowing operators to recover the commit without permitting abandoned worktrees to consume storage indefinitely. Native interactive session roots are excluded because the session subsystem owns their resumable lifetime.
• The crash-recovery reaper runs at startup and hourly. AGENT_ORCHESTRATOR_WORKSPACE_TTL_HOURS controls the orphan TTL (default 24); an atomic PID lock prevents concurrent reapers. Lease state plus the in-memory queued/running inventory protects active workspaces, and terminal jobs with linked interactive sessions stay protected for follow-up turns. Paused or other non-terminal Symphony task states remain active; only configured terminal task states become eligible after the TTL.
• Every GC emits structured agent_orchestrator_workspace_gc telemetry with workspace counts and bytesBefore, bytesAfter, and bytesFreed. AGENT_ORCHESTRATOR_WORKSPACE_HARD_LIMIT_GIB optionally rejects every new local manual or GitHub-automation worktree and emits agent_orchestrator_workspace_backpressure once orchestrator storage reaches the configured limit.
• Run-local dependency trees are never copied between worktrees. Removing a worktree recursively removes its node_modules; package managers should use their host/container-level shared content store (for pnpm, the configured global pnpm store).
• Bare repo caches are retained for reuse and are pruned by age only when Git reports no linked worktrees. The default cache TTL is 336 hours.
• Remote runners keep their own sibling repos cache next to the configured workspaces directory and remove stale completed job workspaces after 24 hours by default.

GitHub API budget:

• Every GitHub REST GET from the orchestrator is a conditional request. The local githubApiFetch wrapper keeps a bounded in-memory LRU of ETag + body per token hash and URL (default 2000 entries / 64 MiB, per-entry cap 4 MiB) and sends If-None-Match. GitHub answers unchanged resources with 304 Not Modified, which is replayed from the cache and does not count against the primary rate limit. Repeated polls of open PR lists, issue lists, PR details, check runs, reviews, and comments therefore only consume quota when something actually changed.
• The wrapper records x-ratelimit-limit/remaining/reset from every response (including 304s) per token. Before the GitHub section of a polling tick runs, the loop checks the most constrained token: it keeps a reserve of 15% of the limit (at least 100 requests) for interactive actions and, if the remaining budget cannot sustain the configured poll interval until the reset given what the previous scan consumed, it stretches the GitHub scan interval (up to 15 minutes) and skips ticks in between with the reason github_budget_throttled. Aufgaben/task-board dispatch is unaffected and still runs every tick.
• After a 403/429 rate-limit response the affected token enters a cooldown until GitHub's reset time. Polling and manual ticks skip the GitHub section during the cooldown (github_rate_limit_cooldown) instead of failing each feature with the same error; webhook ticks are also gated when every resolved token is cooling down.
• Session-history PR recovery (agent_external_sessions scan) runs at most every 5 minutes on polling ticks because it is a fallback path; manual ticks still run it immediately.
• GET /status exposes githubApi with rateLimit (limit, remaining, resetAt, observedAt), cooldown (resetAt, message), throttle (active, intervalMs, nextScanAt, reason, skippedTicks), cache (entries, bytes, hits, misses), and telemetry (requests, conditionalHits, quotaConsumed, errors, lastScanConsumed, lastScanAt). Settings -> Agent Orchestrator shows the remaining budget, reset time, cached-response count, and a cooldown/throttle notice on web, iOS, and macOS.

Detached Claude auth:

• Detached Claude jobs resolve auth from the configured Anthropic provider in Clapilot's database before falling back to legacy app settings or local CLI state.
• Claude subscription auth uses the stored Claude setup-token and maps it to the Claude CLI internally; it does not rely on ANTHROPIC_OAUTH_TOKEN being injected via Docker .env.

Supervisor:

• An optional supervisor loop periodically reviews orchestrator state (for example stale runs) and can be triggered or toggled via GET|POST /supervisor/run, POST /supervisor/start, and POST /supervisor/stop.
• Its configuration is persisted in app_settings (agent_orchestrator_supervisor_enabled, ..._provider, ..._model, ..._interval_ms, ..._stale_run_ms); the interval floor is 1 minute and the stale-run floor is 15 minutes.

Optional env overrides:

• AGENT_ORCHESTRATOR_CODEX_ARGS
• CLAPILOT_AGENT_CODEX_SERVICE_TIER
• AGENT_ORCHESTRATOR_CLAUDE_ARGS
• OPENCLAW_PRIMARY_MODEL
• CLAPILOT_AGENT_ORCHESTRATOR_BOOTSTRAP_ENABLED (force Symphony on at container boot)
• SYMPHONY_MAX_CONCURRENT (configured local automation concurrency, default 1, maximum 8)
• AGENT_ORCHESTRATOR_MIN_FREE_MEMORY_MIB (memory reserve protected from new local automation runs, default 3072)
• AGENT_ORCHESTRATOR_RUN_MEMORY_BUDGET_MIB (estimated memory required per new local automation run, default 4096)
• AGENT_ORCHESTRATOR_GITHUB_CONDITIONAL_CACHE_MAX_ENTRIES (ETag cache entries, default 2000)
• AGENT_ORCHESTRATOR_GITHUB_CONDITIONAL_CACHE_MAX_MIB (ETag cache byte budget, default 64)
• AGENT_ORCHESTRATOR_GITHUB_BUDGET_RESERVE_RATIO (share of the hourly limit the poll loop never spends, default 0.15)
• AGENT_ORCHESTRATOR_GITHUB_BUDGET_MIN_RESERVE (absolute request reserve floor, default 100)
• AGENT_ORCHESTRATOR_GITHUB_MAX_SCAN_INTERVAL_MS (upper bound for the stretched GitHub scan interval, default 900000)
• AGENT_ORCHESTRATOR_GITHUB_SESSION_RECOVERY_INTERVAL_MS (minimum spacing of session-history PR recovery on polling ticks, default 300000)
• AGENT_ORCHESTRATOR_GITHUB_RATE_LIMIT_FALLBACK_COOLDOWN_MS (cooldown when GitHub sends no reset header, default 300000)

API endpoints exposed by module

Base: /api/modules/agent-orchestrator/api — implementation: bundled-modules/agent-orchestrator/api/handler.mjs; reference: bundled-modules/agent-orchestrator/README.md.

• GET /tools
• GET /auth
• POST /auth
• DELETE /auth
• POST /repos
• GET /status
• GET /poll
• POST /config
• GET /webhook/{token}
• POST /webhook/{token}
• POST /orchestrator/start
• POST /orchestrator/stop
• GET|POST /supervisor/run, POST /supervisor/start, POST /supervisor/stop
• GET /remote-runners
• POST /remote-runners/heartbeat
• POST /remote-runners/claim
• GET /remote-runners/{runnerId}/codex-sessions
• GET /remote-runners/{runnerId}/codex-sessions/{sessionId}
• POST /remote-runners/{runnerId}/codex-sessions/{sessionId}/follow-up
• POST /remote-runners/jobs/{id}/events
• GET /coding-models (model options plus resolved default for claude, codex, and clapilot-code; the Clapilot-code list contains only usable non-subscription catalog models)
• GET /jobs
• POST /jobs (supports an optional model for all coding providers — Clapilot-code direct provider models plus concrete Claude/Codex CLI models; executionTarget: "remote" plus optional remoteRunnerId remains Codex-only, and optional codexGoalEnabled enables Codex/Claude first-turn /goal bootstrapping)
• GET /jobs/{id}
• GET /jobs/{id}/stream
• GET /jobs/{id}/workspace-files?q={query}&limit={count} (local jobs only; linked jobs resolve the linked coding session cwd)
• POST /jobs/{id}/follow-up (supports an optional model override and validated relative fileReferences[] for local follow-up runs; remote jobs reject file references. Follow-ups against a running/queued remote or plain-CLI job return 202 with turn.status: "queued" and are queued per job — max 20, dispatched in order after the current run ends with succeeded/failed; serialized jobs expose the pending count as queuedFollowUps)
• DELETE /jobs/{id}
• GET /sessions
• POST /sessions (supports canonical provider: "clapilot-code" plus model; legacy pi/embedded_pi aliases normalize to Clapilot-code and use the internal embedded_pi adapter)
• GET /sessions/{id}
• DELETE /sessions/{id}
• GET /sessions/{id}/workspace-files?q={query}&limit={count} (returns bounded, Git-ignore-aware relative file suggestions for an owned local workspace)
• POST /sessions/{id}/turns (supports an optional per-turn model override plus validated relative fileReferences[], forwarded to the native session broker; stale or invalid selected paths reject the request)
• GET /sessions/{id}/stream
• POST /sessions/{id}/fork
• POST /sessions/{id}/archive

Troubleshooting

• The module is not visible: Agent Orchestrator is hidden while Developer mode is disabled; enable Developer mode first.
• Symphony is enabled but no tasks dispatch: check that a default task board (app_settings.agent_orchestrator_symphony_task_board_id) or a repo-specific board mapping is configured, and that default-board tasks contain a concrete repo:owner/name reference.
• A second job for the same GitHub issue is rejected: the issue-level lock is intentional — one implementation session or open closing PR owns the issue at a time.
• GitHub API error 403 ... rate limit exceeded in the runtime card: the token's hourly budget is exhausted. The loop pauses GitHub scans until the reset shown in the cooldown notice; nothing needs to be restarted. If this recurs, check the githubApi.telemetry.lastScanConsumed value in GET /status: a high number per scan usually means many tracked PRs with running checks or a second consumer (Issue Reporter, another instance) sharing the same token. Raise the poll interval, split tokens per feature, or lower AGENT_ORCHESTRATOR_GITHUB_BUDGET_RESERVE_RATIO only if interactive actions do not need headroom.
• Remote job never starts: verify the runner heartbeat in Settings -> Agent Orchestrator (online/stale status) and that the runner shares the correct CLAPILOT_HUB_SHARED_SECRET; remote execution is outbound-only from the runner.

Specialized agents on remote runners

Fleet remote runner 0.3.0 also accepts pinned specialized-agent sessions via the native ClapilotAICore broker. This path uses persistent Codex app-server threads, a scoped Clapilot MCP bridge, and the runner's installed native computer/browser tools. It is separate from detached coding jobs and does not require installing the Agent Orchestrator module. Fleet connector 1.4.6 starts a specialist-only companion for Fleet-enabled Mac runner apps. See Specialized Agents for setup, approvals, session lifecycle, and limitations.

Local concurrency and Fleet overflow

Max local concurrent limits automation harnesses on the instance, with the existing free-memory reserve. After local capacity is exhausted, compatible Codex and Clapilot Code work can use remote machines: Symphony coding tasks, PR reviews, issue implementation, mentions, and tracked PR follow-ups. Admission reserves capacity synchronously before workspace preparation. Remote work does not consume a local harness slot. Git/workspace preparation and result bookkeeping still run on the hub.

Each machine has an Orchestrator slots setting in Settings → Agent Orchestrator on web, iOS, and macOS. It persists by runner ID, defaults to 1, accepts 0–8, and is separate from the runner's base capacity for manually requested work. Zero disables new automatic work. Reducing the limit lets running work finish and keeps excess queued work waiting. Other active remote jobs consume machine capacity too. The fleet's usable total depends on online runners, harness capabilities, advertised model restrictions, and occupied slots; models are never silently substituted.

Overflow needs remote runner 0.3.1+, delivered by the existing runner self-update mechanism. Its assignment protocol preserves the hub-prepared commit, branch, upstream, and review refs. Runners serialize shared repository preparation to avoid concurrent cache initialization. Older runners remain available for existing manual workflows but receive no automatic overflow. Offline or incompatible machines are skipped. A failed remote assignment uses the normal workflow failure/retry handling; it never starts an unreserved local fallback.

Main-CI repair remains local because its write-disabled harness and hub-owned validation/publication require the local candidate checkout. It still obeys the local limit. Personal specialist runs and manually started jobs/sessions keep their existing scheduling semantics.

Chat/live agents can inspect machines and, with administrator access, update limits using agent_orchestrator_fleet (list, set_slots). Migration 303_agent_orchestrator_runner_limits.sql must run before the updated handler starts.

Fleet overflow reliability

Changing per-runner Orchestrator slots requires an administrator session on the module HTTP endpoint; authenticated internal service calls retain the existing trusted-service access. Agent tools retain their administrator check. Before the first successful database load, unknown runner slot limits allow no automation; after a successful load, transient failures preserve the last known limits. Manual base capacity remains available.

Runner control payloads and listings expose base capacity plus Orchestrator slots (maximum 16 on runner 0.3.2+; 0.3.1 runners retain automation capped at 8 until their idle-time self-update; unsupported native runners retain only their reported base capacity while their saved Orchestrator slot setting remains visible). Once Orchestrator slots are configured, the hub-provided combined capacity can exceed the machine-local CLAPILOT_REMOTE_RUNNER_MAX_CONCURRENT value; that environment setting controls base/manual capacity rather than imposing a hard machine ceiling. Operators should size slots for the runner's available CPU and memory. The bundled runner does not advertise a complete model catalog: absent model restrictions mean the selected model is attempted by the configured harness, and an unsupported-model error follows normal job failure handling. Optional advertised model maps constrain routing; an explicit null clears previous restrictions. Models are never silently substituted.

Codex Cloud per Repository

Unter Einstellungen → Agent Orchestrator → Repositories lässt sich Codex Cloud für jedes GitHub-Repository einzeln aktivieren. Die Cloud-Umgebungs-ID stammt aus der eingerichteten Codex-Cloud-Umgebung (codex cloud zeigt die verfügbaren Umgebungen); der Cloud-Branch muss bereits auf GitHub liegen. Die Zuordnung muss zum Repository passen. Der Clapilot-Host benötigt die Codex CLI mit cloud exec, cloud status und cloud diff sowie ein verbundenes Codex-Konto mit Cloud-Zugang. Ein lokaler GitHub-Token ersetzt diesen Zugang nicht.

Beim Anlegen eines Agenten erscheint für aktivierte Repositories Codex Cloud im Ausführungsmenü. Die Cloud verwendet ihr Standardmodell; lokale Modell-, Workspace-, Anhangs- und MCP-Einstellungen werden nicht übertragen. Code und Aufgabenbeschreibung werden auf OpenAI-Infrastruktur verarbeitet. Bestehende lokale Jobs, Fleet-Runner und Repository-Automatisierungen behalten ihre bisherigen Einstellungen. Die Aktivierung stellt eine zusätzliche Auswahl bereit und schaltet keine automatische Cloud-Ausführung ein.

Clapilot speichert die Cloud-Aufgaben-ID, verfolgt den Status auch nach einem Neustart und zeigt abgeschlossene Änderungen unter Änderungen. Cloud-Aufgabe öffnen führt zur weiteren Bearbeitung in Codex Cloud. Lokales Stoppen, Folgenachrichten und Git-Aktionen sind für diese Aufgaben nicht verfügbar; Diffs werden nicht automatisch angewendet oder veröffentlicht. Bei Verbindungsproblemen bleibt die Aufgabe aktiv und die Statusabfrage wird wiederholt. Eine unklare Übermittlung wird niemals automatisch erneut gestartet: vor einem manuellen Wiederholungsversuch Codex Cloud prüfen, um doppelte Aufgaben zu vermeiden.

Web, iOS und macOS unterstützen dieselbe Repository-Konfiguration und Ausführungsauswahl. Chat- und Live-Agenten verwenden agent_orchestrator_cloud zum Lesen bzw. auf ausdrücklichen Wunsch eines Administrators zum Konfigurieren, agent_orchestrator_start_job mit executionTarget: "cloud" zum Start und agent_orchestrator_get_job für Status, Link und Diff.